search

Iridescent-CM / technovation-app

The team submission platform for the Technovation Challenge
https://technovationchallenge.org
GNU General Public License v3.0
7 stars 4 forks source link

Tests shouldn't fail when season rolls over [40,13,3.1] #2550

Closed stenington closed 4 years ago

stenington commented 4 years ago

Last year, certain tests began failing when the season rolled over until date config was updated, and certificate templates were copied into a new directory. See https://github.com/Iridescent-CM/technovation-app/pull/2233

I don't think the test suite should pass one day and fail the next, with no changes in between other than the actual date. These tests should probably be abstracted away from the actual app settings, and maybe we should be testing multiple cases (e.g. before a configured date, after a configured date).

stenington commented 4 years ago

These are the failures I get by setting the 2020 dates in my .env back to 2019, and removing the 2019 certificate template directory:

rspec ./spec/features/judge/certificates_spec.rb:85 # Judge certificates judge who is suspended - it doesn't display the certificate's page
rspec ./spec/features/judge/certificates_spec.rb:101 # Judge certificates judge who doesn't have a certificate - it doesn't display the certficate's page
rspec ./spec/features/judge/certificates_spec.rb:144 # Judge certificates judge with 5 completed current scores
rspec ./spec/features/judge/certificates_spec.rb[1:11] # Judge certificates judge with 6 completed current scores
rspec ./spec/features/judge/certificates_spec.rb[1:12] # Judge certificates judge with 7 completed current scores
rspec ./spec/features/judge/certificates_spec.rb[1:13] # Judge certificates judge with 8 completed current scores
rspec ./spec/features/judge/certificates_spec.rb[1:14] # Judge certificates judge with 9 completed current scores
rspec ./spec/features/judge/certificates_spec.rb[1:15] # Judge certificates judge with 10 completed current scores
rspec ./spec/features/judge/certificates_spec.rb:201 # Judge certificates judge with 11 or more completed current scores
rspec ./spec/features/judge/certificates_spec.rb:230 # Judge certificates RPE judges get a head judge certificate
rspec ./spec/features/judge/certificates_spec.rb:259 # Judge certificates RPE judges with more than 10 scores get a judge advisor certificate
rspec ./spec/features/judge/starting_scores_spec.rb:87 # starting scores as a virtual judge starting a new score
rspec ./spec/features/mentor/certificates_spec.rb:54 # Mentor certificates appreciation certificate per team
rspec ./spec/features/mentor/find_a_mentor_spec.rb:36 # Mentors find a team browse nearby mentors
rspec ./spec/features/mentor/find_a_mentor_spec.rb:117 # Mentors find a team visit the mentor page
rspec ./spec/features/mentor/find_a_team_spec.rb:114 # Mentors find a team request to join a team
rspec ./spec/features/mentor/find_a_team_spec.rb:132 # Mentors find a team request to join the same team again, even if you deleted an earlier request
rspec ./spec/features/student/certificates_spec.rb:10 # Student certificates virtual semifinalist students receive a semifinalist certificate
rspec ./spec/features/student/certificates_spec.rb:26 # Student certificates virtual semifinalist students no completion certificate is generated
rspec ./spec/features/student/certificates_spec.rb:34 # Student certificates virtual semifinalist students no participation certificate is generated
rspec ./spec/features/student/certificates_spec.rb:48 # Student certificates virtual quarterfinalist students receive a completion certificate
rspec ./spec/features/student/certificates_spec.rb:64 # Student certificates virtual quarterfinalist students no participation certificate is generated
rspec ./spec/features/student/certificates_spec.rb:92 # Student certificates when the student has completed 50-99% of the submission a participation certificate is generated
rspec ./spec/features/student/certificates_spec.rb:108 # Student certificates when the student has completed 50-99% of the submission no completion certificate is generated
rspec ./spec/features/student/find_a_team_spec.rb:31 # Students find a team request to join a team
rspec ./spec/features/student/find_a_team_spec.rb:45 # Students find a team onboarded student sees pending requests
rspec ./spec/jobs/certificate_job_spec.rb:4 # CertificateJob fills and attaches PDFs
rspec ./spec/jobs/certificate_job_spec.rb:17 # CertificateJob tracks the job in the database
rspec ./spec/jobs/certificate_job_spec.rb:32 # CertificateJob adds the certificale file url to the DB job payload
rspec ./spec/models/account_spec.rb:69 # Account removes current certificates if the name is changed
rspec ./spec/system/student/request_to_join_a_team_spec.rb:79 # Students request to join a team a valid student requestor student accepts the request
rspec ./spec/system/student/request_to_join_a_team_spec.rb:118 # Students request to join a team a valid student requestor student accepts from team page
rspec ./spec/system/student/request_to_join_a_team_spec.rb:132 # Students request to join a team a valid student requestor student declines from team page
rspec ./spec/system/student/request_to_join_a_team_spec.rb:146 # Students request to join a team a valid student requestor mentor accepts the request
rspec ./spec/system/student/request_to_join_a_team_spec.rb:165 # Students request to join a team a valid student requestor mentor accepts from team page
rspec ./spec/system/student/request_to_join_a_team_spec.rb:179 # Students request to join a team a valid student requestor mentor declines from team page
rspec ./spec/system/student/request_to_join_a_team_spec.rb:193 # Students request to join a team a valid student requestor student declines the request
rspec ./spec/system/student/request_to_join_a_team_spec.rb:219 # Students request to join a team a valid student requestor mentor declines the request
rspec ./spec/technovation/determine_certificates_spec.rb:40 # DetermineCertificates for student awards nothing if equivalent certificate exists
stenington commented 4 years ago

Rather than setting 2019 variables and letting it be the 2020 season to trigger failures, I tried setting 2020 config but making the season rollover early enough that it's the 2021 season. This seems to have triggered failures in calculating divisions on top of the failures listed above.

Also the extended capybara timeouts that we've had to put in place for certain pages using Vue are really painful when tests start breaking. I'm not sure what to do about it, but it's another downside of how we're currently working with Vue, I think.