ualbertalib / jupiter

Jupiter is a University of Alberta Libraries-based initiative to create a sustainable and extensible digital asset management system. This is phase 2 (Digitization).
https://era.library.ualberta.ca/
MIT License
23 stars 10 forks source link

Resolve flapping tests #3045

Open ConnorSheremeta opened 1 year ago

ConnorSheremeta commented 1 year ago

We added the gem mini-test retry here to retry flapping tests that are resolvable through retries. Ideally these issues would be resolved where these tests would not flap at all.

Below are instances in CI where these flapping tests have failed:

BatchIngestTest#test_invalid_without_files DraftControllerTest#test_should_not_be_able_to_update_a_draft_item_when_saving_upload_files_form_that_has_no_file_attachments DraftControllerTest#test_should_not_be_able_to_update_a_draft_thesis_when_saving_upload_files_form_that_has_no_file_attachments DepositThesisTest#test_be_able_to_deposit_and_edit_a_thesis_successfully AdminUsersIndexTest#test_should_be_able_to_autocomplete_by_name ThesisListFilesTest#test_files_are_alphabetically_sorted_when_depositing_an_item ItemListFilesTest#test_files_are_alphabetically_sorted_when_depositing_an_item

We also currently have flapping tests being skipped that are not resolvable in these tests: test/integration/read_only_mode_integration_test.rb test/system/item_edit_test.rb test/integration/oaisys_list_identifiers_test.rb test/integration/doi_reset_test.rb

pgwillia commented 1 year ago
Error:
BatchIngestTest#test_invalid_without_files:
RSolr::Error::ConnectionRefused: Connection refused - {:params=>{:wt=>:json, :softCommit=>true}, :data=>"{\"delete\":{\"query\":\"*:*\"}}", :headers=>{"Content-Type"=>"application/json"}, :method=>:post, :query=>"wt=json&softCommit=true", :path=>"update", :uri=>#<URI::HTTP http://localhost:8983/solr/test/update?wt=json&softCommit=true>}
    (eval):2:in `post'
    app/models/jupiter_core/solr_services/client.rb:29:in `truncate_index'
    test/test_helper.rb:55:in `teardown'

rails test test/models/batch_ingest_test.rb:19
pgwillia commented 1 year ago
Failure:
Items::DraftControllerTest#test_should_not_be_able_to_update_a_draft_item_when_saving_upload_files_form_that_has_no_file_attachments [/home/runner/work/jupiter/jupiter/test/controllers/items/draft_controller_test.rb:140]:
Expected response to be a <2XX: success>, but was a <422: Unprocessable Entity>

rails test test/controllers/items/draft_controller_test.rb:131
Failure:
Admin::Theses::DraftControllerTest#test_should_not_be_able_to_update_a_draft_thesis_when_saving_upload_files_form_that_has_no_file_attachments [/home/runner/work/jupiter/jupiter/test/controllers/admin/theses/draft_controller_test.rb:137]:
Expected response to be a <2XX: success>, but was a <422: Unprocessable Entity>

rails test test/controllers/admin/theses/draft_controller_test.rb:128
pgwillia commented 1 year ago
Failure:
DepositThesisTest#test_be_able_to_deposit_and_edit_a_thesis_successfully [/home/runner/work/jupiter/jupiter/test/system/deposit_thesis_test.rb:62]:
expected to find visible css "h2" with text "Review and Deposit Thesis" but there were no matches. Also found "Upload Files", which matched the selector but not all filters. 

rails test test/system/deposit_thesis_test.rb:5
pgwillia commented 1 year ago
Failure:
AdminUsersIndexTest#test_should_be_able_to_autocomplete_by_name [/home/runner/work/jupiter/jupiter/test/system/admin_users_index_test.rb:58]:
expected to find visible css "div" with text "Displaying 1 of 1 matching users" but there were no matches. Also found "Dashboard\nUsers\nAnnouncements\nCommunities and Collections\nItems\nFeature Flags\nUsers\nSearch\nRole\nAll\nAdmin\nUser\nStatus\nAll\nSuspended\nActive\nEmail  Name  Registered  Last Seen  Role  Status \nadministrator@example.com Administrator 2023-01-17 less than a minute ago Admin Active\ntyrion_lannister@example.com Tyrion Lannister 2023-01-17 Never Signed In Admin Active\njohn_snow@example.com John Snow 2023-01-17 Never Signed In User Active\ndaenerys_targaryen@example.com Daenerys Targaryen 2023-01-17 Never Signed In User Active\njoffrey_baratheon@example.com Joffrey Baratheon 2023-01-17 Never Signed In User Suspended\nditech@ualberta.ca System user 2023-01-17 Never Signed In User Active\nDisplaying 6 of 6 matching users (6 total users)", "Email  Name  Registered  Last Seen  Role  Status \nadministrator@example.com Administrator 2023-01-17 less than a minute ago Admin Active\ntyrion_lannister@example.com Tyrion Lannister 2023-01-17 Never Signed In Admin Active\njohn_snow@example.com John Snow 2023-01-17 Never Signed In User Active\ndaenerys_targaryen@example.com Daenerys Targaryen 2023-01-17 Never Signed In User Active\njoffrey_baratheon@example.com Joffrey Baratheon 2023-01-17 Never Signed In User Suspended\nditech@ualberta.ca System user 2023-01-17 Never Signed In User Active\nDisplaying 6 of 6 matching users (6 total users)", which matched the selector but not all filters. 

rails test test/system/admin_users_index_test.rb:48
pgwillia commented 1 year ago
Failure:
ThesisListFilesTest#test_files_are_alphabetically_sorted_when_depositing_an_item [/home/runner/work/jupiter/jupiter/test/system/thesis_list_files_test.rb:28]:
expected to find visible css "#js-files-list ul li" 4 times, found 3 matches: "1234-text-sample.txt\nSet as Thumbnail Remove\n1234-text-sample.txt\n55 Bytes", "Set as Thumbnail Remove\nimage-sample.jpeg\n11.8 KB", "pdf-sample.pdf\nRemove\npdf-sample.pdf\n7.76 KB"

rails test test/system/thesis_list_files_test.rb:5
pgwillia commented 1 year ago
Failure:
ItemListFilesTest#test_files_are_alphabetically_sorted_when_depositing_an_item [/home/runner/work/jupiter/jupiter/test/system/item_list_files_test.rb:28]:
expected to find visible css "#js-files-list ul li" 4 times, found 3 matches: "Set as Thumbnail Remove\nimage-sample.jpeg\n11.8 KB", "Other-text-sample.txt\nSet as Thumbnail Remove\nOther-text-sample.txt\n44 Bytes", "pdf-sample.pdf\nRemove\npdf-sample.pdf\n7.76 KB"

rails test test/system/item_list_files_test.rb:5
pgwillia commented 1 year ago

https://github.com/ualbertalib/jupiter/actions/runs/4029084396/jobs/6926614743 still failed

Failure:
DepositThesisTest#test_be_able_to_deposit_and_edit_a_thesis_successfully [/home/runner/work/jupiter/jupiter/test/system/deposit_thesis_test.rb:62]:
expected to find visible css "h2" with text "Review and Deposit Thesis" but there were no matches. Also found "Upload Files", which matched the selector but not all filters. 

rails test test/system/deposit_thesis_test.rb:5