search

cyfronet-fid / marketplace

Marketplace is a place where you can find services you need for your research
https://marketplace.eosc-portal.eu
GNU General Public License v3.0
21 stars 6 forks source link

Research non-deterministic (random) feature tests (capybara) failures #2049

Open michal-szostak opened 3 years ago

michal-szostak commented 3 years ago

Currently some feature tests (using JS) fail in a non deterministic way. Current solution is a workaround forcing tests to repeat set number of times until they pass. This solution will not work in the long term. The end goal of this issue is to find solution for this issue, possible options may include:

@danielkryska - If you can please include here which tests are the usual failing ones.

EDIT: @kmarszalek https://docs.cyfronet.pl/display/FID/MP+-+Rails+-+Async+JS+Tests

danielkryska commented 3 years ago

Tests skipped due to frequent failures. Those aren't all of them, because turning off one case pop another.

  1) ApplicationController Should fetch recommended service ids
     # Temporarily skipped with xit
     # ./spec/controllers/concerns/recommendable_spec.rb:28

  2) Api docs page as a regular user I can see Marketplace API link
     # Marketplace API link shouldn't be here for now
     # ./spec/features/api_docs_spec.rb:13

  3) Comparison doesn't show comparison bar until I click the Add to compare checkbox
     # No reason given
     # ./spec/features/comparison_spec.rb:17

  4) Comparison shows comparison bar after click the Add to compare checkbox
     # No reason given
     # ./spec/features/comparison_spec.rb:23

  5) Comparison shows comparison bar on services view
     # No reason given
     # ./spec/features/comparison_spec.rb:31

  6) Comparison blocks other services checkboxes when 3 are checked
     # No reason given
     # ./spec/features/comparison_spec.rb:41

  7) Comparison clears all elements when click on CLEAR ALL
     # No reason given
     # ./spec/features/comparison_spec.rb:55

  8) Comparison clears chosen element when click on trash icon
     # No reason given
     # ./spec/features/comparison_spec.rb:69

  9) Comparison shows comparison page with correct data after click on compare
     # No reason given
     # ./spec/features/comparison_spec.rb:84

  10) Comparison deletes service from comparison on comparison page
     # No reason given
     # ./spec/features/comparison_spec.rb:162

  11) Comparison redirects to services path when last service is deleted from comparison
     # No reason given
     # ./spec/features/comparison_spec.rb:175

  12) DataAdministrator add some examples to (or delete) /home/runner/work/marketplace/marketplace/spec/models/data_administrator_spec.rb
     # Not yet implemented
     # ./spec/models/data_administrator_spec.rb:6

  13) ServiceRelationship add some examples to (or delete) /home/runner/work/marketplace/marketplace/spec/models/service_relationship_spec.rb
     # Not yet implemented
     # ./spec/models/service_relationship_spec.rb:6
jswk commented 3 years ago

Another failing one:

   1) Recommender::SimpleRecommender Simple recommender service call with n=1 returns services in the order of the most popular
     Failure/Error: expect(recommended_services_ids).to match_array(most_popular_services_ids)

       expected collection contained:  [594]
       actual collection contained:    [595]
       the missing elements were:      [594]
       the extra elements were:        [595]
     # ./spec/services/recommender/simple_recommender_spec.rb:40:in `block (4 levels) in <top (required)>'
jswk commented 3 years ago

Another:

  1) Favourites As anonymous user I can see checkbox set to true in other view
     Failure/Error: expect(page.find("input#favourite-#{service.id}", visible: false)).to be_checked
       expected `#<Capybara::Node::Element tag="input" path="/HTML/BODY[1]/MAIN[1]/DIV[2]/DIV[1]/DIV[1]/DIV[1]/DIV[2]/DIV[1]/DIV[1]/DIV[4]/SPAN[1]/LABEL[1]/INPUT[1]">.checked?` to return true, got false
     # ./spec/features/favourite_spec.rb:36:in `block (3 levels) in <top (required)>'
     # ./spec/rails_helper.rb:84:in `block (2 levels) in <top (required)>'
jswk commented 3 years ago

https://github.com/cyfronet-fid/marketplace/pull/2244/checks?check_run_id=3280904791

   1) Providers in backoffice with JS: As a service portolio manager I can delete external source
     Failure/Error: expect(page).to have_content("Provider updated correctly")
       expected to find text "Provider updated correctly" in "Contact us Portal Home Catalogue & Marketplace Providers Dashboard Backoffice John36 Doe36 Logout\nGetting Started\nOwned Resources\nProviders\nScientific domains\nCategories\nPlatforms\nVocabularies\nLogout\nBackoffice\n› Providers › provider 133 › Edit\nEdit Provider\nBasic\nMarketing\nClassification\nLocation\nContact\nMaturity\nOther\nAdmins\nBasic\nMarketing\nClassification\nLocation\nContact\nMaturity\nOther\nAdmins\nCANCEL AND BACK TO PREVIOUS PAGE\nCopyright 2020 - All rights reserved PRIVACY POLICY\nAbout\nEOSC\nEOSC Portal\nEOSC Doers\nEOSC Glossary\nEOSC Projects\nEOSC Governance\nEOSC Portal FAQs\nEOSC Association\nEOSC Enhance\nServices & Resources\nHelp Desk\nPolicy\nEurope\nMember States\nUse Cases\nCommunity Use Cases\nSubmit Use Case\nIndustry Use Case\nMedia\nNews\nForthcoming events\nPast Events\nSubmit your News/Event\nDocuments\nEuropean Commission\nCommunity\nMember States\nFor providers\nProviders Documentation\nSubscribe\nUsing the Portal\nThe EOSC Portal is operated by the EOSC Enhance (Grant Agreement no. 871160), EOSC-hub (Grant Agreement no. 777536), and OpenAIRE-Advance (Grant Agreement no. 777541) projects funded by the European Union’s Horizon 2020 research and innovation programme.For a complete list of contributors, visit the About EOSC Portal"
     # ./spec/features/backoffice/providers_spec.rb:82:in `block (3 levels) in <top (required)>'
wujuu commented 3 years ago

Another one:

1) Providers in backoffice with JS: As a service portolio manager I can remove data administrator
     Failure/Error: expect(provider.data_administrators.count).to eq(1)

       expected: 1
            got: 2

       (compared using ==)
     # ./spec/features/backoffice/providers_spec.rb:65:in `block (3 levels) in <top (required)>'
kmarszalek commented 3 years ago

https://github.com/cyfronet-fid/marketplace/runs/3351304879

./spec/features/backoffice/providers_spec.rb:82

Providers in backoffice with JS: As a service portolio manager I can delete external source
     Failure/Error: expect(page).to have_content("Provider updated correctly")
       expected to find text "Provider updated correctly" in "Contact us Portal Home Catalogue & Marketplace Providers Dashboard Backoffice John37 Doe37 Logout\nGetting Started\nOwned Resources\nProviders\nScientific domains\nCategories\nPlatforms\nVocabularies\nLogout\nBackoffice\n› Providers › provider 133 › Edit\nEdit Provider\nBasic\nMarketing\nClassification\nLocation\nContact\nMaturity\nOther\nAdmins\nBasic\nMarketing\nClassification\nLocation\nContact\nMaturity\nOther\nAdmins\nCANCEL AND BACK TO PREVIOUS PAGE\nCopyright 2020 - All rights reserved PRIVACY POLICY\nAbout\nEOSC\nEOSC Portal\nEOSC Doers\nEOSC Glossary\nEOSC Projects\nEOSC Governance\nEOSC Portal FAQs\nEOSC Association\nEOSC Enhance\nServices & Resources\nHelp Desk\nPolicy\nEurope\nMember States\nUse Cases\nCommunity Use Cases\nSubmit Use Case\nIndustry Use Case\nMedia\nNews\nForthcoming events\nPast Events\nSubmit your News/Event\nDocuments\nEuropean Commission\nCommunity\nMember States\nFor providers\nProviders Documentation\nSubscribe\nUsing the Portal\nThe EOSC Portal is operated by the EOSC Enhance (Grant Agreement no. 871160), EOSC-hub (Grant Agreement no. 777536), and OpenAIRE-Advance (Grant Agreement no. 777541) projects funded by the European Union’s Horizon 2020 research and innovation programme.For a complete list of contributors, visit the About EOSC Portal"
kmarszalek commented 3 years ago

scenario "I can create new provider with data administrators", skip: true, js: true do

scenario "I can create provider with external source", skip: true do

kmarszalek commented 3 years ago

1) Providers in backoffice with JS: As a service portolio manager I can delete external source Failure/Error: expect(page).to have_content("Provider updated correctly") expected to find text "Provider updated correctly" in "Contact usPortal HomeCatalogue & MarketplaceProviders DashboardProviders DocumentationJohn37 Doe37Logout\nGetting Started\nOwned Resources\nProviders\nScientific domains\nCategories\nPlatforms\nVocabularies\nBackoffice\n› Providers › provider 133 › Edit\nEdit Provider\nBasic\nMarketing\nClassification\nLocation\nContact\nMaturity\nOther\nAdmins\nBasic\nMarketing\nClassification\nLocation\nContact\nMaturity\nOther\nAdmins\nCANCEL AND BACK TO PREVIOUS PAGE\nCopyright 2020 - All rights reservedPRIVACY POLICY\nAbout\nEOSC\nEOSC Portal\nEOSC Doers\nEOSC Glossary\nEOSC Projects\nEOSC Governance\nEOSC Portal FAQs\nEOSC Association\nEOSC Enhance\nServices & Resources\nSharing & Discovery\nTraining & Support\nSecurity & Operation\nProcessing & Analysis\nAccess physical & einfrastructures\nAggregators & Integrators\nOther\nHelp Desk\nPolicy\nEurope\nMember States\nUse Cases\nCommunity Use Cases\nSubmit Use Case\nIndustry Use Case\nMedia\nNews\nForthcoming events\nPast events\nSubmit your News/Event\nDocuments\nEuropean Commission\nCommunity\nMember States\nFor providers\nProviders Documentation\nSubscribe\nUsing the portal\nThe EOSC Portal is currently operated by the EOSC Enhance (Grant Agreement no. 871160) and EOSC Future (Grant Agreement no. 101017536) projects funded by the European Union’s Horizon 2020 research and innovation programme. About EOSC Portal"