DEFRA / ruby-services-team

Guides, info and issue management for the Ruby Services team
Other
1 stars 3 forks source link

Fix intermittent test failures for WCR letter export services in Travis #83

Open irisfaraway opened 4 years ago

irisfaraway commented 4 years ago

Occasionally the unit tests for the AdReminderLettersExportService or the DigitalReminderLettersExportService will fail. However, this only happens on Travis – when the tests are run locally with the same seed, it's fine.

It's unclear what the cause of this is, but possible guesses include:

The seed 13731 reliably triggers the AdReminderLettersExportService specs to fail, so the order of the tests may have something to do with it.


Some previous investigation was done here: https://github.com/DEFRA/waste-carriers-back-office/pull/974/files

Here is an example of a broken build: https://travis-ci.com/github/DEFRA/waste-carriers-back-office/builds/177306791

With the failures looking like:

Failures:
  1) AdReminderLettersExportService.run when there are assisted digital registrations that are due for renewal reminders loads a file to a AWS bucket and records the content created
     Failure/Error: Airbrake.notify e, file_name: file_name
       (Airbrake).notify(#<Errno::ENOENT: No such file or directory @ rb_sysopen - /home/travis/build/DEFRA/waste-carriers-back-office/tmp/ad_reminder_letters_20201025.pdf>, {:file_name=>"ad_reminder_letters_20201025.pdf"})
           expected: 0 times with any arguments
           received: 1 time with arguments: (#<Errno::ENOENT: No such file or directory @ rb_sysopen - /home/travis/build/DEFRA/waste-carriers-back-office/tmp/ad_reminder_letters_20201025.pdf>, {:file_name=>"ad_reminder_letters_20201025.pdf"})
     # ./app/services/reminder_letters_export_service.rb:21:in `rescue in run'
     # ./app/services/reminder_letters_export_service.rb:20:in `run'
     # ./spec/services/ad_reminder_letters_export_service_spec.rb:30:in `block (4 levels) in <top (required)>'
     # ./spec/support/database_cleaner.rb:17:in `block (3 levels) in <top (required)>'
     # ./spec/support/database_cleaner.rb:16:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # Errno::ENOENT:
     #   No such file or directory @ rb_sysopen - /home/travis/build/DEFRA/waste-carriers-back-office/tmp/ad_reminder_letters_20201025.pdf
     #   ./app/services/reminder_letters_export_service.rb:12:in `initialize'
  2) AdReminderLettersExportService.run when there are assisted digital registrations that are due for renewal reminders when one registration is in an invalid state and a PDF cannot be generated for it raises an error on Airbrake but continues generation for the other letters
     Failure/Error: expect(ad_reminder_letters_export.number_of_letters).to eq(4)
       expected: 4
            got: 0
       (compared using ==)
     # ./spec/services/ad_reminder_letters_export_service_spec.rb:47:in `block (5 levels) in <top (required)>'
     # ./spec/support/database_cleaner.rb:17:in `block (3 levels) in <top (required)>'
     # ./spec/support/database_cleaner.rb:16:in `block (2 levels) in <top (required)>'
  3) AdReminderLettersExportService.run when an error happens notify Airbrake
     Failure/Error: expect(AdReminderLettersBulkPdfService).to receive(:run).and_raise("An error")
       (AdReminderLettersBulkPdfService (class)).run(*(any args))
           expected: 1 time with any arguments
           received: 0 times with any arguments
     # ./spec/services/ad_reminder_letters_export_service_spec.rb:55:in `block (4 levels) in <top (required)>'
     # ./spec/support/database_cleaner.rb:17:in `block (3 levels) in <top (required)>'
     # ./spec/support/database_cleaner.rb:16:in `block (2 levels) in <top (required)>'
Finished in 38.19 seconds (files took 4.33 seconds to load)
1035 examples, 3 failures
Failed examples:
rspec ./spec/services/ad_reminder_letters_export_service_spec.rb:27 # AdReminderLettersExportService.run when there are assisted digital registrations that are due for renewal reminders loads a file to a AWS bucket and records the content created
rspec ./spec/services/ad_reminder_letters_export_service_spec.rb:37 # AdReminderLettersExportService.run when there are assisted digital registrations that are due for renewal reminders when one registration is in an invalid state and a PDF cannot be generated for it raises an error on Airbrake but continues generation for the other letters
rspec ./spec/services/ad_reminder_letters_export_service_spec.rb:54 # AdReminderLettersExportService.run when an error happens notify Airbrake