Occasionally the unit tests for the AdReminderLettersExportService or the DigitalReminderLettersExportService will fail. However, this only happens on Travis – when the tests are run locally with the same seed, it's fine.
It's unclear what the cause of this is, but possible guesses include:
inadequate or failed database cleaning
version differences between local environments and Travis (local MongoDb is 3.6.3 and Travis is 3.4.10)
factories not generating test data in time
The seed 13731 reliably triggers the AdReminderLettersExportService specs to fail, so the order of the tests may have something to do with it.
Failures:
1) AdReminderLettersExportService.run when there are assisted digital registrations that are due for renewal reminders loads a file to a AWS bucket and records the content created
Failure/Error: Airbrake.notify e, file_name: file_name
(Airbrake).notify(#<Errno::ENOENT: No such file or directory @ rb_sysopen - /home/travis/build/DEFRA/waste-carriers-back-office/tmp/ad_reminder_letters_20201025.pdf>, {:file_name=>"ad_reminder_letters_20201025.pdf"})
expected: 0 times with any arguments
received: 1 time with arguments: (#<Errno::ENOENT: No such file or directory @ rb_sysopen - /home/travis/build/DEFRA/waste-carriers-back-office/tmp/ad_reminder_letters_20201025.pdf>, {:file_name=>"ad_reminder_letters_20201025.pdf"})
# ./app/services/reminder_letters_export_service.rb:21:in `rescue in run'
# ./app/services/reminder_letters_export_service.rb:20:in `run'
# ./spec/services/ad_reminder_letters_export_service_spec.rb:30:in `block (4 levels) in <top (required)>'
# ./spec/support/database_cleaner.rb:17:in `block (3 levels) in <top (required)>'
# ./spec/support/database_cleaner.rb:16:in `block (2 levels) in <top (required)>'
# ------------------
# --- Caused by: ---
# Errno::ENOENT:
# No such file or directory @ rb_sysopen - /home/travis/build/DEFRA/waste-carriers-back-office/tmp/ad_reminder_letters_20201025.pdf
# ./app/services/reminder_letters_export_service.rb:12:in `initialize'
2) AdReminderLettersExportService.run when there are assisted digital registrations that are due for renewal reminders when one registration is in an invalid state and a PDF cannot be generated for it raises an error on Airbrake but continues generation for the other letters
Failure/Error: expect(ad_reminder_letters_export.number_of_letters).to eq(4)
expected: 4
got: 0
(compared using ==)
# ./spec/services/ad_reminder_letters_export_service_spec.rb:47:in `block (5 levels) in <top (required)>'
# ./spec/support/database_cleaner.rb:17:in `block (3 levels) in <top (required)>'
# ./spec/support/database_cleaner.rb:16:in `block (2 levels) in <top (required)>'
3) AdReminderLettersExportService.run when an error happens notify Airbrake
Failure/Error: expect(AdReminderLettersBulkPdfService).to receive(:run).and_raise("An error")
(AdReminderLettersBulkPdfService (class)).run(*(any args))
expected: 1 time with any arguments
received: 0 times with any arguments
# ./spec/services/ad_reminder_letters_export_service_spec.rb:55:in `block (4 levels) in <top (required)>'
# ./spec/support/database_cleaner.rb:17:in `block (3 levels) in <top (required)>'
# ./spec/support/database_cleaner.rb:16:in `block (2 levels) in <top (required)>'
Finished in 38.19 seconds (files took 4.33 seconds to load)
1035 examples, 3 failures
Failed examples:
rspec ./spec/services/ad_reminder_letters_export_service_spec.rb:27 # AdReminderLettersExportService.run when there are assisted digital registrations that are due for renewal reminders loads a file to a AWS bucket and records the content created
rspec ./spec/services/ad_reminder_letters_export_service_spec.rb:37 # AdReminderLettersExportService.run when there are assisted digital registrations that are due for renewal reminders when one registration is in an invalid state and a PDF cannot be generated for it raises an error on Airbrake but continues generation for the other letters
rspec ./spec/services/ad_reminder_letters_export_service_spec.rb:54 # AdReminderLettersExportService.run when an error happens notify Airbrake
Occasionally the unit tests for the AdReminderLettersExportService or the DigitalReminderLettersExportService will fail. However, this only happens on Travis – when the tests are run locally with the same seed, it's fine.
It's unclear what the cause of this is, but possible guesses include:
The seed
13731
reliably triggers the AdReminderLettersExportService specs to fail, so the order of the tests may have something to do with it.Some previous investigation was done here: https://github.com/DEFRA/waste-carriers-back-office/pull/974/files
Here is an example of a broken build: https://travis-ci.com/github/DEFRA/waste-carriers-back-office/builds/177306791
With the failures looking like: