search

DEFRA / sroc-service-team

Guides, info and issue management for the Charging Module Team
Other
0 stars 1 forks source link

Flaky unit tests #7

Open Cruikshanks opened 4 years ago

Cruikshanks commented 4 years ago

The TCM appears to have some flaky unit tests. This is most obvious in our Travis build environment where you might see the PR build pass but the branch build fail (it can be the other way round). A simple restart of the job and the tests seem to all pass again.

We'll try to capture examples as and when we can and record them here. We also intend t start migrating the unit tests to RSpec which is our standard framework at Defra and used in all our other Rails-based projects. So there is a chance that work might also inadvertently resolve the issue.

N.B. Add a thumbs-up if a test is already logged but you encounter another example of it failing

Cruikshanks commented 4 years ago

Flaky test example

 FAIL["test_can_select_all_region_option_on_retrospective_view", Minitest::Result, 8.102071347999996]
 test_can_select_all_region_option_on_retrospective_view#Minitest::Result (8.10s)
        'B' option not selected
        test/integration/region_selector_test.rb:70:in `block (2 levels) in test_can_select_all_region_option_on_retrospective_view'
Cruikshanks commented 4 years ago

Flaky test example

FAIL["test_can_select_all_region_option_on_exclusions_view", #<Minitest::Reporters::Suite:0x00000006ebb670 @name="RegionSelectorTest">, 25.065493564000008]
 test_can_select_all_region_option_on_exclusions_view#RegionSelectorTest (25.07s)
        'B' option not selected
        test/integration/region_selector_test.rb:107:in `block (2 levels) in test_can_select_all_region_option_on_exclusions_view'
StuAA78 commented 4 years ago

Another flaky test example

 FAIL["test_can_select_all_region_option_on_exclusions_view", #<Minitest::Reporters::Suite:0x007fe396bafed8 @name="RegionSelectorTest">, 35.737085000029765]
 test_can_select_all_region_option_on_exclusions_view#RegionSelectorTest (35.74s)
        All option not selected
        test/integration/region_selector_test.rb:120:in `block in test_can_select_all_region_option_on_exclusions_view'
Cruikshanks commented 4 years ago

test/integration/region_selector_test.rb:70 and test/integration/region_selector_test.rb:107 and starting to fail pretty consistently now (plus I have seen test/integration/region_selector_test.rb:120 as well.

StuAA78 commented 4 years ago 
$ rails test --seed 58544
Started with run options --seed 58544

ERROR["test_it_produces_detail_records", #<Minitest::Reporters::Suite:0x007ff7e00cc4b8 @name="CfdTransactionFilePresenterTest">, 2.917215000023134]
 test_it_produces_detail_records#CfdTransactionFilePresenterTest (2.92s)
Minitest::UnexpectedError:         ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "id" violates not-null constraint
        DETAIL:  Failing row contains (null, 212387086, 1, A60425822C, 2020-07-11 08:28:11, I, 5MQ4AOPP, , GBP, , 20-AUG-2017, , , , , , , , , , 23747, , 3, Consent No - ANQA/1234/1/1, C, D, Green Rd. Pig Disposal, STORM SEWAGE OVERFLOW, 01/04/17 - 10/08/17, 365/132, C 1, E 1, S 1, 684, 96%, , , , , , , 1, Each, 23747, 2020-09-11 08:28:12.037551, 2020-09-11 08:28:14.43389, unbilled, null, ANQA/1234/1/1, 1, 1, null, null, f, null, null, null, null, 2017-04-01 00:00:00, 2017-08-10 00:00:00, 275617942, null, null, null, null, CFDBI00123, 2020-08-28 00:00:00, 1718, f, null, null, f, null, null, null, A).
        : UPDATE "transaction_details" SET "id" = $1, "updated_at" = $2, "transaction_file_id" = $3 WHERE "transaction_details"."id" = $4
            test/presenters/cfd_transaction_file_presenter_test.rb:25:in `setup'

ERROR["test_detail_record_consent_number_not_prefixed", #<Minitest::Reporters::Suite:0x007ff7e149f760 @name="CfdTransactionFilePresenterTest">, 2.9615709999925457]
 test_detail_record_consent_number_not_prefixed#CfdTransactionFilePresenterTest (2.96s)
Minitest::UnexpectedError:         ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "id" violates not-null constraint
        DETAIL:  Failing row contains (null, 212387086, 1, A60425822C, 2020-07-11 08:28:11, I, 5MQ4AOPP, , GBP, , 20-AUG-2017, , , , , , , , , , 23747, , 3, Consent No - ANQA/1234/1/1, C, D, Green Rd. Pig Disposal, STORM SEWAGE OVERFLOW, 01/04/17 - 10/08/17, 365/132, C 1, E 1, S 1, 684, 96%, , , , , , , 1, Each, 23747, 2020-09-11 08:28:12.037551, 2020-09-11 08:28:14.482743, unbilled, null, ANQA/1234/1/1, 1, 1, null, null, f, null, null, null, null, 2017-04-01 00:00:00, 2017-08-10 00:00:00, 275617942, null, null, null, null, CFDBI00123, 2020-08-28 00:00:00, 1718, f, null, null, f, null, null, null, A).
        : UPDATE "transaction_details" SET "id" = $1, "updated_at" = $2, "transaction_file_id" = $3 WHERE "transaction_details"."id" = $4
            test/presenters/cfd_transaction_file_presenter_test.rb:25:in `setup'

ERROR["test_it_returns_a_header_record", #<Minitest::Reporters::Suite:0x007ff7dfbfec88 @name="CfdTransactionFilePresenterTest">, 3.034992000029888]
 test_it_returns_a_header_record#CfdTransactionFilePresenterTest (3.04s)
Minitest::UnexpectedError:         ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "id" violates not-null constraint
        DETAIL:  Failing row contains (null, 212387086, 1, A60425822C, 2020-07-11 08:28:11, I, 5MQ4AOPP, , GBP, , 20-AUG-2017, , , , , , , , , , 23747, , 3, Consent No - ANQA/1234/1/1, C, D, Green Rd. Pig Disposal, STORM SEWAGE OVERFLOW, 01/04/17 - 10/08/17, 365/132, C 1, E 1, S 1, 684, 96%, , , , , , , 1, Each, 23747, 2020-09-11 08:28:12.037551, 2020-09-11 08:28:14.5536, unbilled, null, ANQA/1234/1/1, 1, 1, null, null, f, null, null, null, null, 2017-04-01 00:00:00, 2017-08-10 00:00:00, 275617942, null, null, null, null, CFDBI00123, 2020-08-28 00:00:00, 1718, f, null, null, f, null, null, null, A).
        : UPDATE "transaction_details" SET "id" = $1, "updated_at" = $2, "transaction_file_id" = $3 WHERE "transaction_details"."id" = $4
            test/presenters/cfd_transaction_file_presenter_test.rb:25:in `setup'

ERROR["test_detail_records_have_correct_temporary_cessation_value", #<Minitest::Reporters::Suite:0x007ff7e44066e8 @name="CfdTransactionFilePresenterTest">, 3.1157100000418723]
 test_detail_records_have_correct_temporary_cessation_value#CfdTransactionFilePresenterTest (3.12s)
Minitest::UnexpectedError:         ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "id" violates not-null constraint
        DETAIL:  Failing row contains (null, 212387086, 1, A60425822C, 2020-07-11 08:28:11, I, 5MQ4AOPP, , GBP, , 20-AUG-2017, , , , , , , , , , 23747, , 3, Consent No - ANQA/1234/1/1, C, D, Green Rd. Pig Disposal, STORM SEWAGE OVERFLOW, 01/04/17 - 10/08/17, 365/132, C 1, E 1, S 1, 684, 96%, , , , , , , 1, Each, 23747, 2020-09-11 08:28:12.037551, 2020-09-11 08:28:14.634847, unbilled, null, ANQA/1234/1/1, 1, 1, null, null, f, null, null, null, null, 2017-04-01 00:00:00, 2017-08-10 00:00:00, 275617942, null, null, null, null, CFDBI00123, 2020-08-28 00:00:00, 1718, f, null, null, f, null, null, null, A).
        : UPDATE "transaction_details" SET "id" = $1, "updated_at" = $2, "transaction_file_id" = $3 WHERE "transaction_details"."id" = $4
            test/presenters/cfd_transaction_file_presenter_test.rb:25:in `setup'

ERROR["test_is_returns_a_trailer_record", #<Minitest::Reporters::Suite:0x007ff7e28a8d38 @name="CfdTransactionFilePresenterTest">, 3.1640100000076927]
 test_is_returns_a_trailer_record#CfdTransactionFilePresenterTest (3.16s)
Minitest::UnexpectedError:         ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "id" violates not-null constraint
        DETAIL:  Failing row contains (null, 212387086, 1, A60425822C, 2020-07-11 08:28:11, I, 5MQ4AOPP, , GBP, , 20-AUG-2017, , , , , , , , , , 23747, , 3, Consent No - ANQA/1234/1/1, C, D, Green Rd. Pig Disposal, STORM SEWAGE OVERFLOW, 01/04/17 - 10/08/17, 365/132, C 1, E 1, S 1, 684, 96%, , , , , , , 1, Each, 23747, 2020-09-11 08:28:12.037551, 2020-09-11 08:28:14.68476, unbilled, null, ANQA/1234/1/1, 1, 1, null, null, f, null, null, null, null, 2017-04-01 00:00:00, 2017-08-10 00:00:00, 275617942, null, null, null, null, CFDBI00123, 2020-08-28 00:00:00, 1718, f, null, null, f, null, null, null, A).
        : UPDATE "transaction_details" SET "id" = $1, "updated_at" = $2, "transaction_file_id" = $3 WHERE "transaction_details"."id" = $4
            test/presenters/cfd_transaction_file_presenter_test.rb:25:in `setup'

ERROR["test_detail_record_has_correct_category_description", #<Minitest::Reporters::Suite:0x007ff7e13c4138 @name="CfdTransactionFilePresenterTest">, 3.2169340000255033]
 test_detail_record_has_correct_category_description#CfdTransactionFilePresenterTest (3.22s)
Minitest::UnexpectedError:         ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "id" violates not-null constraint
        DETAIL:  Failing row contains (null, 212387086, 1, A60425822C, 2020-07-11 08:28:11, I, 5MQ4AOPP, , GBP, , 20-AUG-2017, , , , , , , , , , 23747, , 3, Consent No - ANQA/1234/1/1, C, D, Green Rd. Pig Disposal, STORM SEWAGE OVERFLOW, 01/04/17 - 10/08/17, 365/132, C 1, E 1, S 1, 684, 96%, , , , , , , 1, Each, 23747, 2020-09-11 08:28:12.037551, 2020-09-11 08:28:14.735405, unbilled, null, ANQA/1234/1/1, 1, 1, null, null, f, null, null, null, null, 2017-04-01 00:00:00, 2017-08-10 00:00:00, 275617942, null, null, null, null, CFDBI00123, 2020-08-28 00:00:00, 1718, f, null, null, f, null, null, null, A).
        : UPDATE "transaction_details" SET "id" = $1, "updated_at" = $2, "transaction_file_id" = $3 WHERE "transaction_details"."id" = $4
            test/presenters/cfd_transaction_file_presenter_test.rb:25:in `setup'

ERROR["test_detail_record_variation_is_blank_when_100_percent", #<Minitest::Reporters::Suite:0x007ff7e1bf1fe0 @name="CfdTransactionFilePresenterTest">, 3.2638520000036806]
 test_detail_record_variation_is_blank_when_100_percent#CfdTransactionFilePresenterTest (3.26s)
Minitest::UnexpectedError:         ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "id" violates not-null constraint
        DETAIL:  Failing row contains (null, 212387086, 1, A60425822C, 2020-07-11 08:28:11, I, 5MQ4AOPP, , GBP, , 20-AUG-2017, , , , , , , , , , 23747, , 3, Consent No - ANQA/1234/1/1, C, D, Green Rd. Pig Disposal, STORM SEWAGE OVERFLOW, 01/04/17 - 10/08/17, 365/132, C 1, E 1, S 1, 684, 96%, , , , , , , 1, Each, 23747, 2020-09-11 08:28:12.037551, 2020-09-11 08:28:14.785067, unbilled, null, ANQA/1234/1/1, 1, 1, null, null, f, null, null, null, null, 2017-04-01 00:00:00, 2017-08-10 00:00:00, 275617942, null, null, null, null, CFDBI00123, 2020-08-28 00:00:00, 1718, f, null, null, f, null, null, null, A).
        : UPDATE "transaction_details" SET "id" = $1, "updated_at" = $2, "transaction_file_id" = $3 WHERE "transaction_details"."id" = $4
            test/presenters/cfd_transaction_file_presenter_test.rb:25:in `setup'

ERROR["test_detail_records_have_correct_line_attr_4_pro_rata_days", #<Minitest::Reporters::Suite:0x007ff7e1ba1298 @name="CfdTransactionFilePresenterTest">, 3.3075320000061765]
 test_detail_records_have_correct_line_attr_4_pro_rata_days#CfdTransactionFilePresenterTest (3.31s)
Minitest::UnexpectedError:         ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "id" violates not-null constraint
        DETAIL:  Failing row contains (null, 212387086, 1, A60425822C, 2020-07-11 08:28:11, I, 5MQ4AOPP, , GBP, , 20-AUG-2017, , , , , , , , , , 23747, , 3, Consent No - ANQA/1234/1/1, C, D, Green Rd. Pig Disposal, STORM SEWAGE OVERFLOW, 01/04/17 - 10/08/17, 365/132, C 1, E 1, S 1, 684, 96%, , , , , , , 1, Each, 23747, 2020-09-11 08:28:12.037551, 2020-09-11 08:28:14.828523, unbilled, null, ANQA/1234/1/1, 1, 1, null, null, f, null, null, null, null, 2017-04-01 00:00:00, 2017-08-10 00:00:00, 275617942, null, null, null, null, CFDBI00123, 2020-08-28 00:00:00, 1718, f, null, null, f, null, null, null, A).
        : UPDATE "transaction_details" SET "id" = $1, "updated_at" = $2, "transaction_file_id" = $3 WHERE "transaction_details"."id" = $4
            test/presenters/cfd_transaction_file_presenter_test.rb:25:in `setup'

ERROR["test_detail_record_has_correct_transaction_date", #<Minitest::Reporters::Suite:0x007ff7de43a728 @name="CfdTransactionFilePresenterTest">, 3.346118000044953]
 test_detail_record_has_correct_transaction_date#CfdTransactionFilePresenterTest (3.35s)
Minitest::UnexpectedError:         ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  null value in column "id" violates not-null constraint
        DETAIL:  Failing row contains (null, 212387086, 1, A60425822C, 2020-07-11 08:28:11, I, 5MQ4AOPP, , GBP, , 20-AUG-2017, , , , , , , , , , 23747, , 3, Consent No - ANQA/1234/1/1, C, D, Green Rd. Pig Disposal, STORM SEWAGE OVERFLOW, 01/04/17 - 10/08/17, 365/132, C 1, E 1, S 1, 684, 96%, , , , , , , 1, Each, 23747, 2020-09-11 08:28:12.037551, 2020-09-11 08:28:14.867033, unbilled, null, ANQA/1234/1/1, 1, 1, null, null, f, null, null, null, null, 2017-04-01 00:00:00, 2017-08-10 00:00:00, 275617942, null, null, null, null, CFDBI00123, 2020-08-28 00:00:00, 1718, f, null, null, f, null, null, null, A).
        : UPDATE "transaction_details" SET "id" = $1, "updated_at" = $2, "transaction_file_id" = $3 WHERE "transaction_details"."id" = $4
            test/presenters/cfd_transaction_file_presenter_test.rb:25:in `setup'
Cruikshanks commented 4 years ago

@StuAA78 has had a cracking idea. For now, until we can sort out the test suite we should look to see which seeds pass successfully on Travis. We think it might be connected to the order tests are running.

So though a good test suite should be able to all pass irrespective of what order the tests run, for now, we can get things building consistently (hopefully!) by using a known seed.

Examples

We'll need to double-check how we tell Minitest to use a specific seed and then update the .travis.yml with the new command.

StuAA78 commented 4 years ago

rails test --seed 20924 works (locally at least) and I've confirmed that it runs all the tests in the exact same order each time whereas passing in a different seed runs the test in a different order. Unfortunately the tests are still flaky -- running it 5 times consecutively gave the following results:

446 tests, 120571 assertions, 1 failures, 0 errors, 0 skips
446 tests, 120567 assertions, 2 failures, 0 errors, 0 skips
446 tests, 120571 assertions, 0 failures, 0 errors, 0 skips
446 tests, 120571 assertions, 0 failures, 0 errors, 0 skips
446 tests, 120570 assertions, 1 failures, 0 errors, 0 skips

(I assume the differing number of assertions is down to which tests fail and where)

Cruikshanks commented 4 years ago

weary

StuAA78 commented 4 years ago

Above tests all fixed https://github.com/DEFRA/sroc-tcm-admin/pull/293

StuAA78 commented 4 years ago 
Started with run options --seed 6901
...
 FAIL["test_sort_column_retained_after_exclusion", #<Minitest::Reporters::Suite:0x00000008553770 @name="SearchCriteriaPersistenceTest">, 32.694720970999995]
 test_sort_column_retained_after_exclusion#SearchCriteriaPersistenceTest (32.69s)
        Expected false to be truthy.
        test/integration/search_criteria_persistence_test.rb:30:in `test_sort_column_retained_after_exclusion'

Failed when Travis was running the tests, unable to reproduce locally as yet.

Cruikshanks commented 4 years ago 
Started with run options --seed 29312
...
FAIL["test_style_remains_on_transactions_to_be_billed", #<Minitest::Reporters::Suite:0x00000008bbe858 @name="ExcludedTransactionsTest">, 7.725703580000001]
 test_style_remains_on_transactions_to_be_billed#ExcludedTransactionsTest (7.73s)
        Expected false to be truthy.
        test/integration/excluded_transactions_test.rb:33:in `test_style_remains_on_transactions_to_be_billed'