search

Test-More / Test2-Harness

Alternative to Test::Harness
Other
23 stars 26 forks source link

t/integration/replay.t fails: Could not unlink: No such file or directory at /[...]/Test2-Harness-1.000131/lib/App/Yath/Command/test.pm line 271, <__ANONIO__> line 21 during global destruction #256

Closed ppisar closed 2 years ago

ppisar commented 2 years ago

t/integration/replay.t in Test2-Harness-1.000131 run under the Test::Harness fails for me:

===(     629;26  5/?  4/?  2/?  2/?  0/? )==============================    # Failed test 'Replay has identical output to original'
    # at t/integration/replay.t line 57.
    # +-------------------------------+----+-------------------------------+
    # | GOT                           | OP | CHECK                         |
    # +-------------------------------+----+-------------------------------+
    # | [  FAIL  ]  job  1  + Fail\n  | eq | [  FAIL  ]  job  1  + Fail\n  |
    # | (  DIAG  )  job  1    Failed  |    | (  DIAG  )  job  1    Failed  |
    # | test 'Fail'\n                 |    | test 'Fail'\n                 |
    # | (  DIAG  )  job  1    at t/in |    | (  DIAG  )  job  1    at t/in |
    # | tegration/replay/fail.tx line |    | tegration/replay/fail.tx line |
    # |  4.\n                         |    |  4.\n                         |
    # | (  DIAG  )  job  1    Seeded  |    | (  DIAG  )  job  1    Seeded  |
    # | srand with seed '20220907' fr |    | srand with seed '20220907' fr |
    # | om local date.\n              |    | om local date.\n              |
    # | ( PASSED )  job  2    t/integ |    | ( PASSED )  job  2    t/integ |
    # | ration/replay/pass.tx\n       |    | ration/replay/pass.tx\n       |
    # | ( FAILED )  job  1    t/integ |    | ( FAILED )  job  1    t/integ |
    # | ration/replay/fail.tx\n       |    | ration/replay/fail.tx\n       |
    # | < REASON >  job  1    Test sc |    | < REASON >  job  1    Test sc |
    # | ript returned error (Err: 1)\ |    | ript returned error (Err: 1)\ |
    # | n                             |    | n                             |
    # | < REASON >  job  1    Asserti |    | < REASON >  job  1    Asserti |
    # | on failures were encountered  |    | on failures were encountered  |
    # | (Count: 1)\n                  |    | (Count: 1)\n                  |
    # | The following jobs failed:\n  |    | The following jobs failed:\n  |
    # | +---------------------------- |    | +---------------------------- |
    # | ----------+------------------ |    | ----------+------------------ |
    # | ------------+\n               |    | ------------+\n               |
    # | | Job ID                      |    | | Job ID                      |
    # |           | Test File         |    |           | Test File         |
    # |             |\n               |    |             |\n               |
    # | +---------------------------- |    | +---------------------------- |
    # | ----------+------------------ |    | ----------+------------------ |
    # | ------------+\n               |    | ------------+\n               |
    # | | AD1B6644-2E83-11ED-AA61-A48 |    | | AD1B6644-2E83-11ED-AA61-A48 |
    # | 93E685913 | t/integration/rep |    | 93E685913 | t/integration/rep |
    # | lay/fail.tx |\n               |    | lay/fail.tx |\n               |
    # | +---------------------------- |    | +---------------------------- |
    # | ----------+------------------ |    | ----------+------------------ |
    # | ------------+\n               |    | ------------+\n               |
    # | Yath Result Summary\n         |    | Yath Result Summary\n         |
    # | Fail Count: 1\n               |    | Fail Count: 1\n               |
    # | File Count: 2\n               |    | File Count: 2\n               |
    # | Assertion Count: 2\n          |    | Assertion Count: 2\n          |
    # | -->  Result: FAILED  <--\n    |    | -->  Result: FAILED  <--\n    |
    # | \e[0m\n                       |    | \e[0m\n                       |
    # |                               |    | Could not unlink: No such fil |
    # |                               |    | e or directory at /home/test/ |
    # |                               |    | rpmbuild/BUILD/Test2-Harness- |
    # |                               |    | 1.000131/lib/App/Yath/Command |
    # |                               |    | /test.pm line 271, <__ANONIO_ |
    # |                               |    | _> line 21 during global dest |
    # |                               |    | ruction.\n                    |
    # +-------------------------------+----+-------------------------------+
    # Command = /usr/bin/perl /home/test/rpmbuild/BUILD/Test2-Harness-1.000131/scripts/yath -D/home/test/rpmbuild/BUILD/Test2-Harness-1.000131/blib/lib replay /tmp/yathlog-35828-7tJNJBmR.jsonl
    # Exit = 256
    # ==== Output ====
    # [  FAIL  ]  job  1  + Fail
    # (  DIAG  )  job  1    Failed test 'Fail'
    # (  DIAG  )  job  1    at t/integration/replay/fail.tx line 4.
    # (  DIAG  )  job  1    Seeded srand with seed '20220907' from local date.
    # ( PASSED )  job  2    t/integration/replay/pass.tx
    # ( FAILED )  job  1    t/integration/replay/fail.tx
    # < REASON >  job  1    Test script returned error (Err: 1)
    # < REASON >  job  1    Assertion failures were encountered (Count: 1)
    # The following jobs failed:
    # +--------------------------------------+------------------------------+
    # | Job ID                               | Test File                    |
    # +--------------------------------------+------------------------------+
    # | AD1B6644-2E83-11ED-AA61-A4893E685913 | t/integration/replay/fail.tx |
    # +--------------------------------------+------------------------------+
    # Yath Result Summary
    # Fail Count: 1
    # File Count: 2
    # Assertion Count: 2
    # -->  Result: FAILED  <--
    #
    #
    # ========

# Failed test 'yath replay /tmp/yathl[...]JBmR.jsonl'
# at t/integration/replay.t line 59.
# Seeded srand with seed '20220907' from local date.
t/integration/replay.t ................... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests

Maybe because I run the tests in parallel (I set HARNESS_OPTIONS and T2_HARNESS_JOB_COUNT environment variables) and the ./.test_info.json is deleted by a concurrent job.

ppisar commented 2 years ago

This is a full reproducer:

AUTOMATED_TESTING=1 HARNESS_OPTIONS=j5 T2_HARNESS_JOB_COUNT=1 make test

Unsetting AUTOMATED_TESTING or HARNESS_OPTIONS disappears the failure.

ppisar commented 2 years ago

A smaller reproducer is "AUTOMATED_TESTING=1 prove -l -j5 -r t/integration".

ppisar commented 2 years ago

Now I got a different failure pertaining ./.test_info.json:

    # Failed test 'Exit Value Check'
    # at t/integration/coverage.t line 344.
    # +-----+----+-------+
    # | GOT | OP | CHECK |
    # +-----+----+-------+
    # | 512 | eq | 0     |
    # +-----+----+-------+
    # Failed test 'Test got the correct input about what subtests to run'
    # at t/integration/coverage.t line 323.
    # +-------------------------------+------------------+--------+
    # | PATH                          | GOT              | CHECK  |
    # +-------------------------------+------------------+--------+
    # | {t/integration/coverage/a.tx} | <DOES NOT EXIST> | <HASH> |
    # | {t/integration/coverage/c.tx} | <DOES NOT EXIST> | <HASH> |
    # +-------------------------------+------------------+--------+
    # Failed test 'Skipped mia1 because it does not exist'
    # at t/integration/coverage.t line 334.
    # +-------------------------------+----+-------------------------------+
    # | GOT                           | OP | CHECK                         |
    # +-------------------------------+----+-------------------------------+
    # | ./.test_info.json.pend -> ./. | =~ | (?^:Coverage wants to run tes |
    # | test_info.json: No such file  |    | t 't/integration/coverage/mia |
    # | or directory at /builddir/bui |    | 1\.tx', but it does not exist |
    # | ld/BUILD/Test2-Harness-1.0001 |    | , skipping\.\.\.)             |
    # | 31/lib/Test2/Harness/Util.pm  |    |                               |
    # | line 240.\n                   |    |                               |
    # +-------------------------------+----+-------------------------------+
    # Failed test 'Skipped mia2 because it does not exist'
    # at t/integration/coverage.t line 340.
    # +-------------------------------+----+-------------------------------+
    # | GOT                           | OP | CHECK                         |
    # +-------------------------------+----+-------------------------------+
    # | ./.test_info.json.pend -> ./. | =~ | (?^:Coverage wants to run tes |
    # | test_info.json: No such file  |    | t 't/integration/coverage/mia |
    # | or directory at /builddir/bui |    | 2\.tx', but it does not exist |
    # | ld/BUILD/Test2-Harness-1.0001 |    | , skipping\.\.\.)             |
    # | 31/lib/Test2/Harness/Util.pm  |    |                               |
    # | line 240.\n                   |    |                               |
    # +-------------------------------+----+-------------------------------+
    # Command = /usr/bin/perl /builddir/build/BUILD/Test2-Harness-1.000131/scripts/yath -Dt/integration/coverage/lib -D/builddir/build/BUILD/Test2-Harness-1.000131/blib/lib test -I/builddir/build/BUILD/Test2-Harness-1.000131/blib/lib -Dt/integration/coverage/lib -It/integration/coverage/lib --ext=tx --cover-from=/tmp/JNJBmRJDRN.json --plugin +Plugin --changed-only -v
    # Exit = 512
    # ==== Output ====
    # ./.test_info.json.pend -> ./.test_info.json: No such file or directory at /builddir/build/BUILD/Test2-Harness-1.000131/lib/Test2/Harness/Util.pm line 240.
    # 
    # ========
# Failed test 'yath test -Dt/integration/coverage/lib -It/integration/coverage/lib --ext=tx --cover-fr[...]RJDRN.json --plugin +Plugin --changed-only -v'
# at t/integration/coverage.t line 344.

It seems that cleaning up that JSON file has more consequences.