zef incorectly runs tests against (previosly installed) old module version

[codesections]

When running a command such as zef install Foo:ver<0.2>, zef runs Foo's tests. However, if the current repo-chain contains a Foo:ver<0.1> in an earlier Repository than the one Foo:ver<0.2> is being installed to, then Raku's normal rules for dependency resolution result in use Foo; being resolved to v0.1, not v0.2. This is very likely to cause Foo:ver<0.2>'s tests to fail.

Could Zef avoid this problem by somehow forcing the target repo to the front of the repo chain during testing?

Here's a reproducible example:

 $ RAKUDOLIB="inst#$HOME/v0.10,inst#$HOME/v0.17" zef install --to=$HOME/v0.10 'JSON::Fast:ver<0.10>'
===> Searching for: JSON::Fast:ver<0.10>
===> Testing: JSON::Fast:ver<0.10>
===> Testing [OK] for JSON::Fast:ver<0.10>
===> Installing: JSON::Fast:ver<0.10>

 $ RAKUDOLIB="inst#$HOME/v0.10,inst#$HOME/v0.17" zef install --to=$HOME/v0.17 'JSON::Fast:ver<0.17>'
===> Searching for: JSON::Fast:ver<0.17>
===> Testing: JSON::Fast:ver<0.17>:auth<cpan:TIMOTIMO>
[JSON::Fast] # Failed test at t/12-assocpositional.t line 25
[JSON::Fast] # expected: ${"1" => 1, "10" => 10, "2" => 2, "3" => 3, "4" => 4, "5" => 5, "6" => 6, "7" => 7, "8" => 8, "9" => 9}
[JSON::Fast] #      got: $[{"10" => 10}, {"9" => 9}, {"8" => 8}, {"7" => 7}, {"6" => 6}, {"5" => 5}, {"4" => 4}, {"3" => 3}, {"2" => 2}, {"1" => 1}]
[JSON::Fast] # Failed test at t/12-assocpositional.t line 25
[JSON::Fast] # expected: ${"1" => 1, "10" => 10, "2" => 2, "3" => 3, "4" => 4, "5" => 5, "6" => 6, "7" => 7, "8" => 8, "9" => 9}
[JSON::Fast] #      got: $[{"10" => 10}, {"9" => 9}, {"8" => 8}, {"7" => 7}, {"6" => 6}, {"5" => 5}, {"4" => 4}, {"3" => 3}, {"2" => 2}, {"1" => 1}]
[JSON::Fast] # Failed test at t/12-assocpositional.t line 25
[JSON::Fast] # expected: ${"1" => 1, "10" => 10, "2" => 2, "3" => 3, "4" => 4, "5" => 5, "6" => 6, "7" => 7, "8" => 8, "9" => 9}
[JSON::Fast] #      got: $[{"10" => 10}, {"9" => 9}, {"8" => 8}, {"7" => 7}, {"6" => 6}, {"5" => 5}, {"4" => 4}, {"3" => 3}, {"2" => 2}, {"1" => 1}]
[JSON::Fast] # Failed test at t/12-assocpositional.t line 25
[JSON::Fast] # expected: ${"1" => 1, "10" => 10, "2" => 2, "3" => 3, "4" => 4, "5" => 5, "6" => 6, "7" => 7, "8" => 8, "9" => 9}
[JSON::Fast] #      got: $[{"10" => 10}, {"9" => 9}, {"8" => 8}, {"7" => 7}, {"6" => 6}, {"5" => 5}, {"4" => 4}, {"3" => 3}, {"2" => 2}, {"1" => 1}]
[JSON::Fast] # You failed 4 tests of 4
[JSON::Fast] ===SORRY!=== Error while compiling /tmp/.zef/JSON%3A%3AFast%3Aver%3C0.17%3E%3Aauth%3Ccpan%3ATIMOTIMO%3E.tar.gz/JSON-Fast-0.17/t/13-scopes.t
[JSON::Fast] Error while importing from 'JSON::Fast':
[JSON::Fast] no EXPORT sub, but you provided positional argument in the 'use' statement
[JSON::Fast] at /tmp/.zef/JSON%3A%3AFast%3Aver%3C0.17%3E%3Aauth%3Ccpan%3ATIMOTIMO%3E.tar.gz/JSON-Fast-0.17/t/13-scopes.t:10
[JSON::Fast] ------>     use JSON::Fast <immutable !pretty>⏏;
===> Testing [FAIL]: JSON::Fast:ver<0.17>:auth<cpan:TIMOTIMO>
Aborting due to test failure: JSON::Fast:ver<0.17>:auth<cpan:TIMOTIMO> (use --force-test to override)

[ugexe]

It won’t help unless the versions being used are pinned — the same issue would just reveal itself after installation instead of before.

[ugexe]

Actually I also expect the repos zef sets for testing to already be in front of existing repositories… I wonder if it’s specific to using the rakulib env instead of e.g. -I

[ugexe]

$ RAKULIB=foo raku -Ibar -e '.say for $*REPO.repo-chain'
file#/Users/ugexe/bar
file#/Users/ugexe/foo
inst#/Users/ugexe/.raku
inst#/Users/ugexe/.rakubrew/versions/moar-2022.07/install/share/perl6/site
inst#/Users/ugexe/.rakubrew/versions/moar-2022.07/install/share/perl6/vendor
inst#/Users/ugexe/.rakubrew/versions/moar-2022.07/install/share/perl6/core
ap#
nqp#
perl5#

It does seem like the test libs (zef uses -I /path/to/repo) should actually be in front, hmmm

[ugexe]

Ah, JSON::Fast uses use lib "lib" in its tests, something that shouldn't be done. I would wager that is the cause

[codesections]

Ah, that explains it. Thanks for looking into it!