request: Omit no test failure and should pass --fail-fast

[Tokenyet]

Is there an existing feature request for this?

• [X] I have searched the existing issues.

Command

--fail-fast 1

Description

I have used melos to test with --tags arguments. With 20+ private packages, some packages used certain tags, but some not. That would cause melos FAILED on packages which don't have the specified tag. For me, It looks wrong since It's not really a failure, but a empty test run. Maybe we should have an exception for no tests?

ex. melos exec -c 1 --fail-fast 1 -- \ flutter test --tags sometags

--------------------------------------------------------------------------------
console_repository:
00:00 +0: loading D:\Project\McDedicatedServer\minecraft_cube_desktop\minecraft_cube_desktop\packages\console_repository\test\console_reposit00:01 +0: loading D:\Project\McDedicatedServer\minecraft_cube_desktop\minecraft_cube_desktop\packages\console_repository\test\console_repositNo tests ran.                                                                                                                                

No tests match the requested tag selectors:
  include: "integration"
  exclude: "<none>"

--------------------------------------------------------------------------------

$ melos exec
   └> flutter test -j1 --no-pub --tags integration
       └> FAILED (in 1 packages)
           └> console_repository (with exit code 1)
pub finished with exit code 1

Reasoning

For the melos users who work with the arguments --tags, --excluded-tags, or another filter arguments that might cause no test.

Additional context and comments

Here is the exitCode list from dart test, not sure If It's matched, since the logs above shows with exit code 1.

[Tokenyet]

Share the research for the current limitation of flutter.

https://github.com/flutter/flutter/issues/100467

[blaugold]

I don't think I understand how the --fail-fast option would work. Wouldn't an option that allows you to specify a list of exit codes that should be interpreted as success be enough?

You could also just run melos exec -- "flutter test --tags sometags || [ $? = 79 ]", where 79 is the exit code for when no tests are found.

In any case, as you mentioned, the limitation in flutter needs to be resolved for this to work.

[sagar-tide]

I am in the same scenario. I have a package with only one test that should only run on CI as a separate step. I get the following output on my CI and then the final step fails:

[my_service]: ::group::❎ (suite) (skipped)
[my_service]: Skip: Should only be run during pull request
[my_service]: ::endgroup::
[my_service]: 
[my_service]: 🎉 0 tests passed, 1 skipped.

output:

melos exec
  └> flutter test --coverage --exclude-tags=golden
     └> FAILED (in 1 packages)
        └> my_service (with exit code 1)
Error: Process completed with exit code 1.

[spydon]

I don't think we should proxy the flutter test result codes and output another exit code, this should be done outside of melos, in your CI workflow in this case.

[gabsn]

same issue there, really annoying that melos fails on empty tests

[spydon]

same issue there, really annoying that melos fails on empty tests

It really should be solved in Flutter as Tokenyet showed in https://github.com/flutter/flutter/issues/100467, but maybe we could have a slightly dirtier workaround meanwhile and interpret exit code 79 (noTestRan) as 0 instead.

If anyone is keen on working on a PR doing that, feel free!