search

cachix / devenv

Fast, Declarative, Reproducible, and Composable Developer Environments
https://devenv.sh
Apache License 2.0
3.61k stars 262 forks source link

devenv test reporting success even when errors and failures occur #1199

Open kourtni opened 2 weeks ago

kourtni commented 2 weeks ago

Describe the bug When running devenv test, failures and errors can get swallowed resulting in devenv test reporting that "Tests passed".

devenv_test_errors

To reproduce This gist, https://gist.github.com/kourtni/7ee2749b0bb75253e0d29d2deab1b4fb, is a complete example that reproduces the issue. I expect that running devenv test from that environment should fail, however it currently passes.

Version devenv 1.0.5 (x86_64-linux)

domenkozar commented 1 week ago

Could you also attach the python files so I can reproduce the failure?

kourtni commented 1 week ago

Hi,

The issue is that devenv test swallows errors and test failures if enterTest ends with a command that exits normally.

The screenshot is from tests run in a private repo, so I can't share those files. But as I indicated in the Gist, https://gist.github.com/kourtni/7ee2749b0bb75253e0d29d2deab1b4fb, this behavior is completely reproducible without actually requiring any python files. In fact the point of that Gist is to show that devenv test passes even without ANY of the required packages installed or source files available. I am able to reproduce the issue with only the devenv.nix and devenv.yaml files included in the Gist, and then simply running devenv test.

But if you absolutely want to include a dummy test for some reason, you could create a file with a simple test where you assert 1 == 2 or something that will obviously fail.

As long as the final command in enterTest is an echo or some other command guaranteed to succeed, then any failures that occurred prior will be swallowed and the devenv test will complete in a Tests passed state.