codecov.exe returns success on invalid arguments

[marcomorain]

👋 Hi there, thanks for the codecov tool!

Actual

I invoked the codecov.exe tool incorrectly on Windows:

codecov -f coverage.txt -F windows

I was using -F from the bash uploader, rather than --flag, which print the usage information, but the exit code is 0 (success)

This caused my CircleCI step to pass:

Even though the output showed the error:

codecov 1.3.0
Copyright (c) 2017-Present Larz White, Kim J. Nordmo
ERROR(S):
Option 'F' is unknown.
<snip the full usage info>

Full example:

 codecov -f coverage.txt -F windows ; echo $LASTEXITCODE
codecov 1.1.0
Copyright (c) 2017-Present Larz White, Kim J. Nordmo
ERROR(S):
Option 'F' is unknown.

  --branch             Specify the branch name.

  -b, --build          Specify the build number.

  -c, --sha            Specify the commit sha.

  --disable-network    Toggle functionalities. (1) --disable-network. Disable uploading the file network.

  -d, --dump           Don't upload and dump to stdin.

  -e, --env            Specify enviornment variables to be included with this build. (1) CODECOV_ENV=VAR1,VAR2. (2) -e VAR1
                       VAR2.

  -f, --file           Target file(s) to upload. (1) -f 'path/to/file'. Only upload this file. (2) -f 'path/to/file1
                       path/to/file2'. Only upload these files.

  --flag               Flag the upload to group coverage metrics. (1) --flag unittests. This upload is only unittests. (2)
                       --flag integration. This upload is only integration tests. (3) --flag ut,chrome. This upload is chrome
                       - UI tests.

  -n, --name           Custom defined name of the upload. Visible in Codecov UI.

  --no-color           Remove color from the output.

  --pr                 Specify the pull request number.

  --root               Used when not in git/hg project to identify project root directory.

  --required           Exit with 1 if not successful. Default will Exit with 0.

  -r, --slug           owner/repo slug used instead of the private repo token in Enterprise. (option) Set environment
                       variable CODECOV_SLUG=:owner/:repo.

  --tag                Specify the git tag.

  -t, --token          Set the private repository token. (option) set the enviornment variable CODECOV_TOKEN=:uuid. (1) -t
                       uuid.

  -u, --url            Set the target url for Enterprise customers. (option) Set environment variable
                       CODECOV_URL=https://my-hosted-codecov.com.

  -v, --verbose        Verbose mode.

  --help               Display this help screen.

  --version            Display version information.

0

Expected

I expect the tool to return a non-zero status code, to indicate failure.

[AdmiringWorm]

Thank you for opening this issue.

This is unfortunately by design. The only time the application is meant to return anything other than 0 as a exit code is when the argument --required is used.

For this case though, I'll take it under consideration and think about whether we should return something else when wrong arguments are used.

[marcomorain]

Thanks Kim.

Let me tell you about how I ended up in the situation. I was working on my macOS Workstation. I wanted to add a flag to designate the platform to our codecov uploads. We are currently building a project on Linux and Windows. I added -F Linux to our linux job, using the bash uploaded docs for reference. Then I applied the same change to the Windows job, without checking the Windows docs (since we are using just the -f <file> flag on both platforms, it appeared like the argument were the same across all uploaders on all platforms.

I pushed the Windows job, and the CodeCov step “succeeded”, so I assumed that it worked, and moved on. It took me a few days to realize that CodeCov on Windows was no longer working.

[AdmiringWorm]

I see.

Sounds like I should look into making the arguments using in this application be similiar/equal to the ones used in codecov-bash.

I've opened #53 for tracking the progress of this.

[AdmiringWorm]

I have decided that invalid arguments should definitely return a non-zero exit code. I plan to have this change available whenever version 1.6.0 is released.

[marcomorain]

Thanks Kim 👍

[AdmiringWorm]

This have now been implemented and will be part of the 1.6.0 release.

When the next release hits, invalid arguments will cause codecov to exit with the exit code of 160 (this seemed to be the most appropriate exit code when I did some research).