search

codecov / codecov-bash

Global coverage report uploader for Codecov
https://codecov.io
Apache License 2.0
234 stars 155 forks source link

Exits non-zero although upload worked with "-X s3" (regression) #415

Closed blueyed closed 3 years ago

blueyed commented 3 years ago

codecov-bash appears to have started exiting non-zero although the upload worked (when using -Z).

I am using this script:

#!/bin/sh

set -ex

# Set --connect-timeout to work around https://github.com/curl/curl/issues/4461
curl -S -L --connect-timeout 5 --retry 6 -s -o /tmp/codecov.sh https://codecov.io/bash

for _ in 1 2 3 4 5; do
  bash /tmp/codecov.sh -Z -X fix -X s3 -f coverage-ci.xml "$@" && break
done

Build log on AppVeyor (Windows) (https://ci.appveyor.com/project/blueyed/pdbpp/builds/38280556/job/jrkm6mo1sckjubd2#L374):

bash .ci/report-coverage.sh -n "%TOXENV%-Windows" -F AppVeyor,Windows
+ curl -S -L --connect-timeout 5 --retry 6 -s -o /tmp/codecov.sh https://codecov.io/bash
+ for _ in 1 2 3 4 5
+ bash /tmp/codecov.sh -Z -X fix -X s3 -f coverage-ci.xml -n py27-coverage-Windows -F AppVeyor,Windows
  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-20210309-2b87ace
==> git version 2.26.2.windows.1 found
==> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: [unreleased]
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL 
==> Appveyor CI detected.
    project root: C:/projects/pdbpp
    Yaml found at: codecov.yml
    -> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + coverage-ci.xml bytes=97234
==> Gzipping contents
        12K /tmp/codecov.bJwjQU.gz
==> Uploading reports
    url: https://codecov.io
    query: branch=master&commit=73f289f5e34609e1ecc08d7f27e6d2f0940cc135&build=jrkm6mo1sckjubd2&build_url=https%3A%2F%2Fci.appveyor.com%2Fproject%2Fpdbpp%2Fpdbpp%2Fbuilds%2F38280556%2Fjob%2Fjrkm6mo1sckjubd2&name=py27-coverage-Windows&tag=&slug=pdbpp%2Fpdbpp&service=appveyor&flags=AppVeyor,Windows&pr=435&job=blueyed%2Fpdbpp%2F1.0.232&cmd_args=Z,X,X,f,n,F
==> Uploading to Codecov
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8456  100   227  100  8229    227   8229  0:00:01 --:--:--  0:00:01 12028
    {"message": "Coverage reports upload successfully", "uploaded": true, "queued": true, "id": "c18234ba-0b41-4be4-9d74-9fb36a2bd36b", "url": "https://codecov.io/github/pdbpp/pdbpp/commit/73f289f5e34609e1ecc08d7f27e6d2f0940cc135"}
+ for _ in 1 2 3 4 5
+ bash /tmp/codecov.sh -Z -X fix -X s3 -f coverage-ci.xml -n py27-coverage-Windows -F AppVeyor,Windows
  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-20210309-2b87ace
==> git version 2.26.2.windows.1 found
==> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: [unreleased]
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL 
==> Appveyor CI detected.
    project root: C:/projects/pdbpp
    Yaml found at: codecov.yml
    -> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + coverage-ci.xml bytes=97234
==> Gzipping contents
        12K /tmp/codecov.v3Aa9x.gz
==> Uploading reports
    url: https://codecov.io
    query: branch=master&commit=73f289f5e34609e1ecc08d7f27e6d2f0940cc135&build=jrkm6mo1sckjubd2&build_url=https%3A%2F%2Fci.appveyor.com%2Fproject%2Fpdbpp%2Fpdbpp%2Fbuilds%2F38280556%2Fjob%2Fjrkm6mo1sckjubd2&name=py27-coverage-Windows&tag=&slug=pdbpp%2Fpdbpp&service=appveyor&flags=AppVeyor,Windows&pr=435&job=blueyed%2Fpdbpp%2F1.0.232&cmd_args=Z,X,X,f,n,F
==> Uploading to Codecov
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8456  100   227  100  8229    227   8229  0:00:01 --:--:--  0:00:01 12290
    {"message": "Coverage reports upload successfully", "uploaded": true, "queued": true, "id": "87aaeeba-c3b7-4c01-baaa-cfc7c3b306e1", "url": "https://codecov.io/github/pdbpp/pdbpp/commit/73f289f5e34609e1ecc08d7f27e6d2f0940cc135"}
+ for _ in 1 2 3 4 5
+ bash /tmp/codecov.sh -Z -X fix -X s3 -f coverage-ci.xml -n py27-coverage-Windows -F AppVeyor,Windows

It worked 3 months ago using Bash-20201223-5ad85e3 (https://ci.appveyor.com/project/blueyed/pdbpp/builds/37052549/job/k86598o8359e97oy#L364):

bash .ci/report-coverage.sh -n "%TOXENV%-Windows" -F AppVeyor,Windows
+ curl -S -L --connect-timeout 5 --retry 6 -s -o /tmp/codecov.sh https://codecov.io/bash
+ for _ in 1 2 3 4 5
+ bash /tmp/codecov.sh -Z -X fix -X s3 -f coverage-ci.xml -n py38-coverage-Windows -F AppVeyor,Windows
  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-20201223-5ad85e3
==> git version 2.25.0.windows.1 found
==> curl 7.55.1 (x86_64-pc-win32) libcurl/7.55.1 OpenSSL/1.1.0f zlib/1.2.11 WinIDN libssh2/1.8.0 nghttp2/1.24.0
Release-Date: 2017-08-14
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz TLS-SRP HTTP2 HTTPS-proxy 
==> Appveyor CI detected.
    project root: C:/projects/pdbpp
    Yaml found at: codecov.yml
    -> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + coverage-ci.xml bytes=99130
==> Gzipping contents
        12K /tmp/codecov.ytLEJ8.gz
==> Uploading reports
    url: https://codecov.io
    query: branch=master&commit=a7aaf5864e15cd3aa162dcc1505da99de60784f3&build=k86598o8359e97oy&build_url=https%3A%2F%2Fci.appveyor.com%2Fproject%2Fpdbpp%2Fpdbpp%2Fbuilds%2F37052549%2Fjob%2Fk86598o8359e97oy&name=py38-coverage-Windows&tag=&slug=pdbpp%2Fpdbpp&service=appveyor&flags=AppVeyor,Windows&pr=431&job=blueyed%2Fpdbpp%2F1.0.223&cmd_args=Z,X,X,f,n,F
==> Uploading to Codecov
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8462  100    95  100  8367     95   8367  0:00:01 --:--:--  0:00:01 12592
    View reports at https://codecov.io/github/pdbpp/pdbpp/commit/81d2b0b70cbcad93a97ab6808b267f02c5f78d80
+ break
Build success
blueyed commented 3 years ago

Using the previously working version (https://raw.githubusercontent.com/codecov/codecov-bash/5ad85e34cb5e93e330220d746997f53efaab8e7d/codecov) now also fails:

  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-tbd
==> git version 2.26.2.windows.1 found
==> curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: [unreleased]
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL 
==> Appveyor CI detected.
    project root: C:/projects/pdbpp
    Yaml found at: codecov.yml
    -> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + coverage-ci.xml bytes=97234
==> Gzipping contents
        12K /tmp/codecov.I3LsRn.gz
==> Uploading reports
    url: https://codecov.io
    query: branch=master&commit=b95f82de1cf8dc38a97e77a3f23e2eb6d3d50163&build=765sql0hx1m39yq3&build_url=https%3A%2F%2Fci.appveyor.com%2Fproject%2Fpdbpp%2Fpdbpp%2Fbuilds%2F38526803%2Fjob%2F765sql0hx1m39yq3&name=py27-coverage-Windows&tag=&slug=pdbpp%2Fpdbpp&service=appveyor&flags=AppVeyor,Windows&pr=435&job=blueyed%2Fpdbpp%2F1.0.237&cmd_args=Z,X,X,f,n,F
==> Uploading to Codecov
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8456  100   227  100  8229    227   8229  0:00:01 --:--:--  0:00:01 11274
    {"message": "Coverage reports upload successfully", "uploaded": true, "queued": true, "id": "f45744a1-2d69-4cc1-b220-f7e94f02ed17", "url": "https://codecov.io/github/pdbpp/pdbpp/commit/b95f82de1cf8dc38a97e77a3f23e2eb6d3d50163"}

Therefore this appears to be an issue due to the server's response / and/or curl's output having changed?

blueyed commented 3 years ago

Yeah, it fails to parse the server's response:

==> Uploading to Codecov
++ curl -X POST --data-binary @/tmp/codecov.cFhsDr.gz --retry 5 --retry-delay 2 --connect-timeout 2 -H 'Content-Type: text/plain' -H 'Content-Encoding: gzip' -H 'X-Content-Encoding: gzip' -H 'Accept: text/plain' 'https://codecov.io/upload/v2?package=bash-tbd&token=&branch=master&commit=483102a77fa459b91cb04ae866f52fb41ea4a2e5&build=dbbkk3rj2krr4258&build_url=https%3A%2F%2Fci.appveyor.com%2Fproject%2Fpdbpp%2Fpdbpp%2Fbuilds%2F38526896%2Fjob%2Fdbbkk3rj2krr4258&name=py38-coverage-Windows&tag=&slug=pdbpp%2Fpdbpp&service=appveyor&flags=AppVeyor,Windows&pr=435&job=blueyed%2Fpdbpp%2F1.0.239&cmd_args=v,Z,X,X,f,n,F&attempt=woff|eot|otf|gif|png|jpg|jpeg|psd|ptt|pptx|numbers|pages|md|txt|xlsx|docx|doc|pdf|csv|.gitignore|html|yml|yaml'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8599  100   227  100  8372    227   8372  0:00:01 --:--:--  0:00:01 13435
+ res='{"message": "Coverage reports upload successfully", "uploaded": true, "queued": true, "id": "f5894529-3e3c-4f70-9c2a-8df2d7690e5f", "url": "https://codecov.io/github/pdbpp/pdbpp/commit/483102a77fa459b91cb04ae866f52fb41ea4a2e5"}'
++ echo '{"message": "Coverage reports upload successfully", "uploaded": true, "queued": true, "id": "f5894529-3e3c-4f70-9c2a-8df2d7690e5f", "url": "https://codecov.io/github/pdbpp/pdbpp/commit/483102a77fa459b91cb04ae866f52fb41ea4a2e5"}'
++ head -1
++ cut '-d ' -f2
+ status='"Coverage'
+ '[' '"Coverage' = '' ']'
+ '[' '"Coverage' = 200 ']'
+ say '    \033[0;32m{"message": "Coverage reports upload successfully", "uploaded": true, "queued": true, "id": "f5894529-3e3c-4f70-9c2a-8df2d7690e5f", "url": "https://codecov.io/github/pdbpp/pdbpp/commit/483102a77fa459b91cb04ae866f52fb41ea4a2e5"}\033[0m'
+ echo -e '    \033[0;32m{"message": "Coverage reports upload successfully", "uploaded": true, "queued": true, "id": "f5894529-3e3c-4f70-9c2a-8df2d7690e5f", "url": "https://codecov.io/github/pdbpp/pdbpp/commit/483102a77fa459b91cb04ae866f52fb41ea4a2e5"}\033[0m'
    {"message": "Coverage reports upload successfully", "uploaded": true, "queued": true, "id": "f5894529-3e3c-4f70-9c2a-8df2d7690e5f", "url": "https://codecov.io/github/pdbpp/pdbpp/commit/483102a77fa459b91cb04ae866f52fb41ea4a2e5"}
+ exit 1
blueyed commented 3 years ago

@thomasrockhu are you aware of this issue? It appears changes on the server side (response) make -Z fail always now.

thomasrockhu commented 3 years ago

@blueyed no I wasn't aware of it, but I will take a look next week.

blueyed commented 3 years ago

Note that there is --fail-with-body (only added recently) and --fail which might be useful here - in case parsing JSON is out of scope / the server's response does not get adjusted/reverted.

Note that it probably could have different responses for different user agents (codecov-bash uses the old format, codecov-python the new(?) one).

thomasrockhu commented 3 years ago

@blueyed I'll be honest I dug in a little and I'm not sure why we are throwing an exit code 1. Would you be able to run in verbose mode and reproduce?

blueyed commented 3 years ago

@thomasrockhu https://github.com/codecov/codecov-bash/issues/415#issuecomment-812477810 is in verbose mode, isn't it? From what I can tell the server's output changed.

blueyed commented 3 years ago

@thomasrockhu the issue is now two months old already.

Am I supposed to remove -Z, or add custom parsing of the response? (there is the GitHub Action, which uses -Z actually, but then ignores any non-zero return value instead! (https://github.com/codecov/codecov-action/issues/286))

blueyed commented 3 years ago

I've just noticed that this happens only with "-X s3", i.e. the method that gets used as a fallback. That certainly explains why it is not getting more attention, since it is usually "only" flaky then (given that users using -Z typically do not use -X s3 also, which I've only added as a workaround for a temporary issue initially).

thomasrockhu commented 3 years ago

@blueyed sorry for the delay here, I've opened a PR here