search

codeclimate / test-reporter

Code Climate Test Reporter
MIT License
156 stars 76 forks source link

why is it trying to load html files? #333

Closed chrisbutler closed 6 years ago

chrisbutler commented 6 years ago

I've been banging my head against a wall trying to understand this output from the cc-test-reporter binary:

/bin/cc-test-reporter format-coverage -t simplecov -o coverage/codeclimate.cucumber.json $CIRCLE_ARTIFACTS/coverage/cucumber/.resultset.json

/bin/cc-test-reporter: line 1: html: No such file or directory
/bin/cc-test-reporter: line 2: syntax error near unexpected token `<'
'

It seems like it's trying to load the index.html (also generated from simplecov) file in the coverage/ directory... not sure how that's possible since the .resultset.json file is clearly specified

chrisbutler commented 6 years ago

followup, i've ensured that simplecov isn't generating any html and i'm still getting the same errors. this is on circle-ci, with caching disabled

chrisbutler commented 6 years ago

second followup... attempting to debug and setting -d or --debug and suddenly there are no errors? what is going on?!?!?

maxjacobson commented 6 years ago

Hi @chrisbutler

Just to be clear -- has the error totally stopped, or it only stops when you use the debug flag?

If it's totally resolved, then I think I can explain what's going on. This morning, we had a small incident with our DNS/SSL that was only affecting some people.

I think you may be hitting https://github.com/codeclimate/test-reporter/issues/248, which is to say that the test reporter isn't trying to read HTML files from your file system, but rather it's trying to parse an API response from Code Climate as JSON, and finding that it can't, because it responded with an HTML error page. We really need to improve that error handling experience because it's totally confusing.

We've recently made some additional measures (just in the last half hour) to shake loose the remaining DNS issues, so it's possible that that helped you.

Does that sound right? If so I'll go ahead and close this. Sorry for the confusion!

chrisbutler commented 6 years ago

hey @maxjacobson, thanks for the response! it looks like the -d thing was a red herring... i think the codeclimate step never actually ran that time

it's still happening in a fresh build just now 😭

seems like you're right about it receiving an html response from the api:

circleci@95b699b761a7:~$ /bin/cc-test-reporter
/bin/cc-test-reporter: line 1: html: No such file or directory
/bin/cc-test-reporter: line 2: syntax error near unexpected token `<'
'bin/cc-test-reporter: line 2: `<head><title>503 Service Temporarily Unavailable</title></head>
chrisbutler commented 6 years ago

@maxjacobson sorry for so many posts, but i finally managed to track down all of the pieces of this. and it's a fun one...

looks like when we built a docker image this morning, it wrote the html error to the file instead of the reporter code. that pre-built image is used as the container for cucumber/rspec tests, so no matter what code changes i made the executable was just html 🤦‍♂️

the example curl script should add a --fail flag which would prevent this, i think?

    - curl -fL https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 > ./cc-test-reporter
    - chmod +x ./cc-test-reporter
maxjacobson commented 6 years ago

Aha, great catch.

Re your suggestion: hmm, in that case, what would get written to the file instead of the HTML? I'm concerned it would still write the file, make it executable, and then try to execute it, and you would get another error. Does that make sense?

chrisbutler commented 6 years ago

it should just fail to write anything in this case, so the docker build step would fail

maxjacobson commented 6 years ago

Ah. We generally recommend people add an || true to these lines, because it can be frustrating for a temporary CC outage to lead to a failing build. If you're comfortable with that (and we certainly will do our best to avoid/minimize outages!) then that sounds like a good approach! I'm going to go ahead and close this now since your sleuthing has gotten to the bottom of it :mag: