Failed tarball fetches are ignored

[ironfroggy]

Bug Report

MAZER VERSION

name = mazer
version = 0.5.0
config_file = /home/calvin/.ansible/mazer.yml
uname = Linux, willow-epsilon, 4.4.0-17134-Microsoft, #706-Microsoft Mon Apr 01 18:13:00 PST 2019, x86_64
executable_location = /mnt/c/Users/calvin/redhat/.virtualenvs/orion/bin/mazer
python_version = 3.7.1 (default, Oct 22 2018, 11:21:55) [GCC 8.2.0]
python_executable = /mnt/c/Users/calvin/redhat/.virtualenvs/orion/bin/python3.7

CONFIGURATION

defaults

SUMMARY

If something is wrong with the API server's file serving and an HTTP error response comes back from a request to fetch an artifact tarball, Mazer does not detect this, saves the HTML error page contents as a .tar.gz and tries to unpack it. We should detect this and give a more appropriate, earlier error to the user.

STEPS TO REPRODUCE

I don't know how to intentionally reproduce this at this time. It happened when we deployed and tested on galaxy-qa because of a misconfiguration, but that has been resolved. I can imagine intermittent network errors causing something like this unpredictably and would like to be prepared for that.

EXPECTED RESULTS

An error message that indicates a network or server error has occured.

ACTUAL RESULTS

The actual error looks like a corrupted tarball, which feels like a problem with the build I uploaded rather than the server at this time, and doesn't actually indicate the source of the problem.

Error opening the tar file /tmp/tmp-ansible-galaxy-content-archive-1gdn042y.tar.gz with flags: r:gz for repo: orionuser1.collection_dep_a_wplnfajx,1.0.0:
- you can use --ignore-errors to skip failed collections and finish processing the list.

[alikins]

@ironfroggy Can you grab the results from the server including headers? ie, grab the url, then 'curl -v' or http -v --follow on the original url?

And, or, a copy of the mazer log file from DEBUG level with the error?

[ironfroggy]

@alikins Unfortunately, I can't. Not right now, because the problem that caused that has been resolved on the server. We can try to do it again temporarily post-launch?

[awcrosby]

@alikins my current local dev environment gives a result that causes the same mazer response, here is the server results:

(vmaz) awc:code$ curl -v http://localhost:8000/download/awcrosby-collection_reqs_test-0.0.3.tar.gz
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8000 (#0)
> GET /download/awcrosby-collection_reqs_test-0.0.3.tar.gz HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< content-type: text/plain; charset=utf-8
< content-length: 14
< date: Thu, 30 May 2019 20:14:18 GMT
< server: Python/3.6 aiohttp/3.5.4
< connection: close
< 
* Closing connection 0
404: Not Found(vmaz) awc:code$ 
(vmaz) awc:code$ 
(vmaz) awc:code$ 
(vmaz) awc:code$ mazer install awcrosby.collection_reqs_test
Installing:
  awcrosby.collection_reqs_test,*
WARNING| - awcrosby.collection_reqs_test,0.0.3 was NOT installed successfully: Error opening the tar file /tmp/tmp-ansible-galaxy-content-archive-2kr4h0vk.tar.gz with flags: r:gz for repo: awcrosby.collection_reqs_test,0.0.3 
Error opening the tar file /tmp/tmp-ansible-galaxy-content-archive-2kr4h0vk.tar.gz with flags: r:gz for repo: awcrosby.collection_reqs_test,0.0.3:
- you can use --ignore-errors to skip failed collections and finish processing the list.
(vmaz) awc:code$ 

[alikins]

@ironfroggy ah, I'm not using requests.raise_for_exception() for the downloads, so the 404 isn't raising errors (rest_api does this, I need to update download to do it as well).