search

ansible-collections / google.cloud

GCP Ansible Collection https://galaxy.ansible.com/google/cloud
https://cloud.google.com
GNU General Public License v3.0
99 stars 129 forks source link

gcp_storage_object plugin does not support to download file extensions other than flat files #26

Open PedDavid opened 5 years ago

PedDavid commented 5 years ago

From GoogleCloudPlatform/magic-modules#2428

Related issue from @sparampalli ansible/ansible#56173

SUMMARY

gcp_storage_object plugin does not support to download file extensions other than flat files. For e.g. if you are attempting to download .pkg/.zip files etc, files will be downloaded but corrupted. The downloaded files MD5/Checksum does not match with original files. ISSUE TYPE

Bug Report

COMPONENT NAME

gcp_storage_object ANSIBLE VERSION

Ansible version - ansible 2.8.0.dev0

CONFIGURATION

ansible-playbook example.yml [WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [localhost] *****

TASK [Gathering Facts] *** ok: [localhost]

TASK [SaaS Actions | Patch Controller | Register the temporary Storage] ** ok: [localhost]

TASK [SaaS Actions | Patch Controller | Get the package from GCP Storage bucket] ***** ok: [localhost]

TASK [debug] ***** ok: [localhost] => { "msg": { "bucket": "XXXXXXXX", "changed": false, "contentType": "application/octet-stream", "crc32c": "grZeMQ==", "etag": "CMTGuN6DkeECEAE=", "failed": false, "generation": "1553095537337156", "id": "XXXXXXX/17.2.13/1p1/patch.pkg/1553095537337156", "kind": "storage#object", "md5Hash": "cQ1nu3ZXBkDA46MNSJ5KBQ==", "mediaLink": "https://www.googleapis.com/download/storage/v1/b/XXXXXXX/o/17.2.13%2F1p1%2Fpatch.pkg?generation=1553095537337156&alt=media", "metageneration": "1", "name": "17.2.13/1p1/patch.pkg", "selfLink": "https://www.googleapis.com/storage/v1/b/XXXXXXX/o/17.2.13%2F1p1%2Fpatch.pkg", "size": "135010", "storageClass": "MULTI_REGIONAL", "timeCreated": "2019-03-20T15:25:37.336Z", "timeStorageClassUpdated": "2019-03-20T15:25:37.336Z", "updated": "2019-03-20T15:25:37.336Z" } }

PLAY RECAP *** localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

OS / ENVIRONMENT

Ubuntu 16.04 Mac OS STEPS TO REPRODUCE

EXPECTED RESULTS

File will be downloaded having the same MD5/Checksum of the file which is in the bucket ACTUAL RESULTS

File md5 downloaded through ansible module $ md5 patch.pkg MD5 (patch.pkg) = 14ad8dac4c6a05e544eeda078a47744a $ cksum patch.pkg 4179052778 269112 patch.pkg

File md5 downloaded directly from bucket $ md5 .pkg MD5 (patch.pkg) = 710d67bb76570640c0e3a30d489e4a05 $ cksum .pkg 3630494301 135010 patch.pkg

$ ansible-playbook -vvvv getfile_single.yml ansible-playbook 2.8.0.dev0 config file = None configured module search path = [u'/Users/username/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /Library/Python/2.7/site-packages/ansible executable location = /usr/local/bin/ansible-playbook python version = 2.7.10 (default, Feb 22 2019, 21:17:52) [GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)] No config file found; using defaults setting up inventory plugins host_list declined parsing /etc/ansible/hosts as it did not pass it's verify_file() method Skipping due to inventory source not existing or not being readable by the current user script declined parsing /etc/ansible/hosts as it did not pass it's verify_file() method auto declined parsing /etc/ansible/hosts as it did not pass it's verify_file() method Skipping due to inventory source not existing or not being readable by the current user yaml declined parsing /etc/ansible/hosts as it did not pass it's verify_file() method Skipping due to inventory source not existing or not being readable by the current user ini declined parsing /etc/ansible/hosts as it did not pass it's verify_file() method Skipping due to inventory source not existing or not being readable by the current user toml declined parsing /etc/ansible/hosts as it did not pass it's verify_file() method [WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

Loading callback plugin default of type stdout, v2.0 from /Library/Python/2.7/site-packages/ansible/plugins/callback/default.pyc

PLAYBOOK: getfile_single.yml ***** Positional arguments: getfile_single.yml become_method: sudo inventory: (u'/etc/ansible/hosts',) forks: 5 tags: (u'all',) verbosity: 4 connection: smart timeout: 10 1 plays in getfile_single.yml

PLAY [localhost] *****

TASK [Gathering Facts] *** task path: /Users/username/sri-working/ansible_plabooks/gcp/getfile_single.yml:2 <127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: username <127.0.0.1> EXEC /bin/sh -c 'echo ~username && sleep 0' <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /Users/username/.ansible/tmp/ansible-tmp-1557238791.06-108244522845969" && echo ansible-tmp-1557238791.06-108244522845969="echo /Users/username/.ansible/tmp/ansible-tmp-1557238791.06-108244522845969" ) && sleep 0' Using module file /Library/Python/2.7/site-packages/ansible/modules/system/setup.py <127.0.0.1> PUT /Users/username/.ansible/tmp/ansible-local-628558fHMIu/tmpS0ivRT TO /Users/username/.ansible/tmp/ansible-tmp-1557238791.06-108244522845969/AnsiballZ_setup.py <127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/username/.ansible/tmp/ansible-tmp-1557238791.06-108244522845969/ /Users/username/.ansible/tmp/ansible-tmp-1557238791.06-108244522845969/AnsiballZ_setup.py && sleep 0' <127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /Users/username/.ansible/tmp/ansible-tmp-1557238791.06-108244522845969/AnsiballZ_setup.py && sleep 0' <127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/username/.ansible/tmp/ansible-tmp-1557238791.06-108244522845969/ > /dev/null 2>&1 && sleep 0' ok: [localhost] META: ran handlers

TASK [SaaS Actions | Patch Controller | Register the temporary Storage] ** task path: /Users/username/sri-working/ansible_plabooks/gcp/getfile_single.yml:5 ok: [localhost] => { "ansible_facts": { "bucket_path": "17.2.13%2F1p1%2Fpatch.pkg", "local_path": "/Users/username/Downloads/17.2.13_1p1_patch.pkg" }, "changed": false }

TASK [SaaS Actions | Patch Controller | Get the package from GCP Storage bucket] ***** task path: /Users/username/sri-working/ansible_plabooks/gcp/getfile_single.yml:10 <127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: username <127.0.0.1> EXEC /bin/sh -c 'echo ~username && sleep 0' <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /Users/username/.ansible/tmp/ansible-tmp-1557238792.52-205689716613005" && echo ansible-tmp-1557238792.52-205689716613005="echo /Users/username/.ansible/tmp/ansible-tmp-1557238792.52-205689716613005" ) && sleep 0' Using module file /Library/Python/2.7/site-packages/ansible/modules/cloud/google/gcp_storage_object.py <127.0.0.1> PUT /Users/username/.ansible/tmp/ansible-local-628558fHMIu/tmp535NoH TO /Users/username/.ansible/tmp/ansible-tmp-1557238792.52-205689716613005/AnsiballZ_gcp_storage_object.py <127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/username/.ansible/tmp/ansible-tmp-1557238792.52-205689716613005/ /Users/username/.ansible/tmp/ansible-tmp-1557238792.52-205689716613005/AnsiballZ_gcp_storage_object.py && sleep 0' <127.0.0.1> EXEC /bin/sh -c '/usr/bin/python /Users/username/.ansible/tmp/ansible-tmp-1557238792.52-205689716613005/AnsiballZ_gcp_storage_object.py && sleep 0' <127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/username/.ansible/tmp/ansible-tmp-1557238792.52-205689716613005/ > /dev/null 2>&1 && sleep 0' ok: [localhost] => { "bucket": "XXXXXXXX", "changed": false, "contentType": "application/octet-stream", "crc32c": "grZeMQ==", "etag": "CMTGuN6DkeECEAE=", "generation": "1553095537337156", "id": "XXXXXXXX/17.2.13/1p1/patch.pkg/1553095537337156", "invocation": { "module_args": { "action": "download", "auth_kind": "serviceaccount", "bucket": "XXXXXXXX", "dest": "/Users/username/Downloads/17.2.13_1p1_patch.pkg", "overwrite": true, "project": "saastest-202018", "scopes": [ "https://www.googleapis.com/auth/cloud-platform", "https://www.googleapis.com/auth/cloud-platform.read-only", "https://www.googleapis.com/auth/devstorage.full_control", "https://www.googleapis.com/auth/devstorage.read_only", "https://www.googleapis.com/auth/devstorage.read_write" ], "service_account_email": null, "service_account_file": "XXXXXXXX", "src": "17.2.13%2F1p1%2Fpatch.pkg", "state": "present" } }, "kind": "storage#object", "md5Hash": "cQ1nu3ZXBkDA46MNSJ5KBQ==", "mediaLink": "https://www.googleapis.com/download/storage/v1/b/XXXXXXXX/o/17.2.13%2F1p1%2Fpatch.pkg?generation=1553095537337156&alt=media", "metageneration": "1", "name": "17.2.13/1p1/patch.pkg", "selfLink": "https://www.googleapis.com/storage/v1/b/XXXXXXXX/o/17.2.13%2F1p1%2Fpatch.pkg", "size": "135010", "storageClass": "MULTI_REGIONAL", "timeCreated": "2019-03-20T15:25:37.336Z", "timeStorageClassUpdated": "2019-03-20T15:25:37.336Z", "updated": "2019-03-20T15:25:37.336Z" }

TASK [debug] ***** task path: /Users/username/sri-working/ansible_plabooks/gcp/getfile_single.yml:28 ok: [localhost] => { "msg": { "bucket": "XXXXXXXX", "changed": false, "contentType": "application/octet-stream", "crc32c": "grZeMQ==", "etag": "CMTGuN6DkeECEAE=", "failed": false, "generation": "1553095537337156", "id": "XXXXXXXX/17.2.13/1p1/patch.pkg/1553095537337156", "kind": "storage#object", "md5Hash": "cQ1nu3ZXBkDA46MNSJ5KBQ==", "mediaLink": "https://www.googleapis.com/download/storage/v1/b/XXXXXXXX/o/17.2.13%2F1p1%2Fpatch.pkg?generation=1553095537337156&alt=media", "metageneration": "1", "name": "17.2.13/1p1/patch.pkg", "selfLink": "https://www.googleapis.com/storage/v1/b/XXXXXXXX/o/17.2.13%2F1p1%2Fpatch.pkg", "size": "135010", "storageClass": "MULTI_REGIONAL", "timeCreated": "2019-03-20T15:25:37.336Z", "timeStorageClassUpdated": "2019-03-20T15:25:37.336Z", "updated": "2019-03-20T15:25:37.336Z" } } META: ran handlers META: ran handlers

PLAY RECAP *** localhost : ok=4 changed=0 unreachable=0 failed=0 skipped=0

PedDavid commented 5 years ago

@ericsysmin solution didn't work for me as f.write(data.content) gave an error because data.content was bytes.

swackhamer commented 5 years ago

the file needs to be opened in binary mode to write the content to the file. I opened the #44 to fix this issue.

https://requests.kennethreitz.org/en/master/user/quickstart/#binary-response-content

PedDavid commented 5 years ago

Awesome @swackhamer , thanks a lot 😄

rambleraptor commented 5 years ago

The gcp_storage_object module has been rewritten to use the google-cloud-storage Python library. This library is the Google official way of handling upload/downloads to GCS and fixes most of the problems that the (very buggy) current module has.

This shouldn't be an issue at the moment anymore. Let me know if you're having any more issues!

PedDavid commented 5 years ago

@rambleraptor, it is one of the recommended ways of managing secrets in Google Cloud (source).

rambleraptor commented 5 years ago

I had that completely turned around. My mistake! Yeah, I'll look into this soon. Thanks for the suggestion.

PedDavid commented 5 years ago

Wait, sorry, this comment should've been on #27

ericsysmin commented 4 years ago

We used it to manage secrets, as well. Hence why our issue with gcp_storage_object, and then also why I created the kms filter to handle decrypting the object