S3 deploy error: Python modules botocore or boto3 are missing, please install both

michaelBenin commented 6 years ago

Python 2.7.12 ansible 2.6.3 Ubuntu 16

On the host machine boto is installed and configured properly. I've also tested pulling down the object from a pure ansible script and it worked. I've been working with ansistrano for a while but after upgrading now I'm having issues.

Here's my config:

- hosts: woodhouse
  sudo: yes
  roles:
    - role: ansistrano.deploy
      ansistrano_deploy_from: "." # Where my local project is (relative or absolute path)
      ansistrano_deploy_to: "/var/www/wp-woodhousepainting" # Base path to deploy to.
      ansistrano_version_dir: "releases" # Releases folder name
      ansistrano_current_dir: "current" # Softlink name. You should rarely changed it.
      ansistrano_current_via: "symlink" # Deployment strategy who code should be deployed to current path. Options are symlink or rsync
      ansistrano_shared_paths: [] # Shared paths to symlink to release dir
      ansistrano_keep_releases: 1 # Releases to keep after a new deployment. See "Pruning old releases".
      ansistrano_deploy_via: "s3_unarchive" # Method used to deliver the code to the server. Options are copy, rsync, git, s3 or download.
      ansistrano_allow_anonymous_stats: no
      ansistrano_s3_bucket: wp-woodhousepainting-artifacts
      ansistrano_s3_object: "artifacts/{{ lookup('env','SHA') }}.tgz" # Add the _unarchive suffix to the ansistrano_deploy_via if your object is a package (ie: s3_unarchive)
      ansistrano_s3_region: us-east-1
      ansistrano_s3_aws_access_key: "{{ lookup('env','AWS_ACCESS_KEY') }}"
      ansistrano_s3_aws_secret_key: "{{ lookup('env','AWS_ACCESS_KEY_SECRET') }}"
      ansistrano_after_symlink_tasks_file:  "{{ playbook_dir }}/after-symlink-tasks.yml"

My s3 permissions are set correctly. I've tested on the host machine by pulling down the object I've uploaded.

---
- name: Tar the directory and upload to s3
  hosts: localhost
  connection: local

  tasks:
  - name: Simple S3 Put
    s3:
      aws_access_key: XXX
      aws_secret_key: XXX
      bucket: "wp-woodhousepainting-artifacts"
      dest: 'test.tgz' 
      encrypt: true
      expiry: 600
      max_keys: 1000
      mode: get
      object: artifacts/ba5e233a7dcc2b133bc8e76b80f8276381d7e3c1.tgz
      overwrite: always
      permission:
        - private
      region: us-east-1
      retries: 0
      validate_certs: true

Was there any recent work done around s3?

michaelBenin commented 6 years ago

Related: https://github.com/ansible/ansible/issues/42152 but 2.5.5 had same results.

michaelBenin commented 6 years ago

It works now with this:


---
- hosts: woodhouse
  sudo: yes
  pre_tasks:
  - name: Ensure boto and boto3 modules are installed
    pip:
      name: "{{ item }}"
    with_items:
    - boto3
    - botocore
  roles:
    - role: ansistrano.deploy
      ansistrano_deploy_from: "." # Where my local project is (relative or absolute path)
      ansistrano_deploy_to: "/var/www/wp-woodhousepainting" # Base path to deploy to.
      ansistrano_version_dir: "releases" # Releases folder name
      ansistrano_current_dir: "current" # Softlink name. You should rarely changed it.
      ansistrano_current_via: "symlink" # Deployment strategy who code should be deployed to current path. Options are symlink or rsync
      ansistrano_shared_paths: [] # Shared paths to symlink to release dir
      ansistrano_keep_releases: 1 # Releases to keep after a new deployment. See "Pruning old releases".
      ansistrano_deploy_via: "s3_unarchive" # Method used to deliver the code to the server. Options are copy, rsync, git, s3 or download.
      ansistrano_allow_anonymous_stats: no
      ansistrano_s3_bucket: wp-woodhousepainting-artifacts
      ansistrano_s3_object: "artifacts/{{ lookup('env','SHA') }}.tgz" # Add the _unarchive suffix to the ansistrano_deploy_via if your object is a package (ie: s3_unarchive)
      ansistrano_s3_region: us-east-1
      ansistrano_s3_aws_access_key: "{{ lookup('env','AWS_ACCESS_KEY') }}"
      ansistrano_s3_aws_secret_key: "{{ lookup('env','AWS_ACCESS_KEY_SECRET') }}"
      ansistrano_after_symlink_tasks_file:  "{{ playbook_dir }}/after-symlink-tasks.yml"

Please keep this issue open until ansible fixes it to save anyone else the time I've lost here today.

ricardclau commented 6 years ago

Yeah, this is another reason why we want to bump min requirements to 2.4. We will move to the new aws_s3 and the ansible doc won't be misleading

As of 2.4, the s3 module is dead and it is just an alias of aws_s3 (which happens to have different requirements) https://docs.ansible.com/ansible/latest/modules/aws_s3_module.html

And as usual, there is no nice transition :(

ricardclau commented 6 years ago

Moved to aws_s3 module in 6a7a825cc03bf27ab59dd3804ba561dbedb46650

emre141 commented 5 years ago

Hi,

I have same problem just using aws_ssm lookup

name : Get SSM Parameter Store RAbbitMQ Username debug: msg="{{ lookup('aws_ssm', 'rabbitmq_username', region='eu-central-1' ) }}" I am using ansible role in my autoscaling group so there is a cloud-init file to run ansible-playbook as below

ansible-playbook /etc/ansible/roles/rabbitmq/tests/test.yml --connection=local

So new instance in ASG initializating execute above command then there are some task to use yum module and pip module before using aws_ssm module for install all requirement but i have got below erron on aws_ssm task

TASK [rabbitmq : Install python-pip and other tools to install python packages/ansible] *** [DEPRECATION WARNING]: Invoking "yum" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying name: {{ item }}, please use name: [u'python-pip', u'yum- plugin-versionlock', u'awslogs', u'python2-boto3'] and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. changed: [localhost] => (item=[u'python-pip', u'yum-plugin-versionlock', u'awslogs', u'python2-boto3'])

TASK [rabbitmq : Install Boto & Ansible to run Ansible when instance comes up] *** [DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying name: {{ item }}, please use name: [u'pip', u'jinja2', u'boto', u'boto3', u'botocore', u'awscli', u'cryptography'] and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. changed: [localhost -> localhost] => (item=[u'pip', u'jinja2', u'boto', u'boto3', u'botocore', u'awscli', u'cryptography'])

TASK [rabbitmq : command] ** changed: [localhost]

TASK [rabbitmq : Get SSM Parameter Store RAbbitMQ Username] **** fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'aws_ssm'. Error was a <class 'ansible.errors.AnsibleError'>, original message: botocore and boto3 are required for aws_ssm lookup."} to retry, use: --limit @/etc/ansible/roles/rabbitmq/tests/test.retry

It works when i run my role while ec2 instance running.After that the instance wake up when i execute role again manually via connected to the instance it works and complete successfull but this problemlem only occured new instance initialization

[root@ip-172-16-121-52 ec2-user]# ansible --version ansible 2.7.0 config file = /etc/ansible/ansible.cfg configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /bin/ansible python version = 2.7.14 (default, Jul 26 2018, 19:59:38) [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] [root@ip-172-16-121-52 ec2-user]# pip --version pip 18.1 from /usr/lib/python2.7/site-packages/pip (python 2.7) [root@ip-172-16-121-52 ec2-user]# pip list | grep boto boto 2.49.0 boto3 1.9.24 botocore 1.12.24

ricardclau commented 5 years ago

It may happen that you need to execute the pip modules installation in one play, and then do a second play with ansistrano.

I am not sure if Ansible loads all existing pip modules at the start of each play or it can use the modules installed in the previous task :(

emre141 commented 5 years ago

Hi They are on the same play as below sequence.And also the log file show me boto3 installed correctly and load before ssm execute but although it has done ssm lookup fail each time.After initialization complated when i run playbook again ( ansible-playbook /etc/ansible/roles/rabbitmq/tests/test.yml --connection=local) it works and finished succesfully, execute ssm lookup plugin succesfully

name: Install Boto & Ansible to run Ansible when instance comes up pip: name: '{{ item }}' state: latest with_items:
- pip
- jinja2
- boto
- boto3
- botocore
- awscli
- cryptography delegate_to: localhost
name: Python, Ansible, Pip Version shell: which python; python --version; ansible --version; pip --version register: versions
debug: msg="{{ versions.stdout }}"
name: Boto3 import first shell: "python -c 'import boto3; print(boto3.version)'" register: pythonimport
debug: msg="{{ pythonimport.stdout }}"
name: Boto3 import second shell: "{{ ansible_python_interpreter | default('/usr/bin/python') }} -c 'import boto3; print(boto3.version)'" register: pythonimport2
debug: msg="{{ pythonimport2.stdout }}"
name : Get SSM Parameter Store RAbbitMQ Username debug: msg="{{ lookup('aws_ssm', 'rabbitmq_username', region='eu-central-1' ) }}" register: rabbitmquser delegate_to: localhost tags:
- ssmuser
name: Get SSM Parameter Store RAbbitMQ Password debug: msg="{{ lookup('aws_ssm', 'rabbitmq_password', region='eu-central-1' ) }}" register: rabbitmqpass delegate_to: localhost tags:
- ssmpass

But when i look at the cloud-init-output.log

It fail again

TASK [rabbitmq : Install python-pip and other tools to install python packages/ansible] *** [DEPRECATION WARNING]: Invoking "yum" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying name: {{ item }}, please use name: [u'python-pip', u'yum- plugin-versionlock', u'awslogs', u'python2-boto3'] and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. changed: [localhost] => (item=[u'python-pip', u'yum-plugin-versionlock', u'awslogs', u'python2-boto3'])

TASK [rabbitmq : Install Boto & Ansible to run Ansible when instance comes up] *** [DEPRECATION WARNING]: Invoking "pip" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying name: {{ item }}, please use name: [u'pip', u'jinja2', u'boto', u'boto3', u'botocore', u'awscli', u'cryptography'] and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. changed: [localhost -> localhost] => (item=[u'pip', u'jinja2', u'boto', u'boto3', u'botocore', u'awscli', u'cryptography'])

TASK [rabbitmq : Python, Ansible, Pip Version] ***** changed: [localhost]

TASK [rabbitmq : debug] **** ok: [localhost] => { "msg": "/usr/bin/python\nansible 2.7.0\n config file = /etc/ansible/ansible.cfg\n configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']\n ansible python module location = /usr/lib/python2.7/site-packages/ansible\n executable location = /usr/bin/ansible\n python version = 2.7.14 (default, Jul 26 2018, 19:59:38) [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]\npip 18.1 from /usr/lib/python2.7/site-packages/pip (python 2.7)" }

TASK [rabbitmq : Boto3 import first] *** changed: [localhost]

TASK [rabbitmq : debug] **** ok: [localhost] => { "msg": "1.9.25" }

TASK [rabbitmq : Boto3 import second] ** changed: [localhost]

TASK [rabbitmq : debug] **** ok: [localhost] => { "msg": "1.9.25" }

TASK [rabbitmq : Get SSM Parameter Store RAbbitMQ Username] **** fatal: [localhost]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'aws_ssm'. Error was a <class 'ansible.errors.AnsibleError'>, original message: botocore and boto3 are required for aws_ssm lookup."} to retry, use: --limit @/etc/ansible/roles/rabbitmq/tests/test.retry

PLAY RECAP ***** localhost : ok=25 changed=15 unreachable=0 failed=1

ansistrano / deploy

S3 deploy error: Python modules botocore or boto3 are missing, please install both #312