Ansible playbook to download files from s3

[manisnesan]

Add the encrypted strings for the secrets SHARED_S3_ID and SHARED_S3_SECRET

• inventory/prod/group_vars/ir-diagnostics-all
• This page describes how to encrypt values that can be checked into git using Ansible Vault.
• The basic structure is: ansible-vault encrypt_string --vault-password-file [vault file] -n [key] [value]. Store the vault password file in a separate vault (eg: ir-vault). Store the contents of in your local with the same file named as your vault.
• As an example assume:
  • Encrypting the key foo with the value of bar none.
  • Storing, locally, the content of as a file entitled ir-vault.
• $ ansible-vault encrypt_string --vault-password-file ir-vault -n foo 'bar none'

  Encryption successful
  foo: !vault |
        $ANSIBLE_VAULT;1.1;AES256
        64656236363537656336383666636331646639616634643432363261313063616333393334373361
        3937303034626365373830616165623932366132353662300a626136623664666332353363633463
        33663232616237646230316133646533393765336562633462306237393435616334326330303639
        3861356334303961380a663932616332346231653864336562383837326437393966313532323532
        6264

• Starting with the line foo, this becomes your name and value.

Playbook role to download files from s3 bucket

• Created a Playbook - roles/ir_search_user_pers_deploy/tasks/main.yml

---
- name: Recursively remove personalization directory incorrectly added
  file:
    path: /opt/jboss-eap/modules/com/redhat/gss/diagnostics/search/personalization
    state: absent

- name: Create personalization directory
  file:
    state: directory
    path: /opt/jboss-eap/modules/com/redhat/gss/diagnostics/search/main/personalization
    owner: jboss-eap
    group: jboss-eap
    mode: "0755"

- name: Display the value of ansible_python_interpreter
  debug:
    var: ansible_python_interpreter

- name: Check if boto3 and botocore is installed and install if not
  pip:
    name: ['boto3', 'botocore']
    state: present
  register: boto_check

- name: Display result
  debug:
    msg: "boto3 and botocore are installed"
  when: boto_check is succeeded

- name: Download user preferences artifact `user_index_map.json`
  aws_s3:
    bucket: rh-stage-ssa
    prefix: pnt-cee/dxp/cp-search-rex/
    object: pnt-cee/dxp/cp-search-rex/data/processed/prod/user_index_map.json
    dest: /opt/jboss-eap/modules/com/redhat/gss/diagnostics/search/main/personalization/user_index_map.json
    mode: get
    overwrite: different
    aws_access_key: "{{ SHARED_S3_ID }}"
    aws_secret_key: "{{ SHARED_S3_SECRET }}"

- name: Download user preferences artifact `url_index_map.json`
  aws_s3:
    bucket: rh-stage-ssa
    prefix: pnt-cee/dxp/cp-search-rex/
    object: pnt-cee/dxp/cp-search-rex/data/processed/prod/url_index_map.json
    dest: /opt/jboss-eap/modules/com/redhat/gss/diagnostics/search/main/personalization/url_index_map.json
    mode: get
    overwrite: different
    aws_access_key: "{{ SHARED_S3_ID }}"
    aws_secret_key: "{{ SHARED_S3_SECRET }}"

- name: Check user preferences artifact `preds.txt` has changed
  stat:
    path: /opt/jboss-eap/modules/com/redhat/gss/diagnostics/search/main/personalization/preds.txt
  register: preds_stat

- name: Download user preferences artifact `preds.txt`
  aws_s3:
    bucket: rh-stage-ssa
    prefix: pnt-cee/dxp/cp-search-rex/
    object: pnt-cee/dxp/cp-search-rex/models/prod/preds.txt
    dest: /opt/jboss-eap/modules/com/redhat/gss/diagnostics/search/main/personalization/preds.txt
    mode: get
    overwrite: different
    aws_access_key: "{{ SHARED_S3_ID }}"
    aws_secret_key: "{{ SHARED_S3_SECRET }}"

- name: Check if user preferences artifacts are present
  stat:
    path: "{{ item }}"
  register: pref
  loop:
    - /opt/jboss-eap/modules/com/redhat/gss/diagnostics/search/main/personalization/user_index_map.json
    - /opt/jboss-eap/modules/com/redhat/gss/diagnostics/search/main/personalization/url_index_map.json
    - /opt/jboss-eap/modules/com/redhat/gss/diagnostics/search/main/personalization/preds.txt

- name: Recursively change ownership of the contents of personalization directory
  file:
    state: directory
    path: /opt/jboss-eap/modules/com/redhat/gss/diagnostics/search/main/personalization
    recurse: yes
    owner: jboss-eap
    group: jboss-eap
    mode: "0755"

- name: List user preferences artifacts
  debug:
    msg: "{{ pref.results }}"

[manisnesan]

Before adding the role to ansible tower, create a playbook.yml to test it locally.

---
- hosts: 127.0.0.1
  connection: local
  become: true
  gather_facts: true
  roles:
    - ir_search_user_pers_deploy

• Run the playbook with this command for syntax errors ansible-playbook -K playbook.yml --syntax-check

• Once it runs successfully, run the actual command ansible-playbook -K playbook.yml -v

• Tip: --list-tasks option allows to show the list of tasks and --start-at-task to start the play from a given task instead of running the whole play. This allows to debug specific task.

• Tip: --step is another option to run a single task at a time and confirms each task before running.

• Faced an issue ERROR! couldn't resolve module/action 'aws_s3' when running aws_s3 module in the playbook. So I checked if the module is available locally by running ansible localhost -m aws_s3 and this returned

localhost | FAILED! => {
    "msg": "The module aws_s3 was redirected to amazon.aws.aws_s3, which could not be loaded."
}

This indicates the module is not available since this is a community module. Hence installed the module using ansible-galaxy command

± ansible-galaxy collection install community.aws
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/download/community-aws-5.2.0.tar.gz to /home/msivanes/.ansible/tmp/ansible-local-13648875smm6jis/tmplbcexm4z/community-aws-5.2.0-rf4j_14s
Installing 'community.aws:5.2.0' to '/home/msivanes/.ansible/collections/ansible_collections/community/aws'
Downloading https://galaxy.ansible.com/download/amazon-aws-5.2.0.tar.gz to /home/msivanes/.ansible/tmp/ansible-local-13648875smm6jis/tmplbcexm4z/amazon-aws-5.2.0-cevnhlnl
community.aws:5.2.0 was installed successfully
Installing 'amazon.aws:5.2.0' to '/home/msivanes/.ansible/collections/ansible_collections/amazon/aws'
amazon.aws:5.2.0 was installed successfully

[manisnesan]

Discovered the jupyter kernel for ansible that allows to run the playbook/tasks from Jupyter https://github.com/ansible/ansible-jupyter-kernel