search

openziti-test-kitchen / ansible-collection

Ansible Galaxy Collection of Playbooks, Roles, Plugins, and Modules
Apache License 2.0
3 stars 2 forks source link

OpenZiti download components role #4

Closed ares-b closed 2 years ago

ares-b commented 2 years ago

Hello,

Submitting a role to download openziti components.

This role can :

We might need to discuss about where to store Ziti binaries and ZAC source code by default. I've put them in /opt/openziti by default (can be changed by variables tho), but I'm not sure that's the best location by default.

Also, I need your opinion on whether to use group names, group vars or host vars to specify on which host deploy which OpenZiti component. Atm, the role is using group names, but again, I'm not sure that's the best choice.

One last thing, I'm planning on handling become_passwords with varibles, we need like 2 become_passwords :

I'm thinking about asking users to give those as input in their playbooks using ansible-vault, AWX or a vault. What do you guys think ?

The PR also contains a github actions CI .github/workflows/molecule-ci.yml, it has two jobs, the first one goes over the repository and catches every directory having a sub-directory named molecule. The second, runs molecule test against every one of those directories.

Feel free to give me feedback, I'll take it into consideration and change what needs to be changed.

Regards, Arslane BAHLEL

qrkourier commented 2 years ago

Good stuff, @ares-b . Let's do two things: reconcile the default directory structure with the Linux package structure for ziti-ege-tunnel and put the binaries in /opt/openziti/bin.

Let's also add a simple playbook in the collection that demonstrates how to call the new role.

You could define the sudo password from env like this:

ansible_become_pass: "{{ lookup('env', 'ANSIBLE_BECOME_PASS') | default(omit) }}"

Regarding host vars vs groups for assigning OpenZiti components for install, groups make sense to me for the sake of simplicity. Hostvars would be more flexible, especially if driven by a custom inventory plugin. Anyway, that's an easy change if we ever see the need to use hostvars.

ares-b commented 2 years ago

@qrkourier I added an example playbook, modified default remote binaries path to /opt/openziti/bin, added download support for openziti edge-tunnel and moved openziti install specification from group_names to host_vars

qrkourier commented 2 years ago

@ares-b Here's what I encountered running the demo playbook. Looks like a var doesn't have a default. You might be able to use omit for this one.

❯ rm -rf ~/.ansible/collections/ansible_collections/community/openziti

❯ ansible-galaxy collection install git+https://github.com/ares-b/openziti-ansible-collection.git#/community/openziti,main

# my inventory host is Amazon Linux
❯ ansible-playbook community.openziti.demo_openziti_download -u ec2-user -i 13.57.252.159,                            
 ____________________________________________________________
/ TASK [community.openziti.openziti_download : Make sure tar \
\ is installed on Zac hosts]                                 /
 ------------------------------------------------------------

fatal: [13.57.252.159]: FAILED! => {"msg": "The conditional check ''ziti-console' in openziti_components' failed. The error was: error while evaluating conditional ('ziti-console' in openziti_components): 'openziti_components' is undefined\n\nThe error appears to be in '/home/kbingham/.ansible/collections/ansible_collections/community/openziti/roles/openziti_download/tasks/required_packages.yaml': line 3, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Make sure tar is installed on Zac hosts\n  ^ here\n"}

Note that the upstream repo's directory structure changed to move /community/openziti to /community.openziti.

ares-b commented 2 years ago

Hi @qrkourier, I just pushed the changes you asked for.

As you may have noticed, I've done everything with Ansible existing modules, I think that here's a few tasks that can be simplified with a custom module such as the Select binaries and remote_server_path from openziti components variable on push_to_remote.yaml file.

I was wondering, if you guys would be okay with me making custom module to make the ansible role more succinct.

qrkourier commented 2 years ago

@ares-b I'm definitely in favor of custom modules for the reason you mentioned.

Here's what happened when I tried to run the demo playbook on Amazon Linux.

❯ ansible-playbook community.openziti.demo_openziti_download -u ec2-user -i 13.57.252.159,

 ____________________________________________________________
/ TASK [community.openziti.openziti_download : Make sure tar \
\ is installed on cache server]                              /
 ------------------------------------------------------------

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NoneType: None
fatal: [13.57.252.159 -> localhost]: FAILED! => {"changed": false, "msg": "Could not detect which package manager to use. Try gathering facts or setting the \"use\" option."}
ares-b commented 2 years ago

@qrkourier That's because ansible.builtin.package needs to be executed with sudo privileges in order to infer which package manager to use (apt, yum, etc) on your cache server (localhost if u left openziti_cache_localhost = true). You either need to launch you playbook with --ask-become-pass or -K param, then you enter your localhost sudo password.

ansible-playbook community.openziti.demo_openziti_download -u ec2-user -i 13.57.252.159, -K

Or, you provide the variable localhost_become_pass with your localhost become pass.

I've documented this on the role's ReadME. I'll document this on the example playbook as well.

@ares-b I'm definitely in favor of custom modules for the reason you mentioned.

Alright, I'll do the necessary updates as soon as I finish with the Install & Configure role.

qrkourier commented 2 years ago

@ares-b This appears to be a stray string in the README:

5 Allee Saint Exupery 92390 Villeneuve La Garenne

qrkourier commented 2 years ago

@ares-b I tried to get this working with a remote cache server because, personally, I would strongly prefer to avoid using become on localhost. In this case there is no become password for ec2-user on Amazon Linux.

# hosts.yml
all:
  children:
    dev_network:
      hosts:
        host_one:
          ansible_host: 13.57.252.159
          ansible_ssh_user: ec2-user
          ansible_become_pass: ''
          openziti_components:
            - ziti-controller
            - ziti-router
            - ziti-tunnel
            - ziti
            - ziti-console
            - ziti-edge-tunnel
        cache_server:
          ansible_host: 13.57.252.159
          ansible_ssh_user: ec2-user
          ansible_become_pass: ''

$ ansible-playbook community.openziti.demo_openziti_download -i ./hosts.yml
 ____________________________________________________________
/ TASK [community.openziti.openziti_download : Make sure tar \
\ is installed on cache server]                              /
 ------------------------------------------------------------

fatal: [host_one -> localhost]: FAILED! => {"msg": "{{ my_localhost_sudo_pass_from_secrets }}: 'my_localhost_sudo_pass_from_secrets' is undefined"}
...
ares-b commented 2 years ago

Hi @qrkourier

This appears to be a stray string in the README

Right, joys of copy/pasting haha

I tried to get this working with a remote cache server because, personally, I would strongly prefer to avoid using become on localhost. In this case there is no become password for ec2-user on Amazon Linux.

Can you try and remove localhost_become_pass from your playbook ? I've left it by default to "{{ my_localhost_sudo_pass_from_secrets }}" and the error says that it can't locate variable my_localhost_sudo_pass_from_secrets.

Try running it with this playbook : 

- hosts: '{{ target | default("all") }}'

  tasks:
    - ansible.builtin.include_role:
        name: openziti_download
      vars:
        openziti_cache_localhost: false

I just re-commited the example playbook with the variable localhost_become_pass commented.

qrkourier commented 2 years ago

To test that change to the demo playbook I committed to a local branch then installed from file:// URI like this.

ansible-galaxy collection install git+file://${PWD}

I found the correct components installed with remote cache mode on RHEL8, Fedora34, Ubuntu20, and Amazon Linux.

ares-b commented 2 years ago

I found the correct components installed with remote cache mode on RHEL8, Fedora34, Ubuntu20, and Amazon Linux.

Alright, it should also work with Debian, CentOS and Oracle (and probably every other Unix distribution).

If you want to test it with localhost cache, make sure you have tar, zip and git installed and disable these tasks :

qrkourier commented 2 years ago

@ares-b Local cache testing succeeded with Debian11, SUSE15, and Oracle8. I tagged and did --skip-tags for those three tasks that check required packages on the cache server.

ares-b commented 2 years ago

@ares-b Local cache testing succeeded with Debian11, SUSE15, and Oracle8. I tagged and did --skip-tags for those three tasks that check required packages on the cache server.

Thanks a good Idea, I'll tag those tasks and document the tag skipping if anyone else doesn't want to provide their localhost sudo pass

qrkourier commented 2 years ago

@ares-b This is what I get when I try to run the default test suite.

community.openziti/roles/openziti_download arslane-main
molecule ❯ molecule test molecule/default/molecule.yml 
---
dependency:
  name: galaxy
driver:
  name: docker
lint: 'set -e

  yamllint .

  ansible-lint .

  '
platforms:
  - image: docker.io/amd64/ubuntu
    name: ubuntu_amd64
    pre_build_image: false
  - image: docker.io/amd64/oraclelinux:8
    name: oracle_amd64
    pre_build_image: false
provisioner:
  inventory:
    host_vars:
      oracle_amd64:
        ansible_user: ansible
        openziti_components:
          - ziti-controller
          - ziti-router
          - ziti-tunnel
          - ziti
          - ziti-console
          - ziti-edge-tunnel
      ubuntu_amd64:
        ansible_user: ansible
        openziti_components:
          - ziti-controller
          - ziti-router
          - ziti-tunnel
          - ziti
          - ziti-console
          - ziti-edge-tunnel
  name: ansible
verifier:
  name: ansible

CRITICAL Failed to pre-validate.

{'driver': [{'name': ['unallowed value docker']}]}
qrkourier commented 2 years ago

@ares-b Test passes runs if I install PyPi molecule-docker and yamllint and ansible-lint.

molecule ❯ molecule test molecule/default/molecule.yml                                                                                                                                    
INFO     default scenario test matrix: dependency, lint, cleanup, destroy, syntax, create, prepare, converge, idempotence, side_effect, verify, cleanup, destroy
INFO     Performing prerun with role_name_check=0...                                                                                                                                      INFO     Set ANSIBLE_LIBRARY=/home/kbingham/.cache/ansible-compat/134c05/modules:/home/kbingham/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
INFO     Set ANSIBLE_COLLECTIONS_PATH=/home/kbingham/.cache/ansible-compat/134c05/collections:/home/kbingham/.ansible/collections:/usr/share/ansible/collections
INFO     Set ANSIBLE_ROLES_PATH=/home/kbingham/.cache/ansible-compat/134c05/roles:/home/kbingham/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO     Running ansible-galaxy collection install -v --force -p /home/kbingham/.cache/ansible-compat/134c05/collections ../..
INFO     Running default > dependency
WARNING  Skipping, missing the requirements file.
WARNING  Skipping, missing the requirements file.
INFO     Running default > lint
WARNING: PATH altered to include /home/kbingham/.pyenv/versions/3.9.0/envs/molecule/bin                                                                                                   INFO     Running default > cleanup                                                                                                                                                        
WARNING  Skipping, cleanup playbook not configured.                                                                                                                                       INFO     Running default > destroy                                                                                                                                                        INFO     Sanity checks: 'docker'

PLAY [Destroy] *****************************************************************

TASK [Destroy molecule instance(s)] ********************************************
changed: [localhost] => (item=ubuntu_amd64)
changed: [localhost] => (item=oracle_amd64)                                                  

TASK [Wait for instance(s) deletion to complete] *******************************
ok: [localhost] => (item=ubuntu_amd64)     
ok: [localhost] => (item=oracle_amd64)     

TASK [Delete docker networks(s)] ***********************************************

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

INFO     Running default > syntax
ERROR! A playbook must be a list of plays, got a <class 'ansible.parsing.yaml.objects.AnsibleMapping'> instead

The error appears to be in '/home/kbingham/Sites/netfoundry/github/openziti-ansible-collection/community.openziti/roles/openziti_download/molecule/default/molecule.yml': line 3, column 1, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

dependency:
^ here
WARNING  Retrying execution failure 4 of: ansible-playbook --inventory /home/kbingham/.cache/molecule/openziti_download/default/inventory --skip-tags molecule-notest,notest --syntax-check molecule/default/molecule.yml /home/kbingham/Sites/netfoundry/github/openziti-ansible-collection/community.openziti/roles/openziti_download/molecule/default/converge.yml
CRITICAL Ansible return code was 4, command was: ['ansible-playbook', '--inventory', '/home/kbingham/.cache/molecule/openziti_download/default/inventory', '--skip-tags', 'molecule-notest,notest', '--syntax-check', 'molecule/default/molecule.yml', '/home/kbingham/Sites/netfoundry/github/openziti-ansible-collection/community.openziti/roles/openziti_download/molecule/default/converge.yml']
WARNING  An error occurred during the test sequence action: 'syntax'. Cleaning up.
INFO     Running default > cleanup
WARNING  Skipping, cleanup playbook not configured.
INFO     Running default > destroy

PLAY [Destroy] *****************************************************************

TASK [Destroy molecule instance(s)] ********************************************
changed: [localhost] => (item=ubuntu_amd64)
changed: [localhost] => (item=oracle_amd64)

TASK [Wait for instance(s) deletion to complete] *******************************
ok: [localhost] => (item=ubuntu_amd64)
ok: [localhost] => (item=oracle_amd64)

TASK [Delete docker networks(s)] ***********************************************

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

INFO     Pruning extra files from scenario ephemeral directory

EDIT: updated test output to show error after installing missing PyPi packages

qrkourier commented 2 years ago

@ares-b I found that pip install "molecule[lint]" will install all three PyPi packages.

ares-b commented 2 years ago

@qrkourier The error you're getting is weird, can you try this :

cd community.openziti/roles/openziti_download
molecule test

molecule test runs the default scenario, if you want to try all scenarios, make it molecule test --all

qrkourier commented 2 years ago

@ares-b I'm able to run the default tests when I simply say molecule test in the role dir, as you suggested.

The next issue I found is that this test appears to get stuck. Perhaps it is waiting for a become password?

❯ docker ps
CONTAINER ID   IMAGE                                          COMMAND                  CREATED         STATUS         PORTS     NAMES
28e8ef9a1579   molecule_local/docker.io/amd64/oraclelinux:8   "bash -c 'while true…"   3 minutes ago   Up 3 minutes             oracle_amd64
44fc7d577800   molecule_local/docker.io/amd64/ubuntu          "bash -c 'while true…"   3 minutes ago   Up 3 minutes             ubuntu_amd64

❯ molecule test
# ...
TASK [openziti_download : Make sure tar is installed on cache server] **********                         

^C                                                                                                                                                                                                                
Aborted!
ares-b commented 2 years ago

@ares-b I'm able to run the default tests when I simply say molecule test in the role dir, as you suggested.

The next issue I found is that this test appears to get stuck. Perhaps it is waiting for a become password?

❯ docker ps
CONTAINER ID   IMAGE                                          COMMAND                  CREATED         STATUS         PORTS     NAMES
28e8ef9a1579   molecule_local/docker.io/amd64/oraclelinux:8   "bash -c 'while true…"   3 minutes ago   Up 3 minutes             oracle_amd64
44fc7d577800   molecule_local/docker.io/amd64/ubuntu          "bash -c 'while true…"   3 minutes ago   Up 3 minutes             ubuntu_amd64

❯ molecule test
# ...
TASK [openziti_download : Make sure tar is installed on cache server] **********                         

^C                                                                                                                                                                                                                
Aborted!

Yeah, basically the default scenario launches the role with your localhost as cache server. You can launch with molecule test --skip-tags=cache_server_packages

qrkourier commented 2 years ago

@ares-b I was able to run the default tests like this, but the same test gets stuck. It's as if the skip-tags option was ignored.

# note the added POSIX -- to stop parsing positionals as options for `test` subcommand, and instead pass through to `ansible-playbook`
molecule test -- --skip-tags=cache_server_packages

When running the remote_cache test suite I get that same unfamiliar error.

$ molecule test ./molecule/remote_cache/molecule.yml                                     
# ...
INFO     Running default > syntax
ERROR! A playbook must be a list of plays, got a <class 'ansible.parsing.yaml.objects.AnsibleMapping'> instead

The error appears to be in '/home/kbingham/Sites/netfoundry/github/openziti-ansible-collection/community.openziti/roles/openziti_download/molecule/remote_cache/molecule.yml': line 3, column 1, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

dependency:
^ here
ares-b commented 2 years ago

@qrkourier both of scenarios are working perfectly on my localhost and with github actions https://github.com/ares-b/openziti-ansible-collection/actions

Can you try this :

cd community.openziti/roles/openziti_download
molecule test -- --skip-tags=cache_server_packages

and 

cd community.openziti/roles/openziti_download
molecule test -s remote_cache

I don't think that molecule test ./molecule/remote_cache/molecule.yml is supposed to work, you always need to be into the "working directory" before launching a molecule <cmd> command.

For example, If u want to test a role with specific scenario, you need to cd into that role directory and launch molecule -s <scenario_name>.