jamesla commented 6 years ago

Issue Type

Bug report

Molecule and Ansible details

ansible --version 2.4.1.0
molecule --version 1.25.1

Molecule installation method: pip
Ansible installation method: One of source, pip, OS package pip

Desired Behaviour

Molecule test installs roles to .molecule/roles rather than /etc/ansible/roles

Actual Behaviour (Bug report only)

When using molecule 1.25.1 and ansible 2.4.1.0 when you run molecule test, galaxy roles will be installed to /etc/ansible/roles rather than .molecule/roles. You cannot override this in any way using ansible configuration. When rolling back to ansible 2.3.2.0 it works fine..

(FYI have to use molecule 1 due to testing windows boxes and serverspec not being supported in v2).

retr0h commented 6 years ago

This is by design. Roles are downloaded to .molecule/roles/ in v2.

V2 is not a straight port of v1. So it's behavior is different and cleaned up in many areas.

jamesla commented 6 years ago

I think I wasn't clear with the bug report it's definitely a bug not a question. Molecule 1 is not functioning anymore when using the galaxy dependency with ansible 2.4+

(ensure ansible 2.4+ is installed) To go into more detail:

molecule init --driver vagrant --verifier serverspec
create a requirements.yml file and add a ansible galaxy role (I used ansible-role-docker).
add requirements.yml to molecule.yml
add ansible-galaxy role to playbook.yml

molecule test
--> Destroying instances...
--> Downloading dependencies with 'galaxy'...
- changing role ansible-role-docker from master to master
- extracting ansible-role-docker to /etc/ansible/roles/ansible-role-docker
- ansible-role-docker (master) was installed successfully
--> Checking playbook's syntax...
ERROR! the role 'ansible-role-docker' was not found in /home/vagrant/Documents/test/roles:/home/vagrant/Documents/test/.molecule/.molecule/roles:/home/vagrant/Documents/test/.molecule/roles:/home/vagrant/Documents/test:/home/vagrant/Documents:/home/vagrant/Documents/test

The error appears to have been in '/home/vagrant/Documents/test/playbook.yml': line 5, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

    - role: test
    - role: ansible-role-docker
      ^ here

now downgrade to ansible 2.3.2.0

pip uninstall ansible && pip install ansible==2.3.2.0

and it works fine.

molecule test
--> Destroying instances...
--> Downloading dependencies with 'galaxy'...
- extracting ansible-role-docker to .molecule/roles/ansible-role-docker
- ansible-role-docker (master) was installed successfully
--> Checking playbook's syntax...

playbook: playbook.yml
--> Creating instances...
Bringing machine 'test' up with 'virtualbox' provider...
==> test: Importing base box 'trusty64'...
==> test: Matching MAC address for NAT networking...
==> test: Setting the name of the VM: test_test_1511058965115_80730
==> test: Clearing any previously set forwarded ports...
==> test: Clearing any previously set network interfaces...
==> test: Preparing network interfaces based on configuration...
    test: Adapter 1: nat
==> test: Forwarding ports...
    test: 22 (guest) => 2222 (host) (adapter 1)
==> test: Running 'pre-boot' VM customizations...
==> test: Booting VM...
==> test: Waiting for machine to boot. This may take a few minutes...
    test: SSH address: 127.0.0.1:2222
    test: SSH username: vagrant
    test: SSH auth method: private key
    test: Warning: Connection reset. Retrying...
    test:
    test: Vagrant insecure key detected. Vagrant will automatically replace
    test: this with a newly generated keypair for better security.
    test:
    test: Inserting generated public key within guest...
    test: Removing insecure key from the guest if it's present...
    test: Key inserted! Disconnecting and reconnecting using new SSH key...
==> test: Machine booted and ready!
==> test: Checking for guest additions in VM...
    test: The guest additions on this VM do not match the installed version of
    test: VirtualBox! In most cases this is fine, but in rare cases it can
    test: prevent things such as shared folders from working properly. If you see
    test: shared folder errors, please make sure the guest additions within the
    test: virtual machine match the version of VirtualBox you have installed on
    test: your host and reload your VM.
    test:
    test: Guest Additions Version: 4.3.36
    test: VirtualBox Version: 5.1
==> test: Setting hostname...
==> test: Mounting shared folders...
    test: /vagrant => /home/vagrant/Documents/test
==> test: Machine not provisioned because `--no-provision` is specified.
--> Starting Ansible Run...

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [test]

TASK [ansible-role-docker : Ensure old versions of Docker are not installed.] ***
skipping: [test] => (item=docker)
skipping: [test] => (item=docker-common)
skipping: [test] => (item=docker-engine)
...

Looks like a bug to me and the only difference I can see is that ansible 2.3 installs to .molecule/roles with and 2.4 or installs roles to /etc/ansible/roles.

Any ideas on how I can get this going with ansible 2.4?

retr0h commented 6 years ago

Hey @jamesla I can't seem to reproduce.

[jodewey:~/git/molecule/test/scenarios/requirements_file] [molecule-env-2.7.13] v1+* ± molecule --version
molecule, version 1.25.1
[jodewey:~/git/molecule/test/scenarios/requirements_file] [molecule-env-2.7.13] v1+ 1 ± molecule --debug dependency
--> Downloading dependencies with 'galaxy'...
DEBUG: COMMAND
/Users/jodewey/.pyenv/versions/molecule-env-2.7.13/bin/ansible-galaxy install --roles-path=.molecule/roles --force --role-file=requirements.yml
- changing role timezone from v1.1.0 to unspecified
- downloading role 'timezone', owned by yatesr
- downloading role from https://github.com/yatesr/ansible-timezone/archive/v1.1.0.tar.gz
- extracting timezone to /Users/jodewey/git/molecule/test/scenarios/requirements_file/.molecule/roles/timezone
- timezone (v1.1.0) was installed successfully
[jodewey:~/git/molecule/test/scenarios/requirements_file] [molecule-env-2.7.13] v1+ ± ansible --version
ansible 2.4.1.0
  config file = None
  configured module search path = [u'/Users/jodewey/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/jodewey/.pyenv/versions/2.7.13/envs/molecule-env-2.7.13/lib/python2.7/site-packages/ansible
  executable location = /Users/jodewey/.pyenv/versions/molecule-env-2.7.13/bin/ansible
  python version = 2.7.13 (default, May  4 2017, 15:45:50) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]

As you see molecule passes the -r flag to ansible-galaxy, which dictates the path the roles will be installed into.

jamesla commented 6 years ago

Very strange.... When I use that timezone role it works with 2.4.1.0 however when I use all of my other roles it still puts them into /etc/ansible/roles

Could you try it with this role:

- src: https://github.com/yatesr/ansible-timezone.git
  version: master
  name: timezone

- src: https://github.com/geerlingguy/ansible-role-docker.git
  version: master
  name: ansible-role-docker

Here is my output

molecule test
--> Destroying instances...
--> Downloading dependencies with 'galaxy'...
- extracting timezone to /home/vagrant/Documents/test/.molecule/roles/timezone
- timezone (master) was installed successfully
- changing role ansible-role-docker from master to master
- extracting ansible-role-docker to /etc/ansible/roles/ansible-role-docker
- ansible-role-docker (master) was installed successfully
--> Checking playbook's syntax...
ERROR! the role 'ansible-role-docker' was not found in /home/vagrant/Documents/test/roles:/home/vagrant/Documents/test/.molecule/.molecule/roles:/home/vagrant/Documents/test/.molecule/roles:/home/vagrant/Documents/test:/home/vagrant/Documents:/home/vagrant/Documents/test

The error appears to have been in '/home/vagrant/Documents/test/playbook.yml': line 5, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

    - role: test
    - role: ansible-role-docker
      ^ here

It is not this ansible-role-docker role specifically as I have about 10 roles in my project and 8 of them install to /etc/ansible/roles when using ansible 2.4.0.0+ but work fine on 2.3.2.0

retr0h commented 6 years ago

Can you do it with molecule --debug test and paste the relevant ansible-galaxy lines, along with the environment variables at the start of the debug output?

I still can't duplicate.

[jodewey:~/git/molecule/test/scenarios/requirements_file] [molecule-env-2.7.13] v1(+10/-2)+ ± rm .molecule/state.yml && molecule --debug dependency
--> Downloading dependencies with 'galaxy'...
DEBUG: COMMAND
/Users/jodewey/.pyenv/versions/molecule-env-2.7.13/bin/ansible-galaxy install --roles-path=.molecule/roles --force --role-file=requirements.yml
- changing role timezone from master to master
- extracting timezone to /Users/jodewey/git/molecule/test/scenarios/requirements_file/.molecule/roles/timezone
- timezone (master) was installed successfully
- extracting ansible-role-docker to /Users/jodewey/git/molecule/test/scenarios/requirements_file/.molecule/roles/ansible-role-docker
- ansible-role-docker (master) was installed successfully

Could it be you have a local ansible.cfg somewhere on your system, and ansible 2.4 is somehow merging those settings?

What does ansible-config dump --only-changed report?

Do you have any ANSIBLE_ vars defined in your environment?

jamesla commented 6 years ago

Thanks for looking into this. Here is a full run + downgrade showing it working.

[I] 16:20:32 ➜  test git:(master) ✗ ansible --version
ansible 2.4.1.0
  config file = None
  configured module search path = [u'/home/vagrant/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
[I] 16:20:35 ➜  test git:(master) ✗ molecule --debug test
--> Destroying instances...
--> Downloading dependencies with 'galaxy'...
DEBUG: COMMAND
/usr/local/bin/ansible-galaxy install --roles-path=.molecule/roles --force --role-file=requirements.yml
- extracting timezone to /home/vagrant/Documents/test/.molecule/roles/timezone
- timezone (master) was installed successfully
- changing role ansible-role-docker from master to master
- extracting ansible-role-docker to /etc/ansible/roles/ansible-role-docker
- ansible-role-docker (master) was installed successfully
--> Checking playbook's syntax...
DEBUG: COMMAND
/usr/local/bin/ansible-playbook playbook.yml --inventory-file=localhost, --syntax-check --limit=all --timeout=30 --become --diff
ERROR! the role 'ansible-role-docker' was not found in /home/vagrant/Documents/test/roles:/home/vagrant/Documents/test/.molecule/.molecule/roles:/home/vagrant/Documents/test/.molecule/roles:/home/vagrant/Documents/test:/home/vagrant/Documents:/home/vagrant/Documents/test

The error appears to have been in '/home/vagrant/Documents/test/playbook.yml': line 5, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

    - role: test
    - role: ansible-role-docker
      ^ here

[I] 16:20:45 ➜  test git:(master) ✗ sudo pip install ansible==2.3.2.0
The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/vagrant/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting ansible==2.3.2.0
  Downloading ansible-2.3.2.0.tar.gz (4.3MB)
    100% |████████████████████████████████| 4.3MB 357kB/s
Requirement already satisfied: jinja2 in /usr/local/lib/python2.7/dist-packages (from ansible==2.3.2.0)
Requirement already satisfied: PyYAML in /usr/local/lib/python2.7/dist-packages (from ansible==2.3.2.0)
Requirement already satisfied: paramiko in /usr/local/lib/python2.7/dist-packages (from ansible==2.3.2.0)
Requirement already satisfied: pycrypto>=2.6 in /usr/lib/python2.7/dist-packages (from ansible==2.3.2.0)
Requirement already satisfied: setuptools in /usr/local/lib/python2.7/dist-packages (from ansible==2.3.2.0)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python2.7/dist-packages (from jinja2->ansible==2.3.2.0)
Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python2.7/dist-packages (from paramiko->ansible==2.3.2.0)
Requirement already satisfied: cryptography>=1.1 in /usr/local/lib/python2.7/dist-packages (from paramiko->ansible==2.3.2.0)
Requirement already satisfied: six>=1.4.1 in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.1->paramiko->ansible==2.3.2.0)
Requirement already satisfied: cffi>=1.7; platform_python_implementation != "PyPy" in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.1->paramiko->ansible==2.3.2.0)
Requirement already satisfied: enum34; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.1->paramiko->ansible==2.3.2.0)
Requirement already satisfied: idna>=2.1 in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.1->paramiko->ansible==2.3.2.0)
Requirement already satisfied: asn1crypto>=0.21.0 in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.1->paramiko->ansible==2.3.2.0)
Requirement already satisfied: ipaddress; python_version < "3" in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.1->paramiko->ansible==2.3.2.0)
Requirement already satisfied: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=1.1->paramiko->ansible==2.3.2.0)
Installing collected packages: ansible
  Found existing installation: ansible 2.4.1.0
    Uninstalling ansible-2.4.1.0:
      Successfully uninstalled ansible-2.4.1.0
  Running setup.py install for ansible ... done
Successfully installed ansible-2.3.2.0
[I] 16:21:02 ➜  test git:(master) ✗ molecule --debug test
--> Destroying instances...
--> Downloading dependencies with 'galaxy'...
DEBUG: COMMAND
/usr/local/bin/ansible-galaxy install --roles-path=.molecule/roles --force --role-file=requirements.yml
- extracting timezone to .molecule/roles/timezone
- timezone (master) was installed successfully
- extracting ansible-role-docker to .molecule/roles/ansible-role-docker
- ansible-role-docker (master) was installed successfully
--> Checking playbook's syntax...
DEBUG: COMMAND
/usr/local/bin/ansible-playbook playbook.yml --inventory-file=localhost, --syntax-check --limit=all --timeout=30 --become --diff

playbook: playbook.yml
--> Creating instances...
Bringing machine 'test' up with 'virtualbox' provider...
==> test: Machine not provisioned because `--no-provision` is specified.
DEBUG: ANSIBLE ENVIRONMENT
ANSIBLE_CONFIG: .molecule/ansible.cfg
ANSIBLE_FORCE_COLOR: 'true'
ANSIBLE_HOST_KEY_CHECKING: 'false'
ANSIBLE_SSH_ARGS: -o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o ControlPersist=60s
  -o IdentitiesOnly=yes

--> Starting Ansible Run...
DEBUG: COMMAND
/usr/local/bin/ansible-playbook playbook.yml --inventory-file=.molecule/ansible_inventory --connection=ssh --limit=all --user=vagrant --timeout=30 --become --diff

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
kkkkok: [test]

TASK [ansible-role-docker : Ensure old versions of Docker are not installed.] ***
kskipping: [test] => (item=docker)
kskipping: [test] => (item=docker-common)
skipping: [test] => (item=docker-engine)

TASK [ansible-role-docker : Add Docker GPG key.] *******************************
kskipping: [test]

TASK [ansible-role-docker : Add Docker repository.] ****************************
kskipping: [test]

TASK [ansible-role-docker : Configure Docker Edge repo.] ***********************
kskipping: [test]

TASK [ansible-role-docker : Configure Docker Test repo.] ***********************
kskipping: [test]

TASK [ansible-role-docker : Ensure old versions of Docker are not installed.] ***
kkkok: [test] => (item=docker)
ok: [test] => (item=docker-engine)

TASK [ansible-role-docker : Ensure depdencies are installed.] ******************
ok: [test] => (item=[u'apt-transport-https', u'ca-certificates'])

TASK [ansible-role-docker : Add Docker apt key.] *******************************

retr0h commented 6 years ago

Could it be you have a local ansible.cfg somewhere on your system, and ansible 2.4 is somehow merging those settings?

What does ansible-config dump --only-changed report (2.4 only command)?

Do you have any ANSIBLE_ vars defined in your environment?

For SnGs can you move /etc/ansible/ on your system elsewhere so /etc/ansible doesn't exist at all, just to see what happens?

jamesla commented 6 years ago

I just created a repo of the demo i have been using to confirm that it wasn't my project

https://github.com/jamesla/broken-molecule-test

I even checked it out brand new and molecule test'd it and it did the same thing.

Surely this will break for you!

jamesla commented 6 years ago

rm -rf /etc/ansible

fixed it

!!!!!! thanks for your help dude

Hopefully someone else will find this thread and it won't be a complete waste of time...

retr0h commented 6 years ago

!!!!!! thanks for your help dude

Awesome, glad we figured it out!

jmartinspt commented 3 years ago

I recently started using the molecule to integrate my project's Ansible roles into the CI/CD process and quickly ran into two problems:

Duplication of the dependency file, for the use of the molecule (project/requirements.yml) and the role itself (project/meta/requirements.yml)
Inability to force the download of roles if they had already been installed previously, being cached in the folder ~/.cache.

After much research and many attempts, I found the following solution:

.env.yml

---
ANSIBLE_ROLES_PATH: galaxy_roles

.gitignore

(...)
galaxy_roles/
(...)

molecule/default/molecule.yml

---
dependency:
  name: shell
  command: ansible-galaxy install -r meta/requirements.yml -p galaxy_roles/ --force
driver:
  name: docker
platforms:
  - name: molecule-ubuntu
    image: ubuntu
    pre_build_image: true
provisioner:
  name: ansible
verifier:
  name: ansible
lint: |
  set -e
  yamllint .
  ansible-lint

This way, just by changing dependency to shell and executing the command to install roles to a different folder (always forcing the installation) I can use the same dependency file for the molecule and the role itself. So I always have the most up-to-date dependencies and avoid code duplication.

I hope it's useful to someone.

ansible / molecule

Molecule installs ansible dependencies to the wrong location using ansible 2.4 #1042

Issue Type

Molecule and Ansible details

Desired Behaviour

Actual Behaviour (Bug report only)