On systems with slow disks, Ansible 2.10 runs generally much slower than 2.9

[geerlingguy]

SUMMARY

At first, I thought this may just be a problem that was caused by 2.10 using collections and shipping with dozens of collections out of the box when you pip install it now.

But after exploring further, I found that basic ansible commands like ansible --version are 3x slower than Ansible 2.9, even if I'm just installing and using ansible-base, with no collections installed.

Note that these tests were done on a Raspberry Pi 4 (after noticing it took about 2 minutes to run ansible --version on my Pi Zero after upgrading to 2.10). I haven't yet tested on my Mac, where the system's blazing-fast NVM drive and i9 CPU will make the absolute numbers much better—but I would like to see if the relative performance difference is the same.

ISSUE TYPE

• Bug Report

COMPONENT NAME

ansible (and other ansible-* commands)

ANSIBLE VERSION

$ time ansible --version
ansible 2.10.1
  config file = /home/pi/pi-webcam/ansible.cfg
  configured module search path = ['/home/pi/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.7.3 (default, Jul 25 2020, 13:03:44) [GCC 8.3.0]

CONFIGURATION

N/A

OS / ENVIRONMENT

Linux (Debian 10, Raspberry Pi OS)

STEPS TO REPRODUCE

pip3 install -y ansible
time ansible --version # run 4 times, discard 1st result
pip3 uninstall -y ansible ansible-base

pip3 install -y ansible-base
time ansible --version # run 4 times, discard 1st result
pip3 uninstall -y ansible ansible-base

pip3 install -y ansible==2.9.*
time ansible --version # run 4 times, discard 1st result
pip3 uninstall -y ansible ansible-base

EXPECTED RESULTS

2.10.1 version should be at least similar, maybe a little slower, but not twice or three times slower.

ACTUAL RESULTS

Benchmarking results:

Ansible version	Pip package size	Time	Delta vs 2.9
2.9.13	16.2MB	2.09s	-
2.10.0 (ansible)	43.1MB	6.09s	3x slower
2.10.1 (ansible-base)	1.9MB	6.33s	3x slower

[mkrizek]

Duplicate of https://github.com/ansible/ansible/issues/71940

[mkrizek]

Related https://github.com/ansible/ansible/pull/71948

[geerlingguy]

71940 didn't have a lot of helpful info about overcoming the problem. If possible, I'd like to go through the steps to remediate this issue in here before it's closed/locked :)

[sivel]

Generally speaking, you will need to ensure you have libyaml and libyaml-dev installed at the time that you pip install pyyaml. If you have already installed pyyaml, you'd need to ensure that you remove the pip cache (~/.cache/pip), so that the prior built and cached wheel isn't used. Otherwise you can use pip install --no-cache-dir -I pyyaml which may also work to bypass the cache and force install pyyaml.

You can check if you have libyaml support in pyyaml using python -c 'import _yaml'. If that succeeds you have it, if it fails you do not.

[geerlingguy]

On the Raspberry Pi (running Debian 10), I did the following to re-install Ansible with pyyaml/libyaml support:

# The first step would not be required if Ansible is not already installed.
pip3 uninstall pyyaml ansible ansible-base
sudo apt install -y libyaml-dev
pip3 install --no-cache-dir --no-binary pyyaml ansible

Ansible version	Pip package size	Time	Delta vs 2.9
2.9.13	16.2MB	2.09s	-
2.10.0 (ansible)	43.1MB	1.55s	faster, yay!
2.10.1 (ansible-base)	1.9MB	1.55s	faster, yay!

You can check if you have libyaml support in pyyaml using python -c 'import _yaml'. If that succeeds you have it, if it fails you do not.

And it looks like if https://github.com/ansible/ansible/pull/71948 gets merged, then the ansible --version output would be an easier indicator to check.

Also, on my Mac, I noticed I had it installed previously via brew install libyaml. On RHEL/Fedora/CentOS it seems like it would be libyaml-devel.

[geerlingguy]

The problem of pyyaml not using the libyaml bindings may be an upstream issue with running Ansible/PyYAML/Python on ARM: https://github.com/yaml/pyyaml/issues/445

[geerlingguy]

All right, I discovered the root cause of the pyyaml installation difference was a wheel built by piwheels that lacks support for libyaml and bypasses setup.py install running. There's an open issue for it here: https://github.com/piwheels/packages/issues/130. Anyways, the fix was to run:

sudo pip3 uninstall -y pyyaml
sudo pip3 install --no-cache-dir --no-binary pyyaml pyyaml

[gundalow]

Docs don't appear to talk about libyaml from a user point of view.

only mention I can see is in docs/docsite/rst/community/documentation_contributions.rst

[geerlingguy]

Ugh, I'm realizing all my Ansible Docker image builds need to be updated to install libyaml-devel / libyaml-dev, as I never had done that before 2.10 came out. They are very slow even on my core i9 Mac, with operations like just running ansible taking 2-3 seconds.

Does pyyaml have to manually be installed via pip as well, or does Ansible automatically add that dep if libyaml is present? (Nope, but you do need to reinstall, as stated earlier, without using cache, if you want it to pick up the libyaml bindings.)

[geerlingguy]

I'm on a new M1 Mac, and trying to install Ansible in any way that gets libyaml support, but I seem to keep failing (see point 2 here: https://github.com/geerlingguy/mac-dev-playbook/issues/100#issuecomment-848390871).

Here's what I tried:

pip3 uninstall ansible ansible-core pyyaml
rm -rf ~/.cache/pip
pip3 install --no-cache-dir --no-binary pyyaml pyyaml
pip3 install ansible

But after multiple attempts...

 21:08:49 ~ $ ansible --version
ansible [core 2.11.1] 
  config file = None
  configured module search path = ['/Users/jgeerling/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/jgeerling/Library/Python/3.8/lib/python/site-packages/ansible
  ansible collection location = /Users/jgeerling/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/jgeerling/Library/Python/3.8/bin/ansible
  python version = 3.8.2 (default, Apr  8 2021, 23:19:18) [Clang 12.0.5 (clang-1205.0.22.9)]
  jinja version = 3.0.1
  libyaml = False

[geerlingguy]

Finally figured out the issue—had to clear out some $PATH settings that were still causing Ansible to be installed using system Python instead of the version of Python I installed via Homebrew (alongside libyaml: brew install python3 libyaml).

And now I am able to get libyaml support on my ARM64 M1 Macs! It just takes a lot of extra work since libyaml doesn't come with anything I use by default :(

[geerlingguy]

I wrote up my findings over on my blog: Ansible might be running slow if libyaml is not available. (Basically summarizing all the comments for Pi and macOS from above.)

[sivel]

Is there something further we need to do in this issue? Doesn't seem like this is really an ansible issue.

[geerlingguy]

@sivel - I think somewhere @gundalow had mentioned we don't really have any documentation about this anywhere; to help people not think Ansible is just slow by default, it might be nice to at least have some sort of documentation to reference (like "optimizing Ansible" or "common performance issues") where this is mentioned.

Otherwise I'm okay with closing the issue, as at least for me I figured out how to fix the issue on both environments where I use Ansible the most—the annoying thing is unless you explicitly know to install libyaml before installing Ansible, you'll never benefit from it being present and just live with Ansible being slower :)

[arsenicks]

Do you guy's know if Ansible 2.9 have the same performance impact if the python -c 'import _yaml' test fail ?

Thanks!

[geerlingguy]

@arsenicks - Ansible 2.9 should not have this issue (AFAICT)—it's only 2.10 and newer (and by extension, Ansible 3.0.0 and newer).

[bcoca]

This issue is probably present since the 1.x versions of Ansible (always faster with the C YAML parser), the reason this became a lot more noticeable in 2.10 is the inclusion of a large YAML file in core that is always loaded (ansible_builtin_runtime.yml).

You probably would have seen another such slow down in 2.4 when we added the config system, which relies on another always loaded YAML file (base.yml), though this one is smaller than the previous one.

Anyone with large YAML inventories or group/host_vars would have noticed the difference even w/o accounting for the 2 files I just mentioned.

[arsenicks]

Anyone with large YAML inventories or group/host_vars would have noticed the difference even w/o accounting for the 2 files I just mentioned.

Thanks, We do have a pretty huge inventory, I'll test it with a new venv on our tower and see the differences.