search

ansible-collections / community.kubernetes

Kubernetes Collection for Ansible
https://galaxy.ansible.com/community/kubernetes
GNU General Public License v3.0
265 stars 106 forks source link

PoC: ansible_module.turbo integration #270

Closed goneri closed 3 years ago

goneri commented 3 years ago

Aloha!

This PoC PR introduces the use of ansible_module.turbo[1] to speed up the k8s_info module. It breaks everything else :-).

This two recording runs the k8s_facts 20 times in row. The first one use the current main branch, the second use this PR.

This PR improves performance at two level.

Reuse the same client object

With ansible_module.turbo, the module are actually run in a long running Python process. Thanks to it, we can cache large Python objects. In this case, we focus on the openshift.dynamic.DynamicClient instance. We use the lru_cache[2] decorator to ensure we will always get the same python object, as soon as we use the same authentication parameters. Everytime we call get_api_client() in a new task, we get the same Python client that is already in memory.

Speed up the import

The loading of the OpenShift SDK take a lot of time, and this is a penality we pay everytime we start a new task:

$ time python3 -c "from openshift.dynamic import DynamicClient"

real    0m0.653s
user    0m0.966s
sys     0m0.469s

This PR try to defeer all the non-critical imports. This allow us to load the modules once in the long running Python process. We can reuse the same module task after task.

About this PR changes

The use of object complexify the ansible_module.turbo integration. Currently, everytime we instanciate the K8sAnsibleMixin object we load all the SDK. In addition, everytime we run a task, we end up with a new K8sAnsibleMixin instance that can hardly cache.

These are the reasons why the flatten the nested structure.

If you like the approach, I will be happy to start a larger refactoring to generalize the use of ansible_module.turbo.

codecov[bot] commented 3 years ago

Codecov Report

Merging #270 (9e3307f) into main (1513bce) will increase coverage by 0.14%. The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #270      +/-   ##
==========================================
+ Coverage   37.00%   37.15%   +0.14%     
==========================================
  Files           3        4       +1     
  Lines         727      716      -11     
  Branches      144      144              
==========================================
- Hits          269      266       -3     
+ Misses        409      401       -8     
  Partials       49       49
Impacted Files Coverage Δ
...ommunity/kubernetes/plugins/module_utils/common.py 35.74% <0.00%> (-0.19%) :arrow_down:
...ity/kubernetes/plugins/module_utils/args_common.py 100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1513bce...3e17db2. Read the comment docs.

goneri commented 3 years ago

Another example, this time with k8s_service and k8s_info. I run each module 20 times.

goneri commented 3 years ago

See: https://github.com/ansible-collections/community.kubernetes/pull/313