Binding of oslo.config causes oslo.log to not be usable within a pack

[tonybaloney]

ISSUE TYPE

• Bug Report

STACKSTORM VERSION

2.7.2

OS / ENVIRONMENT / INSTALL METHOD

one-line install

SUMMARY

StackStorm requires a specific version of oslo.config. This version does not support mutable as an argument.

The newer versions of oslo.config do.

The issue is that when installing a newer version of oslo.log, it will depend upon this setting. So any packs which inadvertantly use oslo.log will crash.

STEPS TO REPRODUCE

1. Create a pack with the dependency oslo.log>1.8.0
2. Create an action with

from oslo_log import log as logging
LOG = logging.getLogger(__name__)

EXPECTED RESULTS

No error :-)

ACTUAL RESULTS

Traceback (most recent call last):
  File "/opt/stackstorm/runners/python_runner/python_runner/python_action_wrapper.py", line 320, in <module>
    obj.run()
  File "/opt/stackstorm/runners/python_runner/python_runner/python_action_wrapper.py", line 178, in run
    action = self._get_action_instance()
  File "/opt/stackstorm/runners/python_runner/python_runner/python_action_wrapper.py", line 231, in _get_action_instance
    raise exc_cls(msg)
TypeError: Failed to load action class from file "/opt/stackstorm/packs/infoblox/actions/get_object.py" (action file most likely doesn't exist or contains invalid syntax): __init__() got an unexpected keyword argument 'mutable'

Traceback (most recent call last):
  File "/opt/stackstorm/runners/python_runner/python_runner/python_action_wrapper.py", line 224, in _get_action_instance
    actions_cls = action_loader.register_plugin(Action, self._file_path)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/util/loader.py", line 173, in register_plugin
    module = imp.load_source(module_name, plugin_abs_file_path)
  File "/opt/stackstorm/packs/infoblox/actions/get_object.py", line 1, in <module>
    from lib.action import InfobloxBaseAction
  File "/opt/stackstorm/packs/infoblox/actions/lib/action.py", line 2, in <module>
    from infoblox_client import connector
  File "/opt/stackstorm/virtualenvs/infoblox/lib/python2.7/site-packages/infoblox_client/connector.py", line 27, in <module>
    from oslo_log import log as logging
  File "/opt/stackstorm/virtualenvs/infoblox/lib/python2.7/site-packages/oslo_log/log.py", line 47, in <module>
    from oslo_log import _options
  File "/opt/stackstorm/virtualenvs/infoblox/lib/python2.7/site-packages/oslo_log/_options.py", line 39, in <module>
    help='If set to true, the logging level will be set to '
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/oslo_config/cfg.py", line 982, in __init__
    super(BoolOpt, self).__init__(name, type=types.Boolean(), **kwargs)
TypeError: __init__() got an unexpected keyword argument 'mutable'
ACTION INPUT
No Input

[tonybaloney]

the issue is this: https://github.com/StackStorm/st2/blob/master/requirements.txt#L25

[Kami]

We intentionally pin it to an older version because we had issues with upgrades in the past (in fact, we had a lot of issues with upgrading most of the oslo libraries so we tried to not touch and update it because it worked).

We can look into upgrading it again and hope it doesn't break anything this time.

Managing dependencies is always a PITA, especially when you are dealing with user content (packs). That's why we try to isolate user content as much as possible in a virtual environment, but sadly some system dependencies are still shared and used by the pack virtual environments.

[Kami]

I looked into upgrading oslo.config and oslo.utils, but it's sadly a big mess.

Upgrading to latest version of oslo config, substantially slows importing of the modules which rely on oslo.config. For example, importing of /data/stanley/contrib/runners/python_runner/python_runner/python_action_wrapper.py goes from 0.12s to 0.50s which is unacceptable at the moment.

I tried to dig in, but it's a big rabbit hole and not much we can do about it, because we don't directly control code in oslo.config.

So sadly, for the time being, we can't upgrade because it negatively affects the performance.

[Kami]

@tonybaloney As far as a work-around goes - if you need to use newer version of oslo.config in your pack, you can try adding a more recent version of oslo.config in your pack requirements.txt.

I believe that should work.

Can you please also clarify how you encountered that problem?

All the StackStorm code should work just fine with the oslo.config version we currently use so I assume it affects some custom pack which relies on newer version of oslo.log (or some openstack library) or similar.

[LindsayHill]

@Kami see the Infoblox pack https://github.com/StackStorm-Exchange/stackstorm-infoblox/blob/master/requirements.txt

Note the changes made https://github.com/tonybaloney/infoblox-client/compare/infobloxopen:master...master

[lorddaren]

@Kami I am looking at getting the oslo.config upgraded as it supports different configuration backends (like remote_file). I have created a Hashicorp Vault plugin that allows me to pull all the information out of Vault instead of storing the credentials in a plain text file.

What tests did you run to view the performance hit? I would like to try and observe if the latest release of oslo.conf has this issue.

[Kami]

@Xorso Updating to a new release added a huge overhead to just importing that module which subsequently slowed every service startup and initialization (https://github.com/StackStorm/st2/issues/4160#issuecomment-394386433).

The most critical part was and is the Python runner action execution which must be fast. There we try to keep imports at the minimum and avoid any expensive imports.

Should be able to check that using profimp or similar by using different oslo.config releases - https://github.com/StackStorm/st2/blob/master/contrib/runners/python_runner/tests/integration/test_python_action_process_wrapper.py#L17.

Maybe they fixed / improved on it in a recent release, it has been a while since I last looked into it.

[lorddaren]

Looks like there still is a bit of a difference. profimp is reporting that 6.6 is 243.91ms compared to 41.67ms for 1.6.1. That seems to be at the .24 seconds you mentioned earlier but seems faster than the .5s. That was running it on my MacBook Pro. I am guessing that big of an increase is something that still isn't acceptable?

[AndroxxTraxxon]

Hello from the future! This issue is rearing its head for additional reasons now. If you configure a pack with python version 3.10 or newer, you run into import errors because the required version of oslo.config (1.12.x) is using imports from the builtin collections module that were deprecated in Python 3.3 (in 2012), and removed in Python 3.10.

This StackOverflow post pretty much sums up the issue

Currently, no installations run (by default) with Python 3.10 or newer, but manually configuring /etc/st2/st2.conf to have actionrunner.python_binary to point to a binary versioned 3.10, 3.11, or newer of these reveals this issue. Since #4571 was merged, this can be currently overcome by just installing a newer version of oslo.config at the pack level (via the requirements.txt -- version 8.8.0 seems to have worked for me) This will need to be addressed whenever this project decides to move beyond Python3.9.