saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

[BUG] `mysql` module fails with `mysqlclient>=1.3.13` #65108

Closed 11qeq11 closed 1 year ago

11qeq11 commented 1 year ago

Description After upgrade to mysqlclient==1.3.13, which is one of the requirements for Salt's mysql module to work according to documentation, module calling throws AttributeError: module 'MySQLdb' has no attribute 'err'. Reverting to mysqlclient=1.3.12 fixes the issue.

Setup Info from VM (CentOS 7):

$ pip3 --version
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)

Info from parent host (Arch Linux):

$ VBoxManage -version
7.0.10r158379
$ vagrant --version
Vagrant 2.3.7

Steps to Reproduce the behaviour

  1. Try to install mysqlclient==1.3.13 with $ sudo pip3 install mysqlclient==1.3.13.
  2. Considering that MySQL service is up and running, run any query using mysql module, in my case:
    $ sudo salt-call mysql.query mysql "PURGE BINARY LOGS BEFORE NOW();"
  3. Observe traceback:
Full trace ```python [ERROR ] An un-handled exception was caught by Salt's global exception handler: AttributeError: module 'MySQLdb' has no attribute 'err' Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 436, in _connect dbc = MySQLdb.connect(**connargs) File "/usr/lib64/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect return Connection(*args, **kwargs) File "/usr/lib64/python3.6/site-packages/MySQLdb/connections.py", line 247, in __init__ self.converter[t].append((None, string_decoder)) AttributeError: type object 'str' has no attribute 'append' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/bin/salt-call", line 11, in load_entry_point('salt==3005.2', 'console_scripts', 'salt-call')() File "/usr/lib/python3.6/site-packages/salt/scripts.py", line 441, in salt_call client.run() File "/usr/lib/python3.6/site-packages/salt/cli/call.py", line 50, in run caller.run() File "/usr/lib/python3.6/site-packages/salt/cli/caller.py", line 95, in run ret = self.call() File "/usr/lib/python3.6/site-packages/salt/cli/caller.py", line 203, in call self.opts, data, func, args, kwargs File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__ return self.loader.run(run_func, *args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1228, in run return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs) File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run return callable(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1243, in _run_as return _func_or_method(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 10, in execute return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__ return self.loader.run(run_func, *args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1228, in run return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs) File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run return callable(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1243, in _run_as return _func_or_method(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 777, in query dbc = _connect(**connection_args) File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 442, in _connect except MySQLdb.err.InternalError as exc: AttributeError: module 'MySQLdb' has no attribute 'err' Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 436, in _connect dbc = MySQLdb.connect(**connargs) File "/usr/lib64/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect return Connection(*args, **kwargs) File "/usr/lib64/python3.6/site-packages/MySQLdb/connections.py", line 247, in __init__ self.converter[t].append((None, string_decoder)) AttributeError: type object 'str' has no attribute 'append' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/bin/salt-call", line 11, in load_entry_point('salt==3005.2', 'console_scripts', 'salt-call')() File "/usr/lib/python3.6/site-packages/salt/scripts.py", line 441, in salt_call client.run() File "/usr/lib/python3.6/site-packages/salt/cli/call.py", line 50, in run caller.run() File "/usr/lib/python3.6/site-packages/salt/cli/caller.py", line 95, in run ret = self.call() File "/usr/lib/python3.6/site-packages/salt/cli/caller.py", line 203, in call self.opts, data, func, args, kwargs File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__ return self.loader.run(run_func, *args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1228, in run return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs) File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run return callable(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1243, in _run_as return _func_or_method(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/executors/direct_call.py", line 10, in execute return func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__ return self.loader.run(run_func, *args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1228, in run return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs) File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run return callable(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1243, in _run_as return _func_or_method(*args, **kwargs) File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 777, in query dbc = _connect(**connection_args) File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 442, in _connect except MySQLdb.err.InternalError as exc: AttributeError: module 'MySQLdb' has no attribute 'err' ```
  1. Downgrade mysqlclient with $ sudo pip3 install mysqlclient==1.3.12.
  2. Command's working.
local:
    ----------
    query time:
        ----------
        human:
            0.5ms
        raw:
            0.00046
    rows affected:
        0

Expected behaviour

I'd expect that mysql module is compatible with mysqlclient on versions mysqlclient>=1.3.13, I think it's not a problem of my setup. Perhaps diff in https://github.com/PyMySQL/mysqlclient/compare/1.3.12...1.3.13 will be helpful to fix this incompatibility, until then I can hard-code 1.3.12.

Versions Report

salt --versions-report Master: ```yaml Salt Version: Salt: 3005.2 Dependency Versions: cffi: Not Installed cherrypy: unknown dateutil: 2.8.2 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.11.1 libgit2: Not Installed M2Crypto: 0.35.2 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 3.6.8 (default, Jun 20 2023, 11:53:23) python-gnupg: 0.5.1 PyYAML: 3.13 PyZMQ: 20.0.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.1.4 System Versions: dist: centos 7 Core locale: UTF-8 machine: x86_64 release: 3.10.0-1160.95.1.el7.x86_64 system: Linux version: CentOS Linux 7 Core ``` Minion: ```yaml Salt Version: Salt: 3005.2 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: Not Installed docker-py: Not Installed gitdb: 4.0.9 gitpython: Not Installed Jinja2: 2.11.1 libgit2: Not Installed M2Crypto: 0.35.2 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: 1.3.12 pycparser: Not Installed pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 3.6.8 (default, Jun 20 2023, 11:53:23) python-gnupg: Not Installed PyYAML: 3.13 PyZMQ: 20.0.0 smmap: 5.0.0 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.1.4 System Versions: dist: centos 7 Core locale: UTF-8 machine: x86_64 release: 3.10.0-1160.95.1.el7.x86_64 system: Linux version: CentOS Linux 7 Core ```
welcome[bot] commented 1 year ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

OrangeDog commented 1 year ago

The actual problem is:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/modules/mysql.py", line 436, in _connect
    dbc = MySQLdb.connect(**connargs)
  File "/usr/lib64/python3.6/site-packages/MySQLdb/__init__.py", line 85, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/MySQLdb/connections.py", line 247, in __init__
    self.converter[t].append((None, string_decoder))
AttributeError: type object 'str' has no attribute 'append'

That appears to be caused by https://github.com/PyMySQL/mysqlclient/pull/215

I don't think there's anything Salt can do about that. You should file an issue with mysqlclient.

11qeq11 commented 1 year ago

@OrangeDog Got it, thank you!

Looks like this problem is present in mysqlclient v1.3.13 and v1.3.14 only, upgrading further actually solves the issue as well, and essentially I noticed it only because RPM package python36-mysql from CentOS7 baserepo seems to contain exactly mysqlclient v1.3.14, which can be seen here.