[BUG] The whole mongodb code is a mess

dhs-rec commented 9 months ago

Description Using MongoDB for tops and pillars is currently broken, due to differences in how they connect to the database.

pillar/mongo.py can use mongo.uri, but throws an exception if mongo.host is also provided
tops/mongo.py ignores mongo.uri, so it is required to provide all the mongo.* options
none of the above provides an authSource parameter, so pymongo defaults to admin, resulting in authentication failures if the DB user has been created in the salt DB.
modules/mongodb.py provides a method for connecting, but that's not used
in general, the code for connecting to the DB looks quite different in pillar/mongo.py and tops/mongo.py
documentation doesn't tell anything about pymongo versions (Hint: Versions >= 3.13.0 don't work)

Setup

ext_pillar:
  - ...
  - mongo: {collection: pillar}
...
master_tops:
  mongo:
    collection: tops
    id_field: _id
    states_field: states
    environment_field: environment
mongo.db: salt
mongo.host: localhost
mongo.indexes: true
mongo.password: Secret
mongo.user: salt

Please be as specific as possible and give set-up details.

[ ] on-prem machine
[ ] VM (Virtualbox, KVM, etc. please specify)
[X] VM running on a cloud service, please be explicit and add details
[ ] container (Kubernetes, Docker, containerd, etc. please specify)
[ ] or a combination, please be explicit
[ ] jails if it is FreeBSD
[ ] classic packaging
[X] onedir packaging
[ ] used bootstrap to install

Steps to Reproduce the behavior

Install/update Salt master to 3006.7 using onedir packages on Ubuntu Jammy
Install pymongo==3.13.0 using Salts internal Python/pip

Expected behavior Pillars/Tops in MongoDB work as before

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3006.7 Python Version: Python: 3.10.13 (main, Feb 19 2024, 03:34:22) [GCC 11.2.0] Dependency Versions: cffi: 1.14.6 cherrypy: unknown dateutil: 2.8.1 docker-py: Not Installed gitdb: 4.0.11 gitpython: 3.1.42 Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.0.2 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 22.0 pycparser: 2.21 pycrypto: 3.20.0 pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 2.3.1 PyYAML: 6.0.1 PyZMQ: 23.2.0 relenv: 0.15.1 smmap: 5.0.1 timelib: 0.2.4 Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: ubuntu 20.04.2 jammy locale: utf-8 machine: aarch64 release: 6.2.0-1018-aws system: Linux version: Ubuntu 20.04.2 jammy ```

Additional context I have fixed/worked around point 3 above by adding an authSource=db parameter to pymongo.MongoClient() in both pillar/mongo.py and tops/mongo.py. However, this is not as clean as it could be.

cplmayo commented 9 months ago

Additional context I have fixed/worked around point 3 above by adding an authSource=db parameter to pymongo.MongoClient() in both pillar/mongo.py and tops/mongo.py. However, this is not as clean as it could be.

I am currently trying to implement a similar patch in my project; would be willing to provide an example of what you had to do to get it working? I've been poking at the files but I'm getting an error.

I manually rolled back to earlier than the version identified and it's working. So please ignore; and sorry for the spam emails.

dhs-rec commented 2 months ago

@dwoz, any chance to get this into 3007, too?

saltstack / salt

[BUG] The whole mongodb code is a mess #66122