search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.22k stars 5.49k forks source link

[BUG] Elasticsearch module is broken with latest elasticsearch library #61973

Open er0k opened 2 years ago

er0k commented 2 years ago

Description Trying to use the elasticsearch returner with the latest elasticsearch-py library (>= 8.x) fails to initialize, because salt's elasticsearch module is attempting to load a deprecated connection module

Setup I'm trying to use the elasticsearch returner, following the instructions here

The elasticsearch return always fails:

[DEBUG   ] Could not LazyLoad elasticsearch.index_exists: 'elasticsearch' __virtual__ returned False: Cannot load module elasticsearch: elasticsearch libraries not found
[DEBUG   ] LazyLoaded nagios.list_plugins                                                                                                                                                                                                        
[DEBUG   ] Could not LazyLoad elasticsearch.returner: 'elasticsearch.returner' is not available.                                                                                                                                                 
[ERROR   ] Returner elasticsearch.returner could not be loaded: 'elasticsearch.returner' is not available.

elasticsearch-py is installed:

$ pip freeze | grep elastic
elastic-transport==8.1.2
elasticsearch==8.1.2

Salt's elasticsearch module is failing at this line:

https://github.com/saltstack/salt/blob/master/salt/modules/elasticsearch.py#L62

from elasticsearch import RequestsHttpConnection

because RequestsHttpConnection no longer exists

Python 3.8.10 (default, Mar 15 2022, 12:22:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import elasticsearch
>>> from elasticsearch import RequestsHttpConnection
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'RequestsHttpConnection' from 'elasticsearch' (/usr/local/lib/python3.8/dist-packages/elasticsearch/__init__.py)

According to the release notes, the connection module has been removed and replaced with the elastic-transport package

Steps to Reproduce the behavior

  1. Install the latest elasticsearch module (8.x or greater)
  2. Try to use the elasticsearch returner

Expected behavior Elasticsearch module should load successfully.

Versions Report

Salt Version: Salt: 3004.1 Dependency Versions: cffi: 1.14.2 cherrypy: unknown dateutil: 2.7.3 docker-py: Not Installed gitdb: 2.0.6 gitpython: 3.0.7 Jinja2: 2.11.3 libgit2: 1.0.0 M2Crypto: 0.31.0 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.20 pycrypto: Not Installed pycryptodome: 3.6.1 pygit2: 1.2.1 Python: 3.8.10 (default, Mar 15 2022, 12:22:08) python-gnupg: 0.4.5 PyYAML: 5.3.1 PyZMQ: 18.1.1 smmap: 2.0.5 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.2 System Versions: dist: ubuntu 20.04 focal locale: utf-8 machine: x86_64 release: 5.4.0-107-generic system: Linux
welcome[bot] commented 2 years ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

trudesea commented 1 year ago

I'm getting the same thing and reverting to 7.17 of the client resolves it, but since elastic is on version 8.8 now I wonder how long this will be a viable workaround.

I see this has been approved over a year ago, but it is actively being worked on?

msvihra commented 1 year ago

We are also facing problems with this module, since ES 8 is current mainline version we would appreciate update regarding this problem, because some part of config management relies on this module. Thanks!

jbrouwers commented 6 months ago

This is still an issue on salt 3006.8 with elasticsearch-py 8.13.2