Closed alidrsn closed 3 years ago
Please install and use elastalert in a python virtual environment dedicated to elastalert. I don't think we have guaranteed operation in any other way. I'm a passerby and have nothing to do with this project.
example
# Python Create virtual environment
python3 -m venv ~/elastalert-venv/
# Python Virtual environment switching
source ~/elastalert-venv/bin/activate
# ElastAlert Install
pip install elastalert
Hi,
Actually, i did not anything different from your instructions.
These are sources that i use,
OS: Red hat 8 Phyton: 3.6 Elasticsearch: 7.5
I saw similar issues that is closed before without any upgrade comments like #3112
@alidrsn
AWS EC2 t3.medium OS : CentOS 8 Elasticsearch 7.9.3
There was no problem.
sudo dnf -y update
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y docker-ce docker-ce-cli
sudo systemctl start docker
sudo usermod -a -G docker $USER
sudo systemctl enable docker
sudo dnf group install "Development Tools"
sudo dnf -y install python3-pip python36 python3-devel python3-setuptools python3-libs libffi-devel openssl-devel
sudo curl -L https://github.com/docker/compose/releases/download/1.28.5/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
# ssh reconnect
python3 -m venv ~/myvenv/
source ~/myvenv/bin/activate
pip install --upgrade pip
pip install setuptools
pip install elastalert
Collecting elastalert
Downloading elastalert-0.2.4.tar.gz (128 kB)
|████████████████████████████████| 128 kB 21.5 MB/s
Collecting apscheduler>=3.3.0
Downloading APScheduler-3.7.0-py2.py3-none-any.whl (59 kB)
|████████████████████████████████| 59 kB 10.5 MB/s
Collecting aws-requests-auth>=0.3.0
Downloading aws_requests_auth-0.4.3-py2.py3-none-any.whl (6.8 kB)
Collecting blist>=1.3.6
Downloading blist-1.3.6.tar.gz (122 kB)
|████████████████████████████████| 122 kB 51.2 MB/s
Collecting boto3>=1.4.4
Downloading boto3-1.17.29-py2.py3-none-any.whl (131 kB)
|████████████████████████████████| 131 kB 53.1 MB/s
Collecting configparser>=3.5.0
Downloading configparser-5.0.2-py3-none-any.whl (19 kB)
Collecting croniter>=0.3.16
Downloading croniter-1.0.8-py2.py3-none-any.whl (13 kB)
Collecting elasticsearch==7.0.0
Downloading elasticsearch-7.0.0-py2.py3-none-any.whl (80 kB)
|████████████████████████████████| 80 kB 16.9 MB/s
Collecting envparse>=0.2.0
Downloading envparse-0.2.0.tar.gz (7.6 kB)
Collecting exotel>=0.1.3
Downloading exotel-0.1.5.tar.gz (2.2 kB)
Collecting jira>=2.0.0
Downloading jira-2.0.0-py2.py3-none-any.whl (57 kB)
|████████████████████████████████| 57 kB 10.4 MB/s
Collecting jsonschema>=3.0.2
Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
|████████████████████████████████| 56 kB 9.1 MB/s
Collecting mock>=2.0.0
Downloading mock-4.0.3-py3-none-any.whl (28 kB)
Collecting prison>=0.1.2
Downloading prison-0.1.3-py2.py3-none-any.whl (5.8 kB)
Collecting PyStaticConfiguration>=0.10.3
Downloading PyStaticConfiguration-0.10.5.tar.gz (21 kB)
Collecting python-dateutil<2.7.0,>=2.6.0
Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194 kB)
|████████████████████████████████| 194 kB 37.5 MB/s
Collecting PyYAML>=3.12
Downloading PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB)
|████████████████████████████████| 640 kB 39.0 MB/s
Collecting requests>=2.10.0
Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
|████████████████████████████████| 61 kB 8.7 MB/s
Collecting stomp.py>=4.1.17
Downloading stomp.py-6.1.0-py3-none-any.whl (37 kB)
Collecting texttable>=0.8.8
Downloading texttable-1.6.3-py2.py3-none-any.whl (10 kB)
Collecting twilio<6.1,>=6.0.0
Downloading twilio-6.0.0.tar.gz (304 kB)
|████████████████████████████████| 304 kB 33.4 MB/s
Collecting python-magic>=0.4.15
Downloading python_magic-0.4.22-py2.py3-none-any.whl (12 kB)
Collecting cffi>=1.11.5
Downloading cffi-1.14.5-cp36-cp36m-manylinux1_x86_64.whl (401 kB)
|████████████████████████████████| 401 kB 56.5 MB/s
Collecting urllib3>=1.21.1
Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
|████████████████████████████████| 153 kB 63.7 MB/s
Collecting tzlocal~=2.0
Downloading tzlocal-2.1-py2.py3-none-any.whl (16 kB)
Collecting pytz
Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
|████████████████████████████████| 510 kB 65.5 MB/s
Collecting six>=1.4.0
Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Requirement already satisfied: setuptools>=0.7 in ./myvenv/lib/python3.6/site-packages (from apscheduler>=3.3.0->elastalert) (39.2.0)
Collecting s3transfer<0.4.0,>=0.3.0
Downloading s3transfer-0.3.4-py2.py3-none-any.whl (69 kB)
|████████████████████████████████| 69 kB 12.2 MB/s
Collecting botocore<1.21.0,>=1.20.29
Downloading botocore-1.20.29-py2.py3-none-any.whl (7.3 MB)
|████████████████████████████████| 7.3 MB 68.7 MB/s
Collecting jmespath<1.0.0,>=0.7.1
Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting pycparser
Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
|████████████████████████████████| 112 kB 70.3 MB/s
Collecting future
Downloading future-0.18.2.tar.gz (829 kB)
|████████████████████████████████| 829 kB 67.7 MB/s
Collecting natsort
Downloading natsort-7.1.1-py3-none-any.whl (35 kB)
Collecting requests-toolbelt
Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
|████████████████████████████████| 54 kB 4.6 MB/s
Collecting oauthlib[signedtoken]>=1.0.0
Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
|████████████████████████████████| 147 kB 69.3 MB/s
Collecting pbr>=3.0.0
Downloading pbr-5.5.1-py2.py3-none-any.whl (106 kB)
|████████████████████████████████| 106 kB 68.5 MB/s
Collecting defusedxml
Downloading defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting requests-oauthlib>=0.6.1
Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting attrs>=17.4.0
Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
|████████████████████████████████| 49 kB 10.9 MB/s
Collecting importlib-metadata
Downloading importlib_metadata-3.7.3-py3-none-any.whl (12 kB)
Collecting pyrsistent>=0.14.0
Downloading pyrsistent-0.17.3.tar.gz (106 kB)
|████████████████████████████████| 106 kB 67.5 MB/s
Collecting pyjwt>=1.0.0
Downloading PyJWT-2.0.1-py3-none-any.whl (15 kB)
Collecting cryptography
Downloading cryptography-3.4.6-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
|████████████████████████████████| 3.2 MB 51.3 MB/s
Collecting certifi>=2017.4.17
Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
|████████████████████████████████| 147 kB 69.4 MB/s
Collecting chardet<5,>=3.0.2
Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
|████████████████████████████████| 178 kB 68.0 MB/s
Collecting idna<3,>=2.5
Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
|████████████████████████████████| 58 kB 11.6 MB/s
Collecting docopt<0.7.0,>=0.6.2
Downloading docopt-0.6.2.tar.gz (25 kB)
Collecting pysocks
Downloading PySocks-1.7.1-py3-none-any.whl (16 kB)
Collecting zipp>=0.5
Downloading zipp-3.4.1-py3-none-any.whl (5.2 kB)
Collecting typing-extensions>=3.6.4
Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Using legacy 'setup.py install' for elastalert, since package 'wheel' is not installed.
Using legacy 'setup.py install' for blist, since package 'wheel' is not installed.
Using legacy 'setup.py install' for envparse, since package 'wheel' is not installed.
Using legacy 'setup.py install' for exotel, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pyrsistent, since package 'wheel' is not installed.
Using legacy 'setup.py install' for PyStaticConfiguration, since package 'wheel' is not installed.
Using legacy 'setup.py install' for docopt, since package 'wheel' is not installed.
Using legacy 'setup.py install' for twilio, since package 'wheel' is not installed.
Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
Installing collected packages: six, pycparser, urllib3, python-dateutil, jmespath, idna, chardet, cffi, certifi, zipp, typing-extensions, requests, pytz, pyjwt, oauthlib, cryptography, botocore, tzlocal, s3transfer, requests-toolbelt, requests-oauthlib, pysocks, pyrsistent, pbr, natsort, importlib-metadata, future, docopt, defusedxml, attrs, twilio, texttable, stomp.py, PyYAML, python-magic, PyStaticConfiguration, prison, mock, jsonschema, jira, exotel, envparse, elasticsearch, croniter, configparser, boto3, blist, aws-requests-auth, apscheduler, elastalert
Running setup.py install for pyrsistent ... done
Running setup.py install for future ... done
Running setup.py install for docopt ... done
Running setup.py install for twilio ... done
Running setup.py install for PyStaticConfiguration ... done
Running setup.py install for exotel ... done
Running setup.py install for envparse ... done
Running setup.py install for blist ... done
Running setup.py install for elastalert ... done
Successfully installed PyStaticConfiguration-0.10.5 PyYAML-5.4.1 apscheduler-3.7.0 attrs-20.3.0 aws-requests-auth-0.4.3 blist-1.3.6 boto3-1.17.29 botocore-1.20.29 certifi-2020.12.5 cffi-1.14.5 chardet-4.0.0 configparser-5.0.2 croniter-1.0.8 cryptography-3.4.6 defusedxml-0.7.1 docopt-0.6.2 elastalert-0.2.4 elasticsearch-7.0.0 envparse-0.2.0 exotel-0.1.5 future-0.18.2 idna-2.10 importlib-metadata-3.7.3 jira-2.0.0 jmespath-0.10.0 jsonschema-3.2.0 mock-4.0.3 natsort-7.1.1 oauthlib-3.1.0 pbr-5.5.1 prison-0.1.3 pycparser-2.20 pyjwt-2.0.1 pyrsistent-0.17.3 pysocks-1.7.1 python-dateutil-2.6.1 python-magic-0.4.22 pytz-2021.1 requests-2.25.1 requests-oauthlib-1.3.0 requests-toolbelt-0.9.1 s3transfer-0.3.4 six-1.15.0 stomp.py-6.1.0 texttable-1.6.3 twilio-6.0.0 typing-extensions-3.7.4.3 tzlocal-2.1 urllib3-1.26.4 zipp-3.4.1
sudo dnf install -y git
git clone https://github.com/nsano-rururu/elastalert-server_praeco_docker-compose_sample.git
cd elastalert-server_praeco_docker-compose_sample
mkdir -p es/data
chmod 777 es/data
vi docker-compose.yml
# Comment out the description of elastalert and praeco
docke-compose up -d
cd ..
git clone https://github.com/Yelp/elastalert.git
cd elastalert
mkdir -p rules rules_template
vi config.yaml
es_host: localhost
es_port: 9200
rules_folder: rules
run_every:
seconds: 60
buffer_time:
minutes: 1
writeback_index: elastalert_status
alert_time_limit:
days: 2
skip_invalid: True
elastalert-create-index --config config.yaml
Elastic Version: 7.9.3
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!
@alidrsn
Is this still a counterargument?
I have provided the information I have. I don't know if this doesn't work. Solve it on your own.
Hi,
I did not use elastalert with docker-compose.
I just followed the steps in --> https://elastalert.readthedocs.io/en/latest/running_elastalert.html
However, when i look your instructions, it is more complicated than that link.
Trying to understand where is missed.
Let me inform you ASAP
Thanks for your help.
This is the error. I was trying to install elastalert as you mentioned.
pip install elasalert.
Final parts of the results. It gets error the final step ->Running setup.py install for blist ... error
ERROR: Command errored out with exit status 1: /root/myvenv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x_ofqo6q/blist_372e2edd1d66447db6970f3878680484/setup.py'"'"'; file='"'"'/tmp/pip-install-x_ofqo6q/blist_372e2edd1d66447db6970f3878680484/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-uoguay57/install-record.txt --single-version-externally-managed --compile --install-headers /root/myvenv/include/site/python3.6/blist Check the logs for full command output.
@alidrsn
I did not use elastalert with docker-compose.
so what?
@alidrsn
I used docker because I didn't want to pollute the environment, but are there any complaints? I don't know what your environment is
@alidrsn
There should be one of the following that is not included. Or rather, you can find out by searching for the error message on google. I won't say any more.
sudo dnf group install "Development Tools"
sudo dnf -y install python3-pip python36 python3-devel python3-setuptools python3-libs libffi-devel openssl-devel
@alidrsn
This project is currently inactive, so I have to say that it's very difficult to take advantage of it if you can't resolve the error on your own.
@nsano-rururu
I am writing this for who is facing this problem.
Actually, i used your method with python virtual environment as you mentioned above
with this method it works fine!.
However, without venv it gives the same error.
The reason must be the packet conflict or version mismatch which are installing with /elastalert/setup.py
Thanks for your help btw,
It saves my day!
@alidrsn
If you install elastalert in an environment with various things, it is natural that the version inconsistency of the library will appear. This is a natural result.
@alidrsn
Would you consider using elastalert with docker instead of direct installation? https://hub.docker.com/r/jertel/elastalert-docker https://github.com/jertel/elastalert-docker/discussions
elastalert-create-index --config config.yaml
Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 570, in _build_master ws.require(requires) File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 888, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 779, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (urllib3 1.21.1 (/usr/local/lib/python3.6/site-packages), Requirement.parse('urllib3<1.27,>=1.25.4'), {'elastalert'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/bin/elastalert-create-index", line 6, in
from pkg_resources import load_entry_point
File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3095, in
@_call_aside
File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3079, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3108, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 572, in _build_master
return cls._build_from_requirements(requires)
File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 585, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 774, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'urllib3<1.27,>=1.25.4' distribution was not found and is required by elastalert
ES Version: 7.5.0