elastalert-create-index cannot execute

alidrsn commented 3 years ago

elastalert-create-index --config config.yaml

Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 570, in _build_master ws.require(requires) File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 888, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 779, in resolve raise VersionConflict(dist, req).with_context(dependent_req) pkg_resources.ContextualVersionConflict: (urllib3 1.21.1 (/usr/local/lib/python3.6/site-packages), Requirement.parse('urllib3<1.27,>=1.25.4'), {'elastalert'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/elastalert-create-index", line 6, in from pkg_resources import load_entry_point File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3095, in @_call_aside File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3079, in _call_aside f(*args, **kwargs) File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3108, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 572, in _build_master return cls._build_from_requirements(requires) File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 585, in _build_from_requirements dists = ws.resolve(reqs, Environment()) File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 774, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'urllib3<1.27,>=1.25.4' distribution was not found and is required by elastalert

ES Version: 7.5.0

nsano-rururu commented 3 years ago

Please install and use elastalert in a python virtual environment dedicated to elastalert. I don't think we have guaranteed operation in any other way. I'm a passerby and have nothing to do with this project.

example

# Python Create virtual environment
python3 -m venv ~/elastalert-venv/

# Python Virtual environment switching
source ~/elastalert-venv/bin/activate

# ElastAlert Install
pip install elastalert

alidrsn commented 3 years ago

Hi,

Actually, i did not anything different from your instructions.

These are sources that i use,

OS: Red hat 8 Phyton: 3.6 Elasticsearch: 7.5

I saw similar issues that is closed before without any upgrade comments like #3112

nsano-rururu commented 3 years ago

@alidrsn

AWS EC2 t3.medium OS : CentOS 8 Elasticsearch 7.9.3

There was no problem.

sudo dnf -y update
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y docker-ce docker-ce-cli
sudo systemctl start docker
sudo usermod -a -G docker $USER
sudo systemctl enable docker

sudo dnf group install "Development Tools"
sudo dnf -y install python3-pip python36 python3-devel python3-setuptools python3-libs libffi-devel openssl-devel
sudo curl -L https://github.com/docker/compose/releases/download/1.28.5/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# ssh reconnect

python3 -m venv ~/myvenv/
source ~/myvenv/bin/activate
pip install --upgrade pip
pip install setuptools
pip install elastalert

Collecting elastalert
  Downloading elastalert-0.2.4.tar.gz (128 kB)
     |████████████████████████████████| 128 kB 21.5 MB/s 
Collecting apscheduler>=3.3.0
  Downloading APScheduler-3.7.0-py2.py3-none-any.whl (59 kB)
     |████████████████████████████████| 59 kB 10.5 MB/s 
Collecting aws-requests-auth>=0.3.0
  Downloading aws_requests_auth-0.4.3-py2.py3-none-any.whl (6.8 kB)
Collecting blist>=1.3.6
  Downloading blist-1.3.6.tar.gz (122 kB)
     |████████████████████████████████| 122 kB 51.2 MB/s 
Collecting boto3>=1.4.4
  Downloading boto3-1.17.29-py2.py3-none-any.whl (131 kB)
     |████████████████████████████████| 131 kB 53.1 MB/s 
Collecting configparser>=3.5.0
  Downloading configparser-5.0.2-py3-none-any.whl (19 kB)
Collecting croniter>=0.3.16
  Downloading croniter-1.0.8-py2.py3-none-any.whl (13 kB)
Collecting elasticsearch==7.0.0
  Downloading elasticsearch-7.0.0-py2.py3-none-any.whl (80 kB)
     |████████████████████████████████| 80 kB 16.9 MB/s 
Collecting envparse>=0.2.0
  Downloading envparse-0.2.0.tar.gz (7.6 kB)
Collecting exotel>=0.1.3
  Downloading exotel-0.1.5.tar.gz (2.2 kB)
Collecting jira>=2.0.0
  Downloading jira-2.0.0-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 10.4 MB/s 
Collecting jsonschema>=3.0.2
  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)
     |████████████████████████████████| 56 kB 9.1 MB/s 
Collecting mock>=2.0.0
  Downloading mock-4.0.3-py3-none-any.whl (28 kB)
Collecting prison>=0.1.2
  Downloading prison-0.1.3-py2.py3-none-any.whl (5.8 kB)
Collecting PyStaticConfiguration>=0.10.3
  Downloading PyStaticConfiguration-0.10.5.tar.gz (21 kB)
Collecting python-dateutil<2.7.0,>=2.6.0
  Downloading python_dateutil-2.6.1-py2.py3-none-any.whl (194 kB)
     |████████████████████████████████| 194 kB 37.5 MB/s 
Collecting PyYAML>=3.12
  Downloading PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640 kB)
     |████████████████████████████████| 640 kB 39.0 MB/s 
Collecting requests>=2.10.0
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 8.7 MB/s 
Collecting stomp.py>=4.1.17
  Downloading stomp.py-6.1.0-py3-none-any.whl (37 kB)
Collecting texttable>=0.8.8
  Downloading texttable-1.6.3-py2.py3-none-any.whl (10 kB)
Collecting twilio<6.1,>=6.0.0
  Downloading twilio-6.0.0.tar.gz (304 kB)
     |████████████████████████████████| 304 kB 33.4 MB/s 
Collecting python-magic>=0.4.15
  Downloading python_magic-0.4.22-py2.py3-none-any.whl (12 kB)
Collecting cffi>=1.11.5
  Downloading cffi-1.14.5-cp36-cp36m-manylinux1_x86_64.whl (401 kB)
     |████████████████████████████████| 401 kB 56.5 MB/s 
Collecting urllib3>=1.21.1
  Downloading urllib3-1.26.4-py2.py3-none-any.whl (153 kB)
     |████████████████████████████████| 153 kB 63.7 MB/s 
Collecting tzlocal~=2.0
  Downloading tzlocal-2.1-py2.py3-none-any.whl (16 kB)
Collecting pytz
  Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 65.5 MB/s 
Collecting six>=1.4.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Requirement already satisfied: setuptools>=0.7 in ./myvenv/lib/python3.6/site-packages (from apscheduler>=3.3.0->elastalert) (39.2.0)
Collecting s3transfer<0.4.0,>=0.3.0
  Downloading s3transfer-0.3.4-py2.py3-none-any.whl (69 kB)
     |████████████████████████████████| 69 kB 12.2 MB/s 
Collecting botocore<1.21.0,>=1.20.29
  Downloading botocore-1.20.29-py2.py3-none-any.whl (7.3 MB)
     |████████████████████████████████| 7.3 MB 68.7 MB/s 
Collecting jmespath<1.0.0,>=0.7.1
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 70.3 MB/s 
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     |████████████████████████████████| 829 kB 67.7 MB/s 
Collecting natsort
  Downloading natsort-7.1.1-py3-none-any.whl (35 kB)
Collecting requests-toolbelt
  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 4.6 MB/s 
Collecting oauthlib[signedtoken]>=1.0.0
  Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
     |████████████████████████████████| 147 kB 69.3 MB/s 
Collecting pbr>=3.0.0
  Downloading pbr-5.5.1-py2.py3-none-any.whl (106 kB)
     |████████████████████████████████| 106 kB 68.5 MB/s 
Collecting defusedxml
  Downloading defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting requests-oauthlib>=0.6.1
  Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting attrs>=17.4.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 10.9 MB/s 
Collecting importlib-metadata
  Downloading importlib_metadata-3.7.3-py3-none-any.whl (12 kB)
Collecting pyrsistent>=0.14.0
  Downloading pyrsistent-0.17.3.tar.gz (106 kB)
     |████████████████████████████████| 106 kB 67.5 MB/s 
Collecting pyjwt>=1.0.0
  Downloading PyJWT-2.0.1-py3-none-any.whl (15 kB)
Collecting cryptography
  Downloading cryptography-3.4.6-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
     |████████████████████████████████| 3.2 MB 51.3 MB/s 
Collecting certifi>=2017.4.17
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
     |████████████████████████████████| 147 kB 69.4 MB/s 
Collecting chardet<5,>=3.0.2
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 68.0 MB/s 
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 11.6 MB/s 
Collecting docopt<0.7.0,>=0.6.2
  Downloading docopt-0.6.2.tar.gz (25 kB)
Collecting pysocks
  Downloading PySocks-1.7.1-py3-none-any.whl (16 kB)
Collecting zipp>=0.5
  Downloading zipp-3.4.1-py3-none-any.whl (5.2 kB)
Collecting typing-extensions>=3.6.4
  Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Using legacy 'setup.py install' for elastalert, since package 'wheel' is not installed.
Using legacy 'setup.py install' for blist, since package 'wheel' is not installed.
Using legacy 'setup.py install' for envparse, since package 'wheel' is not installed.
Using legacy 'setup.py install' for exotel, since package 'wheel' is not installed.
Using legacy 'setup.py install' for pyrsistent, since package 'wheel' is not installed.
Using legacy 'setup.py install' for PyStaticConfiguration, since package 'wheel' is not installed.
Using legacy 'setup.py install' for docopt, since package 'wheel' is not installed.
Using legacy 'setup.py install' for twilio, since package 'wheel' is not installed.
Using legacy 'setup.py install' for future, since package 'wheel' is not installed.
Installing collected packages: six, pycparser, urllib3, python-dateutil, jmespath, idna, chardet, cffi, certifi, zipp, typing-extensions, requests, pytz, pyjwt, oauthlib, cryptography, botocore, tzlocal, s3transfer, requests-toolbelt, requests-oauthlib, pysocks, pyrsistent, pbr, natsort, importlib-metadata, future, docopt, defusedxml, attrs, twilio, texttable, stomp.py, PyYAML, python-magic, PyStaticConfiguration, prison, mock, jsonschema, jira, exotel, envparse, elasticsearch, croniter, configparser, boto3, blist, aws-requests-auth, apscheduler, elastalert
    Running setup.py install for pyrsistent ... done
    Running setup.py install for future ... done
    Running setup.py install for docopt ... done
    Running setup.py install for twilio ... done
    Running setup.py install for PyStaticConfiguration ... done
    Running setup.py install for exotel ... done
    Running setup.py install for envparse ... done
    Running setup.py install for blist ... done
    Running setup.py install for elastalert ... done
Successfully installed PyStaticConfiguration-0.10.5 PyYAML-5.4.1 apscheduler-3.7.0 attrs-20.3.0 aws-requests-auth-0.4.3 blist-1.3.6 boto3-1.17.29 botocore-1.20.29 certifi-2020.12.5 cffi-1.14.5 chardet-4.0.0 configparser-5.0.2 croniter-1.0.8 cryptography-3.4.6 defusedxml-0.7.1 docopt-0.6.2 elastalert-0.2.4 elasticsearch-7.0.0 envparse-0.2.0 exotel-0.1.5 future-0.18.2 idna-2.10 importlib-metadata-3.7.3 jira-2.0.0 jmespath-0.10.0 jsonschema-3.2.0 mock-4.0.3 natsort-7.1.1 oauthlib-3.1.0 pbr-5.5.1 prison-0.1.3 pycparser-2.20 pyjwt-2.0.1 pyrsistent-0.17.3 pysocks-1.7.1 python-dateutil-2.6.1 python-magic-0.4.22 pytz-2021.1 requests-2.25.1 requests-oauthlib-1.3.0 requests-toolbelt-0.9.1 s3transfer-0.3.4 six-1.15.0 stomp.py-6.1.0 texttable-1.6.3 twilio-6.0.0 typing-extensions-3.7.4.3 tzlocal-2.1 urllib3-1.26.4 zipp-3.4.1

sudo dnf install -y git
git clone https://github.com/nsano-rururu/elastalert-server_praeco_docker-compose_sample.git
cd elastalert-server_praeco_docker-compose_sample
mkdir -p es/data
chmod 777 es/data

vi docker-compose.yml

# Comment out the description of elastalert and praeco

docke-compose up -d

cd ..
git clone https://github.com/Yelp/elastalert.git
cd elastalert
mkdir -p rules rules_template

vi config.yaml

es_host: localhost
es_port: 9200
rules_folder: rules
run_every:
  seconds: 60
buffer_time:
  minutes: 1
writeback_index: elastalert_status
alert_time_limit:
  days: 2
skip_invalid: True

elastalert-create-index --config config.yaml

Elastic Version: 7.9.3
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!

nsano-rururu commented 3 years ago

@alidrsn

Is this still a counterargument?

I have provided the information I have. I don't know if this doesn't work. Solve it on your own.

alidrsn commented 3 years ago

Hi,

I did not use elastalert with docker-compose.

I just followed the steps in --> https://elastalert.readthedocs.io/en/latest/running_elastalert.html

However, when i look your instructions, it is more complicated than that link.

Trying to understand where is missed.

Let me inform you ASAP

Thanks for your help.

alidrsn commented 3 years ago

This is the error. I was trying to install elastalert as you mentioned.

pip install elasalert.

Final parts of the results. It gets error the final step ->Running setup.py install for blist ... error

Installing collected packages: six, pycparser, urllib3, python-dateutil, jmespath, idna, chardet, cffi, certifi, zipp, typing-extensions, requests, pytz, pyjwt, oauthlib, cryptography, botocore, tzlocal, s3transfer, requests-toolbelt, requests-oauthlib, pysocks, pyrsistent, pbr, natsort, importlib-metadata, future, docopt, defusedxml, attrs, twilio, texttable, stomp.py, PyYAML, python-magic, PyStaticConfiguration, prison, mock, jsonschema, jira, exotel, envparse, elasticsearch, croniter, configparser, boto3, blist, aws-requests-auth, apscheduler, elastalert Running setup.py install for pyrsistent ... done Running setup.py install for future ... done Running setup.py install for docopt ... done Running setup.py install for twilio ... done Running setup.py install for PyStaticConfiguration ... done Running setup.py install for exotel ... done Running setup.py install for envparse ... done Running setup.py install for blist ... error ERROR: Command errored out with exit status 1: command: /root/myvenv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x_ofqo6q/blist_372e2edd1d66447db6970f3878680484/setup.py'"'"'; file='"'"'/tmp/pip-install-x_ofqo6q/blist_372e2edd1d66447db6970f3878680484/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-uoguay57/install-record.txt --single-version-externally-managed --compile --install-headers /root/myvenv/include/site/python3.6/blist cwd: /tmp/pip-install-x_ofqo6q/blist_372e2edd1d66447db6970f3878680484/ Complete output (17 lines): running install running build running build_py creating build creating build/lib.linux-x86_64-3.6 creating build/lib.linux-x86_64-3.6/blist copying blist/init.py -> build/lib.linux-x86_64-3.6/blist copying blist/_sortedlist.py -> build/lib.linux-x86_64-3.6/blist copying blist/_btuple.py -> build/lib.linux-x86_64-3.6/blist copying blist/_sorteddict.py -> build/lib.linux-x86_64-3.6/blist running build_ext building 'blist._blist' extension creating build/temp.linux-x86_64-3.6 creating build/temp.linux-x86_64-3.6/blist gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DBLIST_FLOAT_RADIX_SORT=1 -I/root/myvenv/include -I/usr/include/python3.6m -c blist/_blist.c -o build/temp.linux-x86_64-3.6/blist/_blist.o unable to execute 'gcc': No such file or directory error: command 'gcc' failed with exit status 1

ERROR: Command errored out with exit status 1: /root/myvenv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-x_ofqo6q/blist_372e2edd1d66447db6970f3878680484/setup.py'"'"'; file='"'"'/tmp/pip-install-x_ofqo6q/blist_372e2edd1d66447db6970f3878680484/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-uoguay57/install-record.txt --single-version-externally-managed --compile --install-headers /root/myvenv/include/site/python3.6/blist Check the logs for full command output.

nsano-rururu commented 3 years ago

@alidrsn

I did not use elastalert with docker-compose.

so what?

nsano-rururu commented 3 years ago

@alidrsn

I used docker because I didn't want to pollute the environment, but are there any complaints? I don't know what your environment is

nsano-rururu commented 3 years ago

@alidrsn

There should be one of the following that is not included. Or rather, you can find out by searching for the error message on google. I won't say any more.

sudo dnf group install "Development Tools"
sudo dnf -y install python3-pip python36 python3-devel python3-setuptools python3-libs libffi-devel openssl-devel

nsano-rururu commented 3 years ago

@alidrsn

This project is currently inactive, so I have to say that it's very difficult to take advantage of it if you can't resolve the error on your own.

alidrsn commented 3 years ago

@nsano-rururu

I am writing this for who is facing this problem.

Actually, i used your method with python virtual environment as you mentioned above

python3 -m venv ~/myvenv/
source ~/myvenv/bin/activate
pip install --upgrade pip
pip install setuptools
pip install elastalert

with this method it works fine!.

However, without venv it gives the same error.

The reason must be the packet conflict or version mismatch which are installing with /elastalert/setup.py

Thanks for your help btw,

It saves my day!

nsano-rururu commented 3 years ago

@alidrsn

If you install elastalert in an environment with various things, it is natural that the version inconsistency of the library will appear. This is a natural result.

nsano-rururu commented 3 years ago

@alidrsn

Would you consider using elastalert with docker instead of direct installation? https://hub.docker.com/r/jertel/elastalert-docker https://github.com/jertel/elastalert-docker/discussions

Yelp / elastalert

elastalert-create-index cannot execute #3153