Open alidrsn opened 3 years ago
Hi @nsano-rururu ,
Let me understand clearly. I seperated the .pfx file to cer and key with no password, and write it in 'elastalert config.yaml' file.
However, it also did not work.
Do i need to change it on server-side ? Do i need to change it on elasticsearch.yaml with different certificate ?
Thanks.
Do i need to change it on server-side ? Do i need to change it on elasticsearch.yaml with different certificate ?
I'm not familiar with certificates
I've tried everything with docker and elastalert-server. Below is the log. I don't know if it will be helpful.
chmod 777 rules rule_templates
/home/user/dkwork2/es2
|--.env
|--config
| |--api.config.json
| |--elastalert.yaml
|--create-certs.yml
|--docker-compose-praeco.yml
|--docker-compose.yml
|--instances.yml
|--nginx_config
| |--default.conf
| |--nginx.conf
|--public
| |--praeco.config.json
|--rule_templates
|--rules
.env
COMPOSE_PROJECT_NAME=es
CERTS_DIR=/usr/share/elasticsearch/config/certificates
ELASTIC_PASSWORD=PleaseChangeMe
config/api.config.json
{
"appName": "elastalert-server",
"port": 3030,
"wsport": 3333,
"elastalertPath": "/opt/elastalert",
"verbose": false,
"es_debug": false,
"debug": false,
"rulesPath": {
"relative": true,
"path": "/rules"
},
"templatesPath": {
"relative": true,
"path": "/rule_templates"
},
"es_host": "es01",
"es_port": 9200,
"es_username": "elastic",
"es_password": "xxxxxxxxx
",
"es_ca_certs": "/usr/share/elasticsearch/config/certificates/ca/ca.crt",
"es_client_cert": "/usr/share/elasticsearch/config/certificates/es01/es01.crt",
"es_client_key": "/usr/share/elasticsearch/config/certificates/es01/es01.key",
"es_ssl": true,
"writeback_index": "praeco_elastalert_status"
}
config/elastalert.yaml
# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: es01
# The elasticsearch port
es_port: 9200
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules
# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
seconds: 60
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 1
# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch
# Connect with TLS to elasticsearch
use_ssl: True
# Verify TLS certificates
#verify_certs: True
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET
# Option basic-auth username and password for elasticsearch
es_username: elastic
es_password: xxxxxxxxx
# ssl
ca_certs: /usr/share/elasticsearch/config/certificates/ca/ca.crt
client_cert: /usr/share/elasticsearch/config/certificates/es01/es01.crt
client_key: /usr/share/elasticsearch/config/certificates/es01/es01.key
# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: praeco_elastalert_status
# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 2
skip_invalid: True
create-certs.yml
version: '2.2'
services:
create_certs:
container_name: create_certs
image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
command: >
bash -c '
if [[ ! -f /certs/bundle.zip ]]; then
bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip;
unzip /certs/bundle.zip -d /certs;
fi;
chown -R 1000:0 /certs
'
user: "0"
working_dir: /usr/share/elasticsearch
volumes: ['certs:/certs', '.:/usr/share/elasticsearch/config/certificates']
volumes: {"certs"}
instances.yml
instances:
- name: es01
dns:
- es01
- localhost
ip:
- 127.0.0.1
#- name: es02
# dns:
# - es02
# - localhost
# ip:
# - 127.0.0.1
docker-compose.yml
version: '2.2'
services:
es01:
container_name: es01
image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
environment:
- discovery.type=single-node
- cluster.name=es01
#- node.name=es01
#- discovery.seed_hosts=es01,es02
#- cluster.initial_master_nodes=es01,es02
- network.host=0.0.0.0
- ELASTIC_PASSWORD=$ELASTIC_PASSWORD
- "ES_JAVA_OPTS=-Xms256m -Xmx256m"
- xpack.license.self_generated.type=trial
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
- xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
- xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
- xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
volumes: ['data01:/usr/share/elasticsearch/data', 'certs:$CERTS_DIR']
ports:
- 9200:9200
healthcheck:
test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
interval: 30s
timeout: 10s
retries: 5
volumes: {"data01", "certs"}
docker-compose-praeco.yml
version: '3'
services:
elastalert:
image: 'praecoapp/elastalert-server'
ports:
- 3030:3030
- 3333:3333
volumes:
- ./config/elastalert.yaml:/opt/elastalert/config.yaml
- ./config/api.config.json:/opt/elastalert-server/config/config.json
- ./rules:/opt/elastalert/rules
- ./rule_templates:/opt/elastalert/rule_templates
- certs:/usr/share/elasticsearch/config/certificates
#extra_hosts:
# - 'elasticsearch:${PRAECO_ELASTICSEARCH}'
tty: true
networks:
- default
- es_default
webapp:
image: 'praecoapp/praeco'
ports:
- 8080:8080
volumes:
- ./public/praeco.config.json:/var/www/html/praeco.config.json
- ./nginx_config/nginx.conf:/etc/nginx/nginx.conf
- ./nginx_config/default.conf:/etc/nginx/conf.d/default.conf
- certs:/usr/share/elasticsearch/config/certificates
tty: true
networks:
- default
- es_default
volumes: { "certs" }
networks:
es_default:
external: true
$ docker-compose -f create-certs.yml run --rm create_certs
Creating network "es_default" with the default driver
Creating volume "es_certs" with default driver
Archive: /certs/bundle.zip
creating: /certs/ca/
inflating: /certs/ca/ca.crt
creating: /certs/es01/
inflating: /certs/es01/es01.crt
inflating: /certs/es01/es01.key
$ docker-compose up -d
Creating volume "es_data01" with default driver
Creating es01 ... done
$ docker run --rm -v es_certs:/certs --network=es_default docker.elastic.co/elasticsearch/elasticsearch:7.10.0 curl --cacert /certs/ca/ca.crt -u elastic:PleaseChangeMe https://es01:9200
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to es01 port 9200: Connection refused
$ docker exec es01 /bin/bash -c "bin/elasticsearch-setup-passwords \
auto --batch \
--url https://localhost:9200"
# Should be output with a different value each time
Changed password for user apm_system
PASSWORD apm_system = aaaaaaaaaaaa
Changed password for user kibana_system
PASSWORD kibana_system = bbbbbbbbb
Changed password for user kibana
PASSWORD kibana = ddddddddddd
Changed password for user logstash_system
PASSWORD logstash_system = eeeeeeeeee
Changed password for user beats_system
PASSWORD beats_system = fffffffffff
Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = gffffffff
Changed password for user elastic
PASSWORD elastic = xxxxxxxxx
$ curl -u elastic:xxxxxxxxx https://localhost:9200/_cat/indices?v
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
$ curl -u elastic:xxxxxxxxx localhost:9200?pretty
curl: (52) Empty reply from server
$ docker run --rm -v es_certs:/certs --network=es_default docker.elastic.co/elasticsearch/elasticsearch:7.10.0 curl --cacert /certs/ca/ca.crt -u elastic:xxxxxxxxx https://es01:9200/_cat/indices?v
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 214 100 214 0 0 697 0 --:--:-- --:--:-- --:--:-- 697
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .security-7 HVZs78GJTV6A1EbjTG_2Fg 1 0 7 0 25.1kb 25.1kb
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
65263085b175 docker.elastic.co/elasticsearch/elasticsearch:7.10.0 "/tini -- /usr/local…" 2 minutes ago Up 2 minutes (healthy) 0.0.0.0:9200->9200/tcp, 9300/tcp es01
$ docker-compose -f docker-compose-praeco.yml up -d
$ docker logs -f es_elastalert_1
> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh
15:42:37.418Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
15:42:37.421Z INFO elastalert-server: Config: Proceeding to look for normal config file.
15:42:37.443Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config.
15:42:37.506Z INFO elastalert-server: Router: Listening for GET request on /.
15:42:37.507Z INFO elastalert-server: Router: Listening for GET request on /status.
15:42:37.507Z INFO elastalert-server: Router: Listening for GET request on /status/errors.
15:42:37.508Z INFO elastalert-server: Router: Listening for GET request on /rules.
15:42:37.511Z INFO elastalert-server: Router: Listening for GET request on /rules/:id*.
15:42:37.512Z INFO elastalert-server: Router: Listening for POST request on /rules/:id*.
15:42:37.513Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id*.
15:42:37.513Z INFO elastalert-server: Router: Listening for GET request on /templates.
15:42:37.513Z INFO elastalert-server: Router: Listening for GET request on /templates/:id*.
15:42:37.513Z INFO elastalert-server: Router: Listening for POST request on /templates/:id*.
15:42:37.514Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id*.
15:42:37.514Z INFO elastalert-server: Router: Listening for PUT request on /folders/:type/:path*.
15:42:37.515Z INFO elastalert-server: Router: Listening for DELETE request on /folders/:type/:path*.
15:42:37.515Z INFO elastalert-server: Router: Listening for POST request on /test.
15:42:37.515Z INFO elastalert-server: Router: Listening for POST request on /silence/:path*.
15:42:37.515Z INFO elastalert-server: Router: Listening for GET request on /config.
15:42:37.516Z INFO elastalert-server: Router: Listening for POST request on /config.
15:42:37.516Z INFO elastalert-server: Router: Listening for POST request on /download.
15:42:37.516Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert.
15:42:37.516Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_status.
15:42:37.516Z INFO elastalert-server: Router: Listening for GET request on /metadata/silence.
15:42:37.524Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_error.
15:42:37.525Z INFO elastalert-server: Router: Listening for GET request on /metadata/past_elastalert.
15:42:37.525Z INFO elastalert-server: Router: Listening for GET request on /indices.
15:42:37.525Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index.
15:42:37.527Z INFO elastalert-server: Router: Listening for POST request on /search/:index.
15:42:37.528Z INFO elastalert-server: Router: Listening for GET request on /config.
15:42:37.535Z INFO elastalert-server: ProcessController: Starting ElastAlert
15:42:37.535Z INFO elastalert-server: ProcessController: Creating index
15:42:42.772Z INFO elastalert-server:
ProcessController: Elastic Version: 7.10.0
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index praeco_elastalert_status created
Done!
15:42:42.772Z ERROR elastalert-server:
ProcessController: /home/node/.local/lib/python3.8/site-packages/elasticsearch/connection/base.py:193: ElasticsearchDeprecationWarning: Camel case format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead.
warnings.warn(message, category=ElasticsearchDeprecationWarning)
/home/node/.local/lib/python3.8/site-packages/elasticsearch/connection/base.py:193: ElasticsearchDeprecationWarning: [types removal] Using include_type_name in put mapping requests is deprecated. The parameter will be removed in the next major version.
warnings.warn(message, category=ElasticsearchDeprecationWarning)
15:42:42.772Z INFO elastalert-server: ProcessController: Index create exited with code 0
15:42:42.773Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
15:42:42.827Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 37)
15:42:42.831Z INFO elastalert-server: Server: Server listening on port 3030
15:42:42.833Z INFO elastalert-server: Server: Websocket listening on port 3333
15:42:42.835Z INFO elastalert-server: Server: Server started
Hi There,
I am using ssl to connect my elastic cluster on http port (9200)
Certificate is .pfx file, root and sub certificates also included with chain.
Trying different configurations (config.yaml) to connect elastalert to cluster with, everytime i got different errors like,
SSL: EE_KEY_TOO_SMALL SSL: CERTIFICATE_VERIFY_FAILED
How can i use .pfx file for elastalert as truststore ? What is the best configuration ?
Thanks in advance.