Connecting Elasticsearch with SSL

alidrsn commented 3 years ago

Hi There,

I am using ssl to connect my elastic cluster on http port (9200)

Certificate is .pfx file, root and sub certificates also included with chain.

Trying different configurations (config.yaml) to connect elastalert to cluster with, everytime i got different errors like,

SSL: EE_KEY_TOO_SMALL SSL: CERTIFICATE_VERIFY_FAILED

How can i use .pfx file for elastalert as truststore ? What is the best configuration ?

Thanks in advance.

nsano-rururu commented 3 years ago

https://github.com/Yelp/elastalert/issues/2235 https://elastalert.readthedocs.io/en/latest/ruletypes.html#optional-settings

alidrsn commented 3 years ago

Hi @nsano-rururu ,

Let me understand clearly. I seperated the .pfx file to cer and key with no password, and write it in 'elastalert config.yaml' file.

However, it also did not work.

Do i need to change it on server-side ? Do i need to change it on elasticsearch.yaml with different certificate ?

Thanks.

nsano-rururu commented 3 years ago

Do i need to change it on server-side ? Do i need to change it on elasticsearch.yaml with different certificate ?

I'm not familiar with certificates

nsano-rururu commented 3 years ago

I've tried everything with docker and elastalert-server. Below is the log. I don't know if it will be helpful.

chmod 777 rules rule_templates

/home/user/dkwork2/es2
|--.env
|--config
|  |--api.config.json
|  |--elastalert.yaml
|--create-certs.yml
|--docker-compose-praeco.yml
|--docker-compose.yml
|--instances.yml
|--nginx_config
|  |--default.conf
|  |--nginx.conf
|--public
|  |--praeco.config.json
|--rule_templates
|--rules

.env

COMPOSE_PROJECT_NAME=es 
CERTS_DIR=/usr/share/elasticsearch/config/certificates 
ELASTIC_PASSWORD=PleaseChangeMe

config/api.config.json

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "es01",
  "es_port": 9200,
  "es_username": "elastic",
  "es_password": "xxxxxxxxx
  ",
  "es_ca_certs": "/usr/share/elasticsearch/config/certificates/ca/ca.crt", 
  "es_client_cert": "/usr/share/elasticsearch/config/certificates/es01/es01.crt", 
  "es_client_key": "/usr/share/elasticsearch/config/certificates/es01/es01.key", 
  "es_ssl": true,
  "writeback_index": "praeco_elastalert_status"
}

config/elastalert.yaml

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: es01

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 60

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
es_username: elastic
es_password: xxxxxxxxx

# ssl
ca_certs: /usr/share/elasticsearch/config/certificates/ca/ca.crt
client_cert: /usr/share/elasticsearch/config/certificates/es01/es01.crt
client_key: /usr/share/elasticsearch/config/certificates/es01/es01.key

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: praeco_elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

skip_invalid: True

create-certs.yml

version: '2.2'

services:
  create_certs:
    container_name: create_certs
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    command: >
      bash -c '
        if [[ ! -f /certs/bundle.zip ]]; then
          bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip;
          unzip /certs/bundle.zip -d /certs; 
        fi;
        chown -R 1000:0 /certs
      '
    user: "0"
    working_dir: /usr/share/elasticsearch
    volumes: ['certs:/certs', '.:/usr/share/elasticsearch/config/certificates']

volumes: {"certs"}

instances.yml

instances:
  - name: es01
    dns:
      - es01 
      - localhost
    ip:
      - 127.0.0.1

  #- name: es02
  #  dns:
  #    - es02
  #    - localhost
  #  ip:
  #    - 127.0.0.1

docker-compose.yml

version: '2.2'

services:
  es01:
    container_name: es01
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    environment:
      - discovery.type=single-node
      - cluster.name=es01
      #- node.name=es01
      #- discovery.seed_hosts=es01,es02
      #- cluster.initial_master_nodes=es01,es02
      - network.host=0.0.0.0
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD 
      - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
      - xpack.license.self_generated.type=trial 
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate 
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
    volumes: ['data01:/usr/share/elasticsearch/data', 'certs:$CERTS_DIR']
    ports:
      - 9200:9200
    healthcheck:
      test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

volumes: {"data01", "certs"}

docker-compose-praeco.yml

version: '3'

services:
  elastalert:
    image: 'praecoapp/elastalert-server'
    ports:
      - 3030:3030
      - 3333:3333
    volumes:
      - ./config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./config/api.config.json:/opt/elastalert-server/config/config.json
      - ./rules:/opt/elastalert/rules
      - ./rule_templates:/opt/elastalert/rule_templates
      - certs:/usr/share/elasticsearch/config/certificates
    #extra_hosts:
    #  - 'elasticsearch:${PRAECO_ELASTICSEARCH}'
    tty: true
    networks:
      - default
      - es_default

  webapp:
    image: 'praecoapp/praeco'
    ports:
      - 8080:8080
    volumes:
      - ./public/praeco.config.json:/var/www/html/praeco.config.json
      - ./nginx_config/nginx.conf:/etc/nginx/nginx.conf
      - ./nginx_config/default.conf:/etc/nginx/conf.d/default.conf
      - certs:/usr/share/elasticsearch/config/certificates
    tty: true
    networks:
      - default
      - es_default

volumes: { "certs" }

networks:
  es_default:
    external: true

$ docker-compose -f create-certs.yml run --rm create_certs

Creating network "es_default" with the default driver
Creating volume "es_certs" with default driver
Archive:  /certs/bundle.zip
   creating: /certs/ca/
  inflating: /certs/ca/ca.crt        
   creating: /certs/es01/
  inflating: /certs/es01/es01.crt    
  inflating: /certs/es01/es01.key   

$ docker-compose up -d

Creating volume "es_data01" with default driver
Creating es01 ... done

$ docker run --rm -v es_certs:/certs --network=es_default docker.elastic.co/elasticsearch/elasticsearch:7.10.0 curl --cacert /certs/ca/ca.crt -u elastic:PleaseChangeMe https://es01:9200

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to es01 port 9200: Connection refused

$ docker exec es01 /bin/bash -c "bin/elasticsearch-setup-passwords \
auto --batch \
--url https://localhost:9200"

# Should be output with a different value each time

Changed password for user apm_system
PASSWORD apm_system = aaaaaaaaaaaa

Changed password for user kibana_system
PASSWORD kibana_system = bbbbbbbbb

Changed password for user kibana
PASSWORD kibana = ddddddddddd

Changed password for user logstash_system
PASSWORD logstash_system = eeeeeeeeee

Changed password for user beats_system
PASSWORD beats_system = fffffffffff

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = gffffffff

Changed password for user elastic
PASSWORD elastic = xxxxxxxxx

$ curl -u elastic:xxxxxxxxx https://localhost:9200/_cat/indices?v

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

$ curl -u elastic:xxxxxxxxx localhost:9200?pretty

curl: (52) Empty reply from server

$ docker run --rm -v es_certs:/certs --network=es_default docker.elastic.co/elasticsearch/elasticsearch:7.10.0 curl --cacert /certs/ca/ca.crt -u elastic:xxxxxxxxx https://es01:9200/_cat/indices?v

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   214  100   214    0     0    697      0 --:--:-- --:--:-- --:--:--   697
health status index       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .security-7 HVZs78GJTV6A1EbjTG_2Fg   1   0          7            0     25.1kb         25.1kb

$ docker ps

CONTAINER ID        IMAGE                                                  COMMAND                  CREATED             STATUS                   PORTS                              NAMES
65263085b175        docker.elastic.co/elasticsearch/elasticsearch:7.10.0   "/tini -- /usr/local…"   2 minutes ago       Up 2 minutes (healthy)   0.0.0.0:9200->9200/tcp, 9300/tcp   es01

$ docker-compose -f docker-compose-praeco.yml up -d

$ docker logs -f es_elastalert_1

> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh

15:42:37.418Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
15:42:37.421Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
15:42:37.443Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
15:42:37.506Z  INFO elastalert-server: Router:  Listening for GET request on /.
15:42:37.507Z  INFO elastalert-server: Router:  Listening for GET request on /status.
15:42:37.507Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
15:42:37.508Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
15:42:37.511Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
15:42:37.512Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
15:42:37.513Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
15:42:37.513Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
15:42:37.513Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
15:42:37.513Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
15:42:37.514Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
15:42:37.514Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
15:42:37.515Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
15:42:37.515Z  INFO elastalert-server: Router:  Listening for POST request on /test.
15:42:37.515Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
15:42:37.515Z  INFO elastalert-server: Router:  Listening for GET request on /config.
15:42:37.516Z  INFO elastalert-server: Router:  Listening for POST request on /config.
15:42:37.516Z  INFO elastalert-server: Router:  Listening for POST request on /download.
15:42:37.516Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
15:42:37.516Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
15:42:37.516Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
15:42:37.524Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
15:42:37.525Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
15:42:37.525Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
15:42:37.525Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
15:42:37.527Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
15:42:37.528Z  INFO elastalert-server: Router:  Listening for GET request on /config.
15:42:37.535Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
15:42:37.535Z  INFO elastalert-server: ProcessController:  Creating index
15:42:42.772Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.10.0
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index praeco_elastalert_status created
    Done!

15:42:42.772Z ERROR elastalert-server:
    ProcessController:  /home/node/.local/lib/python3.8/site-packages/elasticsearch/connection/base.py:193: ElasticsearchDeprecationWarning: Camel case format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead.
      warnings.warn(message, category=ElasticsearchDeprecationWarning)
    /home/node/.local/lib/python3.8/site-packages/elasticsearch/connection/base.py:193: ElasticsearchDeprecationWarning: [types removal] Using include_type_name in put mapping requests is deprecated. The parameter will be removed in the next major version.
      warnings.warn(message, category=ElasticsearchDeprecationWarning)

15:42:42.772Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
15:42:42.773Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
15:42:42.827Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 37)
15:42:42.831Z  INFO elastalert-server: Server:  Server listening on port 3030
15:42:42.833Z  INFO elastalert-server: Server:  Websocket listening on port 3333
15:42:42.835Z  INFO elastalert-server: Server:  Server started

Yelp / elastalert

Connecting Elasticsearch with SSL #3184