[BUG] Botkube can't create index in AWS ElasticSearch

[soq]

Describe the bug With version 0.11.0 and AWS ElasticSearch usage I get an error in botkube logs: ERRO[2020-10-06T12:50:59Z] Failed to create index. Error:elastic: Error 500 (Internal Server Error): Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes [type=not_x_content_exception] and in AWS ELK logs: org.elasticsearch.common.compress.NotXContentException: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes So, no index, no logs.

To Reproduce Try to use AWS ELK with botkube 0.11.0 and without awsSigning enabled.

Expected behavior Logs with events in ELK.

Additional context This error was mentioned here https://github.com/olivere/elastic/issues/1082 and described more here https://github.com/olivere/elastic/wiki/Using-with-AWS-Elasticsearch-Service#compression. Here is https://github.com/infracloudio/botkube/blob/v0.11.0/pkg/notify/elasticsearch.go#L86 the place as I can understand where this behaviour could be changed.

[sushrut111]

I could not reproduce this issue. I used the following ELS config:

  elasticsearch:
    enabled: true
    awsSigning:
      enabled: false                          # enable awsSigning using IAM for Elastisearch hosted on AWS, if true make sure AWS environment variables are set. Refer https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
      awsRegion: 'us-east-1'                  # AWS region where Elasticsearch is deployed
      roleArn: ''                             # AWS IAM Role arn to assume for credentials, use this only if you dont want to use the EC2 instance role or not running on AWS instance
    server: 'xxxx'           # e.g https://example.com:9243
    # ELS index settings
    index:
      name: botkube
      type: botkube-event
      shards: 1
      replicas: 0

I disabled the auth totally.

I will again try with Basic Auth. Meanwhile @soq can you help me with the steps for reproducing the issue?

[sushrut111]

Could not reproduce event with basic auth

  elasticsearch:
    enabled: true
    awsSigning:
      enabled: false                          # enable awsSigning using IAM for Elastisearch hosted on AWS, if true make sure AWS environment variables are set. Refer https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
      awsRegion: 'us-east-1'                  # AWS region where Elasticsearch is deployed
      roleArn: ''                             # AWS IAM Role arn to assume for credentials, use this only if you dont want to use the EC2 instance role or not running on AWS instance
    server: 'https://search-anothertestdomain-3akoieanofize5qrim3jmnrbfq.us-east-2.es.amazonaws.com:443'           # e.g https://example.com:9243
    username: 'sushrut'        # Basic Auth
    password: 'xxxxxxx'
    # ELS index settings
    index:
      name: botkube
      type: botkube-event
      shards: 1
      replicas: 0

[soq]

Hello @sushrut111 !

I use following config:

elasticsearch:
  enabled: true
  server: 'https://some_aws_name.eu-west-1.es.amazonaws.com:443'             
  username: ''          
  password: ''
  index:
    name: botkube
    type: botkube-event
    shards: 1
    replicas: 0

And we use AWS ElasticSearch version 6.7. Here are some related ES parameters that I can see:

Allow APIs that can span multiple indices and bypass index-specific access policies: true
Fielddata cache allocation: unbounded (default)
Max clause count: 1024 (default)

[sushrut111]

Thank you for the information. I will check with version 6.7 with no auth.

[sushrut111]

I could reproduce the issue and the issue seems to be coming from the olivere/elasticsearch module. I am going to check this with different versions of that module. @PrasadG193 Is there some doc where the compatibility of elasticsearch versions with the botkube has been mentioned? The botkube clearly works with AWS Elasticsearch 7.x.x but not with 6.7.0.

[sushrut111]

Moreover the issue is specific to aws es and the general oss build of es:

$ curl http://localhost:9200
{
  "name" : "A7Q7Ws4",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "lkHIvnEeTJGNb2WgrS0IeQ",
  "version" : {
    "number" : "6.7.0",
    "build_flavor" : "oss",
    "build_type" : "docker",
    "build_hash" : "8453f77",
    "build_date" : "2019-03-21T15:32:29.844721Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

$ curl http://localhost:9200/botkube/_search?pretty
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "botkube",
        "_type" : "botkube-event",
        "_id" : "GhglUXUBbnGlB_sEGu9i",
        "_score" : 1.0,
        "_source" : {
          "Code" : "",
          "Title" : "v1/secrets deleted",
          "Kind" : "Secret",
          "Name" : "default-token-4764p",
          "Namespace" : "testns123",
          "Messages" : null,
          "Type" : "delete",
          "Reason" : "",
          "Error" : "",
          "Level" : "critical",
          "Cluster" : "not-configured",
          "Channel" : "",
          "TimeStamp" : "2020-10-22T21:59:42.199555+05:30",
          "Count" : 0,
          "Action" : "",
          "Recommendations" : null,
          "Warnings" : null
        }
      }
    ]
  }
}