[OpenCTI] Engine attachment processor fail

[remys89]

Describe the bug a clear and concise description of what the bug is.

We deployed OpenCTI using the helm chart, and we tried writing our own deployment files based on the docker variant, but we keep running into the same error following examples and guide. It seems like it has issues with elastichsearch for running the correct engine.

We run OpenCTI with elasticsearch, rabbitmq, redis on a Kubernetes Cluster running 1.28.4 on containerd 1.7.2 with Helm 3.7.2 using the follwing variables:

What's your helm version?

3.7.2

What's your kubectl version?

1.28.4

Chart name

devops-ia/opencti

Chart version

1.2.2

Explain bug

Deployed using helm as wel as writing on k8s files based on docker-compose for OpenCTI, but we run into the same issue on both situations. After deploying all the components, only opencti-server seems to fail after 60-90 seconds:

{"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Engine attachment processor configuration fail","name":"CONFIGURATION_ERROR","stack":"CONFIGURATION_ERROR: Engine attachment processor configuration fail\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at ConfigurationError (/opt/opencti/build/src/config/errors.js:70:53)\n at /opt/opencti/build/src/database/engine.js:816:20\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at elConfigureAttachmentProcessor (/opt/opencti/build/src/database/engine.js:804:5)\n at searchEngineInit (/opt/opencti/build/src/database/engine.js:292:32)\n at checkSystemDependencies (/opt/opencti/build/src/initialization.js:31:3)\n at platformStart (/opt/opencti/build/src/boot.js:14:7)"},{"message":"parse_exception\n\tRoot causes:\n\t\tparse_exception: No processor type exists with name [attachment]","name":"ResponseError","stack":"ResponseError: parse_exception\n\tRoot causes:\n\t\tparse_exception: No processor type exists with name [attachment]\n at Won.request (/opt/opencti/build/node_modules/@elastic/elasticsearch/node_modules/@elastic/transport/src/Transport.ts:553:17)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at ydn.putPipeline (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/ingest.ts:195:12)\n at elConfigureAttachmentProcessor (/opt/opencti/build/src/database/engine.js:804:5)\n at searchEngineInit (/opt/opencti/build/src/database/engine.js:292:32)\n at checkSystemDependencies (/opt/opencti/build/src/initialization.js:31:3)\n at platformStart (/opt/opencti/build/src/boot.js:14:7)"}],"level":"error","message":"Engine attachment processor configuration fail","timestamp":"2024-05-03T08:07:45.697Z","version":"6.0.10"} {"category":"APP","dependencyError":{"$metadata":{"attempts":3,"totalRetryDelay":188},"code":"ECONNREFUSED"},"level":"error","message":"[OPENCTI] System dependencies check failed","timestamp":"2024-05-03T08:07:46.197Z","version":"6.0.10"} {"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"An unknown error has occurred","name":"UNKNOWN_ERROR","stack":"UNKNOWN_ERROR: An unknown error has occurred\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at UnknownError (/opt/opencti/build/src/config/errors.js:76:47)\n at Object._logWithError (/opt/opencti/build/src/config/conf.js:331:23)\n at Object.error (/opt/opencti/build/src/config/conf.js:341:48)\n at platformStart (/opt/opencti/build/src/boot.js:47:12)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)"},{"message":"","name":"AggregateError","stack":"AggregateError: \n at internalConnectMultiple (node:net:1116:18)\n at afterConnectMultiple (node:net:1683:7)"}],"level":"error","message":"Platform unmanaged direct error","timestamp":"2024-05-03T08:07:46.198Z","version":"6.0.10"}

From other deployments in the stack, like i.e. redis or rabbitmq, we can reach the elasticsearch deployment and receive data when curling it.

What you expected to happen?

OpenCTI to start and serve webportal for usage.

How to reproduce it?

Deploy the helm chart using the values files provided below.

Enter the changed values of values.yaml?

# -- Environment variables to configure application
# ref: https://docs.opencti.io/5.12.X/deployment/configuration/#platform
env:
  # Plain vars
  # foo: bar
  # my_env: my_value
  # APP OPENCTI
  APP__ADMIN__EMAIL: test@company.com
  APP__ADMIN__PASSWORD: ***
  APP__ADMIN__TOKEN: ***
  APP__BASE_PATH: "/"
  APP__BASE_URL: "https://opencti.test.eu"
  OPENCTI_URL: "https://opencti.test.eu"
  # MINIO:
  MINIO__ENDPOINT: opencti-minio:9000
  MINIO_ACCESS_KEY: admin
  MINIO_SECRET_KEY: ***
  # ELASTICSEARCH
  ELASTICSEARCH__ENGINE_SELECTOR: elk
  ELASTICSEARCH__INDEX_PREFIX: opencti
  ELASTICSEARCH__URL: http://elasticsearch:9200
  # RABBITMQ
  RABBITMQ__HOSTNAME: opencti-rabbitmq
  RABBITMQ__PORT_MANAGEMENT: 15672
  RABBITMQ__PORT: 5672
  RABBITMQ__USERNAME: admin
  RABBITMQ__PASSWORD: ***
  # REDIS
  REDIS__HOSTNAME: opencti-redis-master
  REDIS__PORT: 6379
  REDIS__MODE: single
  # METRICS AND HEALTHCHECK
  APP__TELEMETRY__METRICS__ENABLED: true
  APP__HEALTH_ACCESS_KEY: ***

# -- ElasticSearch subchart deployment
# ref: https://github.com/bitnami/charts/blob/main/bitnami/elasticsearch/values.yaml
elasticsearch:
  # -- Enable or disable ElasticSearch subchart
  enabled: false

  ## Kernel settings modifier image
  ##
  sysctlImage:
    ## @param sysctlImage.enabled Enable kernel settings modifier image
    ##
    enabled: false

  # -- Elasticsearch cluster name
  clusterName: opencti-elastic

  extraEnvVars:
    - name: ES_JAVA_OPTS
      value: "-Xms512M -Xmx512M"

  ## @section Master-elegible nodes parameters
  master:
    # -- Deploy the Elasticsearch master-elegible nodes as master-only nodes. Recommended for high-demand deployments.
    masterOnly: true
    # -- Number of master-elegible replicas to deploy
    replicaCount: 1
    # -- Enable persistence using Persistent Volume Claims
    # ref: https://kubernetes.io/docs/user-guide/persistent-volumes/
    persistence:
      # -- Enable persistence using a `PersistentVolumeClaim`
      enabled: false

  # -- Data-only nodes parameters
  data:
    # -- Number of data-only replicas to deploy
    replicaCount: 1

    # -- Enable persistence using Persistent Volume Claims
    # ref: https://kubernetes.io/docs/user-guide/persistent-volumes/
    persistence:
      # -- Enable persistence using a `PersistentVolumeClaim`
      enabled: false

  # -- Ingest-only nodes parameters
  ingest:
    # --  Enable ingest nodes
    enabled: false

  # -- Coordinating-only nodes parameters
  coordinating:
    # -- Number of coordinating-only replicas to deploy
    replicaCount: 0

Enter the command that you execute when chart failing/misfunctioning

helm install opencti devops-ia/opencti -f values.yml -n namespace_name

More info

No response

[ialejandro]

@remys89, can you try this? It's weird that the rest of the services can reach the Elasticsearch service but not OpenCTI.

Enforce fullnameOverride flag to set the name which use on ELASTICSEARCH__URL environment.

env:
...
  ELASTICSEARCH__ENGINE_SELECTOR: elk
  ELASTICSEARCH__INDEX_PREFIX: opencti
  ELASTICSEARCH__URL: http://elasticsearch:9200

elasticsearch:
  enabled: true
  fullnameOverride: elasticsearch

If you see the logs: ECONNREFUSED. So, here's my CI values: https://github.com/devops-ia/helm-charts/blob/main/charts/opencti/ci/ci-values.yaml, maybe can help you to set values.

Tell me if doesn't work.

[remys89]

I deployed it using our own elasticsearch in the same namespace, but i still seem to run into the same error. I will investigate more tomorrow and get back to you.

[remys89]

OpenCTI seems to boot up just fine now, had to adjust the token to a valid UUIDv4 and it works now, with our own elasticsearch instance. Had to disable the ES instance in the helm chart, since it requires some additional config for memory reservation. We can close this if its up to me.

[ialejandro]

Thanks for the update @remys89!