5.3.7 -> 5.3.12 upgrade, but docker's state keep "running"(not healthy)

[misohouse]

I upgraded OpenCTI through Portainer. (5.3.7 -> 5.3.12)

There was no problem using the previous version, but I upgraded to fix the image upload error I asked last time.

I waited about 30 minutes after starting Containers, but I couldn't enter OpenCTI platform webpage.

Docker's state is below.

A docker-compose code is below. (I marked the changes with bold and italic)

I simply changed 5.3.7 to 5.3.12.

Please check my situation and reply a message.

Thanks :)

version: '3' services: redis: image: redis:7.0.0 restart: always volumes:

• redisdata:/data elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.17.4 volumes:
• esdata:/usr/share/elasticsearch/data environment:

  Comment out the line below for single-node

• discovery.type=single-node

  Uncomment line below below for a cluster of multiple nodes

  - cluster.name=docker-cluster

• xpack.ml.enabled=false
• "ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE}" restart: always ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 minio: image: minio/minio:RELEASE.2022-05-19T18-20-59Z volumes:
• s3data:/data ports:
• "9000:9000" environment: MINIO_ROOT_USER: ${MINIO_ROOT_USER} MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
  command: server /data healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 restart: always rabbitmq: image: rabbitmq:3.10-management environment:
• RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
• RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS} volumes:
• amqpdata:/var/lib/rabbitmq restart: always opencti: image: opencti/platform:5.3.12 environment:
• NODE_OPTIONS=--max-old-space-size=8096
• APP__PORT=8080
• APP__BASE_URL=${OPENCTI_BASE_URL}
• APPADMINEMAIL=${OPENCTI_ADMIN_EMAIL}
• APPADMINPASSWORD=${OPENCTI_ADMIN_PASSWORD}
• APPADMINTOKEN=${OPENCTI_ADMIN_TOKEN}
• APP__APP_LOGS__LOGS_LEVEL=error
• REDIS__HOSTNAME=redis
• REDIS__PORT=6379
• ELASTICSEARCH__URL=http://elasticsearch:9200
• MINIO__ENDPOINT=minio
• MINIO__PORT=9000
• MINIO__USE_SSL=false
• MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
• MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
• RABBITMQ__HOSTNAME=rabbitmq
• RABBITMQ__PORT=5672
• RABBITMQ__PORT_MANAGEMENT=15672
• RABBITMQ__MANAGEMENT_SSL=false
• RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
• RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
• SMTP__HOSTNAME=${SMTP_HOSTNAME}
• SMTP__PORT=25
• PROVIDERSLOCALSTRATEGY=LocalStrategy ports:
• "8080:8080" depends_on:
• redis
• elasticsearch
• minio
• rabbitmq restart: always worker: image: opencti/worker:5.3.12 environment:
• OPENCTI_URL=http://opencti:8080
• OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
• WORKER_LOG_LEVEL=info depends_on:
• opencti deploy: mode: replicated replicas: 3 restart: always connector-export-file-stix: image: opencti/connector-export-file-stix:5.3.12 environment:
• OPENCTI_URL=http://opencti:8080
• OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
• CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_STIX_ID} # Valid UUIDv4
• CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
• CONNECTOR_NAME=ExportFileStix2
• CONNECTOR_SCOPE=application/json
• CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
• CONNECTOR_LOG_LEVEL=info restart: always depends_on:
• opencti connector-export-file-csv: image: opencti/connector-export-file-csv:5.3.12 environment:
• OPENCTI_URL=http://opencti:8080
• OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
• CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_CSV_ID} # Valid UUIDv4
• CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
• CONNECTOR_NAME=ExportFileCsv
• CONNECTOR_SCOPE=text/csv
• CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
• CONNECTOR_LOG_LEVEL=info restart: always depends_on:
• opencti connector-export-file-txt: image: opencti/connector-export-file-txt:5.3.12 environment:
• OPENCTI_URL=http://opencti:8080
• OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
• CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_TXT_ID} # Valid UUIDv4
• CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
• CONNECTOR_NAME=ExportFileTxt
• CONNECTOR_SCOPE=text/plain
• CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
• CONNECTOR_LOG_LEVEL=info restart: always depends_on:

• opencti

  connector-import-file-stix: image: opencti/connector-import-file-stix:5.3.12 environment:

• OPENCTI_URL=http://opencti:8080
• OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
• CONNECTOR_ID=${CONNECTOR_IMPORT_FILE_STIX_ID} # Valid UUIDv4
• CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
• CONNECTOR_NAME=ImportFileStix
• CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
• CONNECTOR_SCOPE=application/json,text/xml
• CONNECTOR_AUTO=true # Enable/disable auto-import of file
• CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
• CONNECTOR_LOG_LEVEL=info restart: always depends_on:

• opencti

  connector-import-document: image: opencti/connector-import-document:5.3.12 environment:

• OPENCTI_URL=http://opencti:8080
• OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
• CONNECTOR_ID=${CONNECTOR_IMPORT_DOCUMENT_ID} # Valid UUIDv4
• CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
• CONNECTOR_NAME=ImportDocument
• CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
• CONNECTOR_SCOPE=application/pdf,text/plain,text/html
• CONNECTOR_AUTO=true # Enable/disable auto-import of file
• CONNECTOR_ONLY_CONTEXTUAL=false # Only extract data related to an entity (a report, a threat actor, etc.)
• CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
• CONNECTOR_LOG_LEVEL=info
• IMPORT_DOCUMENT_CREATE_INDICATOR=true restart: always depends_on:

• opencti

  connector-mitre: image: opencti/connector-mitre:5.3.12 environment:

• OPENCTI_URL=http://opencti:8080
• OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
• CONNECTOR_ID=${CONNECTOR_IMPORT_MITRE_ID}
• CONNECTOR_TYPE=EXTERNAL_IMPORT
• CONNECTOR_NAME=MITRE Datasets
• CONNECTOR_SCOPE=marking-definition,identity,attack-pattern,course-of-action,intrusion-set,campaign,malware,tool,report,external-reference-as-report
• CONNECTOR_CONFIDENCE_LEVEL=75 # From 0 (Unknown) to 100 (Fully trusted)
• CONNECTOR_UPDATE_EXISTING_DATA=false
• CONNECTOR_RUN_AND_TERMINATE=false
• CONNECTOR_LOG_LEVEL=info
• MITRE_ENTERPRISE_FILE_URL=https://raw.githubusercontent.com/mitre/cti/master/enterprise-attack/enterprise-attack.json
• MITRE_PRE_ATTACK_FILE_URL=https://raw.githubusercontent.com/mitre/cti/master/pre-attack/pre-attack.json
• MITRE_MOBILE_ATTACK_FILE_URL=https://raw.githubusercontent.com/mitre/cti/master/mobile-attack/mobile-attack.json
• MITRE_ICS_ATTACK_FILE_URL=https://raw.githubusercontent.com/mitre/cti/master/ics-attack/ics-attack.json
• MITRE_CAPEC_FILE_URL=https://raw.githubusercontent.com/mitre/cti/master/capec/2.1/stix-capec.json
• MITRE_INTERVAL=7 # In days, must be strictly greater than 1 restart: always depends_on:
• opencti

volumes: esdata: s3data: redisdata: amqpdata:

[misohouse]

First, I upgraded opencti worker, platform to 5.3.12 and that's worked.

But, after changing other images, OpenCTI did not worked.

Even though I rollbacked to 5.3.7, it did not worked.

The images seem to download fine.

[misohouse]

OpenCTI container log below.

{"category":"APP","error":{"context":{"category":"technical","error":{"_error":{},"_showLocations":false,"_showPath":false,"data":{"body":{"script":{"params":{"entity_type":"User","internal_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f","password":"$2a$10$KdR.2jDqmzGs4WDRkPUSR.QN0C3nQio9uA6nDQXfzDAtBLM7GpA6G","updated_at":"2022-09-14T04:15:27.067Z"},"source":"ctx._source['internal_id'] = params['internal_id'];ctx._source['entity_type'] = params['entity_type'];ctx._source['password'] = params['password'];ctx._source['updated_at'] = params['updated_at']"}},"category":"technical","documentId":"88ec0c6a-13ce-5e39-b486-354fe4a7084f","error":{"meta":{"body":{"error":{"reason":"index [opencti_internal_objects-000001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];","root_cause":[{"reason":"index [opencti_internal_objects-000001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];","type":"cluster_block_exception"}],"type":"cluster_block_exception"},"status":429},"headers":{"content-length":"445","content-type":"application/json; charset=UTF-8","warning":"299 Elasticsearch-7.17.4-79878662c54c886ae89206c685d9f1051a9d6411 \"Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.\"","x-elastic-product":"Elasticsearch"},"meta":{"aborted":false,"attempts":0,"connection":{"headers":{},"id":"http://elasticsearch:9200/","status":"alive","url":"http://elasticsearch:9200/"},"context":null,"name":"elasticsearch-js","request":{"id":11,"options":{},"params":{"body":"{\"script\":{\"source\":\"ctx._source['internal_id'] = params['internal_id'];ctx._source['entity_type'] = params['entity_type'];ctx._source['password'] = params['password'];ctx._source['updated_at'] = params['updated_at']\",\"params\":{\"internal_id\":\"88ec0c6a-13ce-5e39-b486-354fe4a7084f\",\"entity_type\":\"User\",\"password\":\"$2a$10$KdR.2jDqmzGs4WDRkPUSR.QN0C3nQio9uA6nDQXfzDAtBLM7GpA6G\",\"updated_at\":\"2022-09-14T04:15:27.067Z\"}}}","headers":{"accept":"application/vnd.elasticsearch+json; compatible-with=8","content-length":"418","content-type":"application/vnd.elasticsearch+json; compatible-with=8","user-agent":"elastic-transport-js/8.2.0 (linux 5.4.0-122-generic-x64; Node.js v16.17.0)","x-elastic-client-meta":"es=8.2.1,js=16.17.0,t=8.2.0,hc=16.17.0"},"method":"POST","path":"/opencti_internal_objects-000001/_update/88ec0c6a-13ce-5e39-b486-354fe4a7084f","querystring":"retry_on_conflict=5&timeout=5m&refresh=true"}}},"statusCode":429,"warnings":["299 Elasticsearch-7.17.4-79878662c54c886ae89206c685d9f1051a9d6411 \"Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security.\""]},"name":"ResponseError"},"http_status":500,"reason":"Error updating elastic"},"internalData":{},"name":"DatabaseError","time_thrown":"2022-09-14T04:15:27.078Z"},"http_status":500,"reason":"[OPENCTI] Platform initialization fail"},"message":"An unknown error has occurred","name":"UnknownError","stack":"UnknownError: An unknown error has occurred\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at UnknownError (/opt/opencti/build/src/config/errors.js:61:47)\n at platformInit (/opt/opencti/build/src/initialization.js:341:13)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at boot (/opt/opencti/build/src/boot.js:14:5)"},"level":"error","message":"[OPENCTI] Platform start fail","timestamp":"2022-09-14T04:15:27.082Z","version":"5.3.12"}

[misohouse]

I did something(erase and reinstall docker images, restart docker etc), and now it worked with opencti-worker/platform 5.3.12.

I don't know why it is working properly... :(

By the way, why content's table looks like below? (it looks like round and thick lines)

When I click the square located at the top left of the table, it looks like a normal table, but when I release the mouse button, it returns to its original shape.

Should I upgrade other images to 5.3.12?

• connector-export-file-stix (5.3.7)
• connector-export-file-csv (5.3.7)
• connector-export-file-txt (5.3.7)
• connector-import-file-stix (5.3.7)
• connector-import-document (5.3.7)
• connector-mitre (5.3.7)

I'm scared that error is occurred again when I upgrade above images all at once.

Please tell me name of image what I upgrade.

[richard-julien]

Looking your logs.

[TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]; Basically everything fail because elastic tell you that you dont have space on your disk....

For the images, you need to upgrade everything, platform and connectors.

[misohouse]

Thank you for comment! I'll add harddisk's capacity of VM image and test again!

[misohouse]

Looking your logs.

[TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]; Basically everything fail because elastic tell you that you dont have space on your disk....

For the images, you need to upgrade everything, platform and connectors.

Hi, I upgraded everything to 5.3.12.

But, table still looks weird... :(

How can I fix this??

[richard-julien]

Thats a different problem. You can create another issue for that. Please create a ticket and join in attachment the html file that is not correctly rendered. Thanks