gritty-Kitty commented 7 months ago

Prerequisites

[X] I read the Deployment and Setup section of the OpenCTI documentation as well as the Troubleshooting page and didn't find anything relevant to my problem.
[X] I went through old GitHub issues and couldn't find anything relevant
[X] I googled the issue and didn't find anything relevant

Description

Has anyone seen this, and if yes, were you able to resolve? Currently investigating...

In Microsoft Azure, installed OpenCTI v6.0.5 on Ubuntu 22.04 LTS (Azure Marketplace) with Docker CE 25.0.3/Portainer CE 2.19.4, and placed behind an Application Gateway/WAF. OpenCTI configured for HTTPS using Entrust 3rd Party Certificate.

OpenCTI loads with HTTPS behind Application Gateway without issue.

From Internet, OpenCTI does NOT load with HTTPS with GraphQL 403 Forbidden messages. Website is white and appears stuck in frantic continuous loading loop that cannot be stopped in either MSIE or CHROME.

Thank you.

Environment

OS (where OpenCTI server runs): { e.g. Mac OS 10, Windows 10, Ubuntu 16.4, etc. }

Microsoft Azure Marketplace Offer = '0001-com-ubuntu-server-jammy' PublisherName = 'Canonical' Skus = '22_04-lts-gen2' Version = 'latest'

OpenCTI version: { e.g. OpenCTI 1.0.2 }

OpenCTI 6.0.5 N.B. Attempted to pull 6.0.6, but Portainer/Docker complains it could not find 6.0.6 on 12 March 2024. Forced to back reference docker-compose.yml to 6.0.5. Referenced from... https://github.com/OpenCTI-Platform/docker

OpenCTI client: { e.g. frontend or python }

Using MSIE and CHROME to access OpenCTI through Azure Application Gateway to backend.

Other environment details:

Docker CE 25.0.3 Portainer CE 2.19.4

Reproducible Steps

Steps to create the smallest reproducible scenario:

{ e.g. Run ... }

Use MSIE or CHROME to access site... https://fqdn. Site will attempt to load, redirect to https://fqdn/dashboard, and then hard cycle over and over again with white web page.

{ e.g. Click ... }

N/A

{ e.g. Error ... }

Additional information

Attempted to configure internal docker-compose.yml to utilize https://fsqn of site, but does not resolve. Reverted to using docker-compose.yml and docker-compose.env below.

Found a handful of internet references related to GraphQL 403 Forbidden that all hint towards code changes related to Authenticated requests, but cannot appreciate content or resolution they are describing.

https://github.com/wp-graphql/wp-graphql/issues/262 https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referrer-Policy

BTW, after many many days of playing with HTTPS with near little examples, finally found the secret sauce. For anyone interested, this is a good place to start.

-=docker-compose.yml (anonymized)=-

version: '3'

version 6.0.6 does not appear to exist, using 6.0.5

services: redis: image: redis:7.2.4 command:

bash
-c
redis-server --appendonly yes --maxmemory 5000000000 --maxmemory-policy volatile-lru
- redis-server config set client-output-buffer-limit "slave 836870912 836870912 0"

restart: always volumes:
redisdata:/data elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2 volumes:
esdata:/usr/share/elasticsearch/data environment:
Comment-out the line below for a cluster of multiple nodes
discovery.type=single-node
Uncomment the line below below for a cluster of multiple nodes

- cluster.name=docker-cluster
xpack.ml.enabled=false
xpack.security.enabled=false
thread_pool.search.queue_size=5000
logger.org.elasticsearch.discovery="ERROR"
"ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE}"
https://github.com/OpenCTI-Platform/opencti/issues/6185

healthcheck: test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"] interval: 30s timeout: 30s retries: 3 restart: always ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 minio: image: minio/minio:RELEASE.2024-01-16T16-07-38Z volumes:
s3data:/data ports:
"9000:9000" environment: MINIO_ROOT_USER: ${MINIO_ROOT_USER} MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
command: server /data
https://github.com/minio/minio/issues/18373

healthcheck: test: timeout 5s bash -c ':> /dev/tcp/127.0.0.1/9000' || exit 1 interval: 5s retries: 1 start_period: 5s timeout: 5s restart: always rabbitmq: image: rabbitmq:3.13-management environment:
RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
RABBITMQ_NODENAME=rabbit01@localhost
RABBITMQ_CONSUMER_TIMEOUT=36000000 volumes:
amqpdata:/var/lib/rabbitmq restart: always opencti: image: opencti/platform:6.0.5 environment:
NODE_OPTIONS=--max-old-space-size=8096
APP__PORT=8080
APP__BASE_URL=${OPENCTI_BASE_URL}
APPADMINEMAIL=${OPENCTI_ADMIN_EMAIL}
APPADMINPASSWORD=${OPENCTI_ADMIN_PASSWORD}
APPADMINTOKEN=${OPENCTI_ADMIN_TOKEN}
APP__APP_LOGS__LOGS_LEVEL=error
APP__HTTPS_CERT__CA=${OPENCTI_CA_PATHS}
APP__HTTPS_CERT__KEY=${OPENCTI_KEY_PATH}
APP__HTTPS_CERT__CRT=${OPENCTI_CRT_PATH}
APP__HTTPS_CERT__REJECT_UNAUTHORIZED=true
REDIS__HOSTNAME=redis
REDIS__PORT=6379
ELASTICSEARCH__URL=http://elasticsearch:9200
MINIO__ENDPOINT=minio
MINIO__PORT=9000
MINIO__USE_SSL=false
MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
RABBITMQ__HOSTNAME=rabbitmq
RABBITMQ__PORT=5672
RABBITMQ__PORT_MANAGEMENT=15672
RABBITMQ__MANAGEMENT_SSL=false
RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
SMTP__HOSTNAME=${SMTP_HOSTNAME}
SMTP__PORT=25
PROVIDERSLOCALSTRATEGY=LocalStrategy ports:
"443:8080" depends_on:
redis
elasticsearch
minio
rabbitmq restart: always volumes:
:ro makes the volume read only

/var/lib/docker/volumes/opencti_opencti_https/_data/
opencti_https:/certs:ro worker: image: opencti/worker:6.0.5 environment:
OPENCTI_URL=${OPENCTI_ADMIN_URL}
OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
WORKER_LOG_LEVEL=info depends_on:
opencti deploy: mode: replicated replicas: 4 restart: always connector-export-file-stix: image: opencti/connector-export-file-stix:6.0.5 environment:
OPENCTI_URL=${OPENCTI_ADMIN_URL}
OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_STIX_ID} # Valid UUIDv4
CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
CONNECTOR_NAME=ExportFileStix2
CONNECTOR_SCOPE=application/json
CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
CONNECTOR_LOG_LEVEL=info restart: always depends_on:
opencti connector-export-file-csv: image: opencti/connector-export-file-csv:6.0.5 environment:
OPENCTI_URL=${OPENCTI_ADMIN_URL}
OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_CSV_ID} # Valid UUIDv4
CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
CONNECTOR_NAME=ExportFileCsv
CONNECTOR_SCOPE=text/csv
CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
CONNECTOR_LOG_LEVEL=info restart: always depends_on:
opencti connector-export-file-txt: image: opencti/connector-export-file-txt:6.0.5 environment:
OPENCTI_URL=${OPENCTI_ADMIN_URL}
OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_TXT_ID} # Valid UUIDv4
CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
CONNECTOR_NAME=ExportFileTxt
CONNECTOR_SCOPE=text/plain
CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
CONNECTOR_LOG_LEVEL=info restart: always depends_on:
opencti connector-import-file-stix: image: opencti/connector-import-file-stix:6.0.5 environment:
OPENCTI_URL=${OPENCTI_ADMIN_URL}
OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
CONNECTOR_ID=${CONNECTOR_IMPORT_FILE_STIX_ID} # Valid UUIDv4
CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
CONNECTOR_NAME=ImportFileStix
CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
CONNECTOR_SCOPE=application/json,text/xml
CONNECTOR_AUTO=true # Enable/disable auto-import of file
CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
CONNECTOR_LOG_LEVEL=info restart: always depends_on:
opencti connector-import-document: image: opencti/connector-import-document:6.0.5 environment:
OPENCTI_URL=${OPENCTI_ADMIN_URL}
OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
CONNECTOR_ID=${CONNECTOR_IMPORT_DOCUMENT_ID} # Valid UUIDv4
CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
CONNECTOR_NAME=ImportDocument
CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
CONNECTOR_SCOPE=application/pdf,text/plain,text/html
CONNECTOR_AUTO=true # Enable/disable auto-import of file
CONNECTOR_ONLY_CONTEXTUAL=false # Only extract data related to an entity (a report, a threat actor, etc.)
CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
CONNECTOR_LOG_LEVEL=info
IMPORT_DOCUMENT_CREATE_INDICATOR=true restart: always depends_on:
opencti

volumes: esdata: s3data: redisdata: amqpdata: opencti_https:

-=docker-compose.env (anonymized)=-

OPENCTI_ADMIN_EMAIL= OPENCTI_ADMIN_PASSWORD= OPENCTI_ADMIN_TOKEN= OPENCTI_BASE_URL=https://localhost OPENCTI_ADMIN_URL=http://opencti:8080

made no difference if used

OPENCTI_BASE_URL=https://fqdn

OPENCTI_ADMIN_URL=https://fqdn

OPENCTI_CA_PATHS=["/certs/Root.crt"] OPENCTI_KEY_PATH=/certs/opencti.key OPENCTI_CRT_PATH=/certs/opencti.crt MINIO_ROOT_USER=opencti MINIO_ROOT_PASSWORD= RABBITMQ_DEFAULT_USER=opencti RABBITMQ_DEFAULT_PASS= CONNECTOR_EXPORT_FILE_STIX_ID= CONNECTOR_EXPORT_FILE_CSV_ID= CONNECTOR_EXPORT_FILE_TXT_ID= CONNECTOR_IMPORT_FILE_STIX_ID= CONNECTOR_IMPORT_DOCUMENT_ID= SMTP_HOSTNAME=localhost ELASTIC_MEMORY_SIZE=6G

gritty-Kitty commented 7 months ago

Don't know if this means anything... but double-clicking on ANY of the MSIE F12 'graphql' line items (from the output/screenshot above) will display the following...

gritty-Kitty commented 7 months ago

Reconfigured OpenCTI to use HTTP and same problem exists.

gritty-Kitty commented 7 months ago

Good morning... despite DISABLING ALL WAF Policies on the Azure Application Gateway the problem persisted with GraphQL 403 Forbidden. Researched how to setup a custom response rule, but could not identify secret sauce.

DISABLING WAF and switching from WAV_v2 to Standard_v2 -has resolved the issue- for allowing OpenCTI to be viewable from the internet on HTTP -and- HTTPS.

This thread can be -closed- as we will not be investigating WAF any further due to deployment pressure.

Thank you.

OpenCTI-Platform / opencti

GraphQL 403 Forbidden on POST Referrer Policy unsafe-url -- OpenCTI HTTPS #6359

Prerequisites

Description

Environment

Reproducible Steps

Additional information

version 6.0.6 does not appear to exist, using 6.0.5

- redis-server config set client-output-buffer-limit "slave 836870912 836870912 0"

Comment-out the line below for a cluster of multiple nodes

Uncomment the line below below for a cluster of multiple nodes

- cluster.name=docker-cluster

https://github.com/OpenCTI-Platform/opencti/issues/6185

https://github.com/minio/minio/issues/18373

:ro makes the volume read only

/var/lib/docker/volumes/opencti_opencti_https/_data/

made no difference if used

OPENCTI_BASE_URL=https://fqdn

OPENCTI_ADMIN_URL=https://fqdn