search

OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.32k stars 932 forks source link

Connector page in GUI not available after proxy set up #6185

Closed emo-sec closed 7 months ago

emo-sec commented 7 months ago

Description

Hello, We're experiencing an issue with accessing the connector page in the GUI after setting the proxy env variable for the platform in the docker-compose.yml.

We set up the env variable to be able to receive RSS feed (which works) but give the above error then. It seems to be the only issue.

Environment

  1. OS (where OpenCTI server runs): Debian GNU/Linux 12 (bookworm)
  2. OpenCTI version: 6.0.0
  3. OpenCTI client: n/a
  4. Other environment details: We're operating behind a proxy.

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Added the following in the docker-compose.yml: 
    opencti:
image: opencti/platform:6.0.0
environment:
  - "HTTP_PROXY=http://xxx:xxx@xxx:8010/"
  - "HTTPS_PROXY=http://xxx:xxx@xxx:8010/"
  - "NO_PROXY=elasticsearch,redis,minio,rabbitmq,opencti,worker,localhost,127.0.0.1"

Expected Output

Having access to connectors in the GUI.

Actual Output

See below screenshot.

Event from platform when trying to access "connectors in the GUI": pencti_opencti.1.zxz6swnbwbqw@opencti3 | {"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Request failed with status code 404","name":"UNKNOWN_ERROR","stack":"UNKNOWN_ERROR: Request failed with status code 404\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at UnknownError (/opt/opencti/build/src/config/errors.js:76:47)\n at Object._logWithError (/opt/opencti/build/src/config/conf.js:331:23)\n at Object.error (/opt/opencti/build/src/config/conf.js:341:48)\n at Object.willSendResponse (/opt/opencti/build/src/graphql/loggerPlugin.js:115:20)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Promise.all (index 1)\n at S (/opt/opencti/build/node_modules/apollo-server-core/src/requestPipeline.ts:530:5)\n at processHTTPRequest (/opt/opencti/build/node_modules/apollo-server-core/src/runHttpQuery.ts:437:24)"},{"message":"Request failed with status code 404","name":"AxiosError","stack":"AxiosError: Request failed with status code 404\n at settle (/opt/opencti/build/node_modules/axios/lib/core/settle.js:19:12)\n at IncomingMessage.handleStreamEnd (/opt/opencti/build/node_modules/axios/lib/adapters/http.js:589:11)\n at IncomingMessage.emit (node:events:530:35)\n at endReadableNT (node:internal/streams/readable:1696:12)\n at processTicksAndRejections (node:internal/process/task_queues:82:21)\n at Jvn.request (/opt/opencti/build/node_modules/axios/lib/core/Axios.js:45:41)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at metricApi (/opt/opencti/build/src/database/rabbitmq.js:115:22)\n at getMetrics (/opt/opencti/build/src/domain/rabbitmqMetrics.js:7:17)"}],"inner_relation_creation":0,"level":"error","message":"Platform unmanaged direct error","operation":"WorkersStatusQuery","operation_query":"query WorkersStatusQuery{...WorkersStatus_data}fragment WorkersStatus_data on Query{elasticSearchMetrics{docs{count}search{query_total fetch_total}indexing{index_total delete_total}get{total}}rabbitMQMetrics{consumers overview{queue_totals{messages messages_ready messages_unacknowledged}message_stats{ack ack_details{rate}}}}}","size":2,"time":40,"timestamp":"2024-02-29T12:59:46.467Z","type":"READ_ERROR","user":{"group_ids":["ea370e59-bb4d-4bae-9d70-542613042a32"],"ip":"10.151.91.10","organization_ids":[],"referer":"https://xxx.xxx.xxx/./opencti3/dashboard/data/ingestion/connectors","socket":"query","user_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f","user_metadata":{}},"variables":{},"version":"6.0.0"}

Additional information

Proxy appears to be better handled in 6.0 than before but the issue in the GUI was already present in 5.12.X.

Screenshots (optional)

image

I hope the information given is clear enough. Please don't hesitate to ask if there is anything missing.

Thanks in advance for your help!

richard-julien commented 7 months ago

Hi @emo-sec , can you share your docker anonymized docked configuration? Thanks

emo-sec commented 7 months ago

Hello @richard-julien ,

Please find below our docker-compose.yml:

version: '3'
services:
  redis:
    image: redis:7.2.4
    restart: always
    volumes:
      - redisdata:/data
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      # Comment-out the line below for a cluster of multiple nodes
      - discovery.type=single-node
      # Uncomment the line below below for a cluster of multiple nodes
      # - cluster.name=docker-cluster
      - xpack.ml.enabled=false
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE} -Dhttps.proxyHost=xxx.xxx.xxx.xxx -Dhttps.proxyPort=8010 -Dhttps.proxyUser=xxx -Dhttps.proxyPassword=xxx"
    healthcheck:
       test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"]
       interval: 30s
       timeout: 30s
       retries: 3
    restart: always
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
  minio:
    image: minio/minio:RELEASE.2024-01-16T16-07-38Z
    volumes:
      - s3data:/data
    ports:
      - "9000:9000"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}    
    command: server /data
    restart: always
  rabbitmq:
    image: rabbitmq:3.12-management
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ_NODENAME=${RABBITMQ_DEFAULT_USER}
    volumes:
      - amqpdata:/var/lib/rabbitmq
    restart: always
  opencti:
    image: opencti/platform:6.0.0
    environment:
      - "HTTP_PROXY=http://xxx:xxx@xxx:8010/"
      - "HTTPS_PROXY=http://xxx:xxx@xxx:8010/"
      - "NO_PROXY=elasticsearch,redis,minio,rabbitmq,worker,localhost,127.0.0.1"
      - NODE_OPTIONS=--max-old-space-size=8096
      - APP__PORT=8080
      - APP__BASE_URL=${OPENCTI_BASE_URL}
      - APP__BASE_PATH=/opencti3
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - REDIS__HOSTNAME=redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=http://elasticsearch:9200
      - MINIO__ENDPOINT=minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=rabbitmq
      - RABBITMQ__PORT=5672
      - RABBITMQ__PORT_MANAGEMENT=15672
      - RABBITMQ__MANAGEMENT_SSL=false
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - SMTP__HOSTNAME=${SMTP_HOSTNAME}
      - SMTP__PORT=25
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy
    ports:
      - "8080:8080"
    depends_on:
      - redis
      - elasticsearch
      - minio
      - rabbitmq
    restart: always
  worker:
    image: opencti/worker:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/opencti3
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - WORKER_LOG_LEVEL=info
    depends_on:
      - opencti
    deploy:
      mode: replicated
      replicas: 3
    restart: always
  connector-export-file-stix:
    image: opencti/connector-export-file-stix:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/opencti3
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_STIX_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileStix2
      - CONNECTOR_SCOPE=application/json
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-export-file-csv:
    image: opencti/connector-export-file-csv:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/opencti3
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_CSV_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileCsv
      - CONNECTOR_SCOPE=text/csv
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-export-file-txt:
    image: opencti/connector-export-file-txt:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/opencti3
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_TXT_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileTxt
      - CONNECTOR_SCOPE=text/plain
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-import-file-stix:
    image: opencti/connector-import-file-stix:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/opencti3
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_IMPORT_FILE_STIX_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
      - CONNECTOR_NAME=ImportFileStix
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
      - CONNECTOR_SCOPE=application/json,text/xml
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-import-document:
    image: opencti/connector-import-document:6.0.0
    environment:
      - OPENCTI_URL=http://opencti:8080/opencti3
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_IMPORT_DOCUMENT_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
      - CONNECTOR_NAME=ImportDocument
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
      - CONNECTOR_SCOPE=application/pdf,text/plain,text/html
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_ONLY_CONTEXTUAL=false # Only extract data related to an entity (a report, a threat actor, etc.)
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
      - IMPORT_DOCUMENT_CREATE_INDICATOR=true
    restart: always
    depends_on:
      - opencti

volumes:
  esdata:
    driver: local
    driver_opts:
      type: none
      device: /data/opencti/elasticsearch/data
      o: bind
  s3data:
    driver: local
    driver_opts:
      type: none
      device: /data/opencti/s3/data
      o: bind
  redisdata:
    driver: local
    driver_opts:
      type: none
      device: /data/opencti/redis/data
      o: bind
  amqpdata:
    driver: local
    driver_opts:
      type: none
      device: /data/opencti/amqp/data
      o: bind
richard-julien commented 7 months ago

Hi @emo-sec , i confirmed the issue. Working on it for the next minor

fabinou commented 7 months ago

Hello

This is blocking in our environnement too, when do you expect to fix it ? We try to figure out a workaround, but didn't manage to make it work

Kalkran commented 7 months ago

This is blocking for us too (we need the proxy for OpenID connect to work) and I was really happy that the issue had already been identified, but am a little disappointed that the milestone keeps being pushed back. I've just now merged the branch above into the master-branch and rebuilt and this resolved the issues we were having (data/ingestion/connectors page and settings/parameter-page load again). We're hoping for a speedy merge.

richard-julien commented 7 months ago

Hi @Kalkran we do our best to fix and improve the product. I will try to finish the testing and merge it in the upcoming 6.0.8. As full proxy usage of the platform is not really common for people starting with opencti im also bit curious of your your professional usage of the platform.

Kalkran commented 7 months ago

That would be awesome. In the meantime I can resolve it by manually building/merging, but it would be easier to skip this step and resolve this for everyone. As to our usage, feel free to reach out on slack, but I don't think it's too special - we keep internal company data so we have to comply with the company's security policy - which include MFA (which we implement through OpenID Connect) and network segmentation.