Unable to connect S3 - Leads to various access denied issues

Geordeaux commented 9 months ago

Prerequisites

[x] I read the Deployment and Setup section of the OpenCTI documentation as well as the Troubleshooting page and didn't find anything relevant to my problem.
[x] I went through old GitHub issues and couldn't find anything relevant
[x] I googled the issue and didn't find anything relevant
[x] I have posted questions in the Slack multiple times without a solution being found

Description

I have been trying to get an S3 attached to my instance of OpenCTI now for weeks and have not been able to get any solution working.

I have an existing S3 with full access permissions. Tested and confirmed the permissions are working. The server is able to access that IAM role.

However whenever I try varius configurationins in the config file nothing seems to work. Whenever the S3 config variables are uncommented dcker just returns:

docker-opencti-1                     | {"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Access Denied","name":"UNKNOWN_ERROR","stack":"UNKNOWN_ERROR: Access Denied\n    at error (/opt/opencti/build/src/config/errors.js:8:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:70:47)\n    at Object._logWithError (/opt/opencti/build/src/config/conf.js:311:23)\n    at Object.error (/opt/opencti/build/src/config/conf.js:321:48)\n    at platformStart (/opt/opencti/build/src/boot.js:236:12)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)"},{"message":"Access Denied","name":"AccessDenied","stack":"AccessDenied: Access Denied\n    at throwDefaultError (/opt/opencti/build/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:8:22)\n    at throwDefaultError (/opt/opencti/build/node_modules/@smithy/smithy-client/dist-cjs/default-error-handler.js:18:21)\n    at de_CreateBucketCommandError (/opt/opencti/build/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:3226:20)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at /opt/opencti/build/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24\n    at /opt/opencti/build/node_modules/@aws-sdk/middleware-signing/dist-cjs/awsAuthMiddleware.js:30:20\n    at /opt/opencti/build/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:31:46\n    at /opt/opencti/build/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-endpoint-middleware.js:14:24\n    at /opt/opencti/build/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/region-redirect-middleware.js:9:20\n    at /opt/opencti/build/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26\n    at initializeBucket (/opt/opencti/build/src/database/file-storage.js:86:5)\n    at checkSystemDependencies (/opt/opencti/build/src/initialization.js:130:3)\n   at platformStart (/opt/opencti/build/src/boot.js:228:5)"}],"level":"error","message":"Platform unmanaged direct error","timestamp":"2024-01-14T15:30:50.467Z","version":"5.12.15"}

Environment

OS: Ubuntu
OpenCTI version: 5.12.15 (also tried various ealier versions)
OpenCTI client: Frontend

Other environment details: Deployed on AWS EC2. EC2 has full IAM permissions to the attached S3 Bucket. I have tested and confirmed all S3 actions from the server and all are working.

opencti:
image: opencti/platform:5.12.15
environment:
  - NODE_OPTIONS=--max-old-space-size=8096
  - APP__PORT=8080
  - APP__BASE_URL=${OPENCTI_BASE_URL}
  - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
  - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
  - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
  - APP__APP_LOGS__LOGS_LEVEL=error
  - REDIS__HOSTNAME=redis
  - REDIS__PORT=6379
  - ELASTICSEARCH__URL=http://elasticsearch:9200
  - MINIO__ENDPOINT=minio
  - MINIO__PORT=9000
  - MINIO__USE_SSL=false
  - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
  - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
  - MINIO__ENDPOINT=s3.amazonaws.com
  - MINIO_BUCKET_NAME=sec-opencti
  - MINIO__BUCKET_REGION=us-northeast-1
  - MINIO__USE_AWS_ROLE=true
  - RABBITMQ__HOSTNAME=rabbitmq
  - RABBITMQ__PORT=5672
  - RABBITMQ__PORT_MANAGEMENT=15672
  - RABBITMQ__MANAGEMENT_SSL=false
  - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
  - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
  - SMTP__HOSTNAME=${SMTP_HOSTNAME}
  - SMTP__PORT=25
  - PROVIDERS__LOCAL__STRATEGY=LocalStrategy

Has anyone has any success in actually getting an S3 bucket attached to OpenCTI? If so can you please share your compose files for reference?

Any help is appreciated

Geordeaux commented 9 months ago

Is there some unspoken requirement that I need to use s3fs to bind to the s3data volume? I am thinking that this is because that while the server has full permissions to access the s3, the opencti container cannot.

nino-filigran commented 9 months ago

@Geordeaux Can you also raise your question on Slack? This will also help to get additional visibility to your question!

OpenCTI-Platform / opencti