Default Connector role not providing enough privilege

[FocusedPanda]

Description

Update 2023-01-17 - It appears that the default connector role does not provide enough privilege for some connectors to work properly. Issues with connectors ingesting data can be worked around by setting up a new role containing the "Bypass all capabilities" privilege checked. This is not ideal.

I have setup the OpenCTI Datasets connector; I am able to see the Connector performing work in OpenCTI, as well as the sessions for the connector user. However, none of the sectors of geographies are being ingested into OpenCTI.

Environment

1. OS: Debian 11 (bullseye)
2. OpenCTI version: 5.5.1
3. OpenCTI client: Frontend
4. Other environment details: All OpenCTI services are running in docker using Portainer; Elastic is in it's own Docker on a separate server.

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Create an OpenCTI user for OpenCTI Datasets connector with 'Connector' role.
2. Add OpenCTI Datasets docker-compose details to existing configuration.
3. Modify configuration to use appropriate OpenCTI admin token, and random UUIDv4.
4. Start docker swarm and start dockers

Expected Output

It is expected that the Locations -> Regions, Countries, and Cities sections in OpenCTI will be populated with data from https://raw.githubusercontent.com/OpenCTI-Platform/datasets/master/data/geography.json, and that Entities -> Sectors would be populated with data from https://raw.githubusercontent.com/OpenCTI-Platform/datasets/master/data/sectors.json.

Actual Output

• The OpenCTI datasets connector successfully connects to the platform according to reviewing user sessions.
• Work is being done by the connector according to the Connector screen.
• No operations are being performed by the Connector.
• The only errors showing up in the OpenCTI interface are related to unicode characters used in the ANSSI entry.

Additional information

When I checked the Elastic logs, I see the a log entry with the following error:

[{'name': 'ForbiddenAccess', 'message': 'You are not allowed to do this.'}, {'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'}, {'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'}, {'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'}, {'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'}, {'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'}, {'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'}, {'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'}, {'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'}, {'name': 'MissingReferenceError', ........................ ,  {'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'},  {'name': 'MissingReferenceError', 'message': 'Missing reference to handle creation'}]

That same log entry also had

Connector successfully run, storing last_run as 1673089379

OpenCTI Docker Configuration

opencti:
    image: opencti/platform:5.5.1
    environment:
      - NODE_OPTIONS=--max-old-space-size=${MAX_OLD_SPACE_SIZE}
      - APP__PORT=8080
      - APP__BASE_URL=${OPENCTI_BASE_URL}
      - APP__SESSION_TIMEOUT=${SESSION_TIMEOUT}
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - SUBSCRIPTION_SCHEDULER__ENABLED=${SUBSCRIPTION_SCHEDULER_ENABLED}
      - REDIS__HOSTNAME=redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=${ELASTICSEARCH_URL}
      - MINIO__ENDPOINT=minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=rabbitmq
      - RABBITMQ__PORT=5672
      - RABBITMQ__PORT_MANAGEMENT=15672
      - RABBITMQ__MANAGEMENT_SSL=false
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - SMTP__HOSTNAME=${SMTP_HOSTNAME}
      - SMTP__USE_SSL=${SMTP_USE_SSL}
      - SMTP__REJECT_UNAUTHORIZED=${SMTP_REJECT_ANAUTHORIZED}
      - SMTP__PORT=${SMTP_PORT}
      - SMTP__USERNAME=${SMTP_USERNAME}
      - SMTP__PASSWORD=${SMTP_PASSWORD}
      - SMTP__FROM_EMAIL=${SMTP_FROM_EMAIL}
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy
    ports:
      - "8080:8080"
    depends_on:
      - redis
      - minio
      - rabbitmq
    restart: always

OpenCTI Datasets Connector Configuration

# OpenCTI Connector
  connector-opencti:
    image: opencti/connector-opencti:5.5.1
    environment:
      - OPENCTI_URL=${OPENCTI_URL}
      - OPENCTI_TOKEN=${CONN_OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONN_OPENCTI_TOKEN}
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=OpenCTI
      - CONNECTOR_SCOPE=marking-definition,identity,location
      - CONNECTOR_CONFIDENCE_LEVEL=50 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=true
      - CONNECTOR_RUN_AND_TERMINATE=false
      - CONNECTOR_LOG_LEVEL=info
      - CONFIG_SECTORS_FILE_URL=https://raw.githubusercontent.com/OpenCTI-Platform/datasets/master/data/sectors.json
      - CONFIG_GEOGRAPHY_FILE_URL=https://raw.githubusercontent.com/OpenCTI-Platform/datasets/master/data/geography.json
      - CONFIG_INTERVAL=7 # In days, must be strictly greater than 1
    restart: always
    depends_on:
      - opencti

Screenshots (optional)

[tialocRT]

Hello,

I have the same problem in a docker environment v5.5.2 (16GB Ram - 6 vCPU - 50GO disk) with the misp-feed connector when i assign the built-in collector role in opencti, I try to give it more rights but it doesn't work (see pic ofr rights). The connector works well when it has the "Bypass all capabilities" right, does the misp-feed connector need to have admin rights to work or am I missing something ?

Thanks by advance

[FocusedPanda]

Confirmed.

1. I created a new user role that included the 'Bypass all capabilities' capability selected. (With that selected, I'm not sure if any of the other capabilities matter)

2. I updated my [C] OpenCTI user with the new role that contains, 'Bypass all capabilities'.

3. I cleared all works from the OpenCTI Connector, and reset the connector state.

4. Once the OpenCTI connector re-ran, data was successfully being ingested.

This would lead me to believe that their is a capability missing from the role definitions, that allows writing of data from certain connectors.

Connectors that I've found this issue with:

• OpenCTI Datasets
• Mitre
• Disarm Framework
• Amitt

[tialocRT]

Hello thanks for the answer and for the tip with the clear jobs and reset connector state. The right "Bypass all capabilities" is the right given to the Admin role, so we are giving an admin right to a connector. I hope another solution will be found in the future.

[0xBEN]

Can confirm that I've also been experiencing this issue with the AlienVault connector. After some lengthy (and frustrating) troubleshooting, I concluded that the only way to resolve the errors was to permit Bypass all capabilities. However, being that this capability is overly permissive, I searched the GitHub repo, and came across this open issue.

[0q1627]

Getting the same error on all active connectors (add MISP to those mentioned above). OpenCTI has become unusable (docker test setup on a 16 GB RAM server). I thought this extreme load was caused by a retention policy deleting eveything older than 70 days that triggered in the same days we updated to 5.5.2, but it's apparently not the right culprit.

[Marpyre]

I've also found that Bypass all capabilities is required for most connectors to function. This is definitely a bug as I should be able to create a connector role with the appropriate least permissions instead.

[tialocRT]

The API is affected by the same problem, i can't access my observables and crawl them from the API if the account used for it does not have the "Bypass all acapbilities" right.

This is a big issue, maybe someone of the OpenCTI team could take a look to this problem. @SamuelHassine , sorry for tagging you directly but I think it's kind of a big issue.

[0xBEN]

I wanted to take a moment to follow up here. I recently upgraded OpenCTI to the latest version -- which at the time of this writing is 5.7.3.

After all of the latest Docker images were pulled and everything came online, I modified the Connector role and removed the Bypass all capabilities scope from the permissions. I then triggered a re-run on all connectors. So far, any connectors that have run, pulled data, and written data to OpenCTI have not thrown any errors.

I documented my upgrade procedure here. I would like to see if anyone else who has commented or comes across this issue to try and upgrade their OpenCTI stack and see if your connectors can run without the overly permissive scope.

[tialocRT]

Hello, I just tested the version 5.7.3 on docker with 2 connectors: MISP-FEED and MITRE. With using the default Connector group and role, I have no permission-based error. The default Connector does not have "Bypass all capabilities" right. I will try other connectors, but the first check seems good.

[GraemeMeyerGT]

@tialocRT I think this is fixed for me in 5.7. Did you confirm also?

[tialocRT]

Hello, @GraemeMeyerGT I had no problem with the default Connector role since the 5.7.X release, it seems good yeah

[GraemeMeyerGT]

@FocusedPanda propose closing this issue now unless you are still experiencing

[0xBEN]

@SamuelHassine can you please consider closing this issue in lieu of the original reporter? It appears to have been resolved quite some time ago.