[diode-import] Question about creator and update of existing data

[PeeBee66]

Prerequisites

• [X] I read the Deployment and Setup section of the OpenCTI documentation as well as the Troubleshooting page and didn't find anything relevant to my problem.
• [X] I went through old GitHub issues and couldn't find anything relevant.
• [X] I googled the issue and didn't find anything relevant.

Description

Background: I have each feed pulling in data as per the below connector example for MITRE and it save's the output to the .JSON files. These are then pushed to the airgapped area.

  connector-mitre:
    image: opencti/connector-mitre:${OPENCTI_VERSION}
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_MITRE_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_MITRE_ID}
      - "CONNECTOR_NAME=MITRE Datasets"
      - CONNECTOR_SCOPE=tool,report,malware,identity,campaign,intrusion-set,attack-pattern,course-of-action,x-mitre-data-source,x-mitre-data-component,x-mitre-matrix,x-mitre-tactic,x-mitre-collection
      - CONNECTOR_CONFIDENCE_LEVEL=75
      - CONNECTOR_UPDATE_EXISTING_DATA=false
      - CONNECTOR_RUN_AND_TERMINATE=false
      - CONNECTOR_LOG_LEVEL=info
      - MITRE_REMOVE_STATEMENT_MARKING=true
      - MITRE_INTERVAL=2
      - CONNECTOR_SEND_TO_QUEUE=true
      - CONNECTOR_SEND_TO_DIRECTORY=true
      - CONNECTOR_SEND_TO_DIRECTORY_PATH=/data
      - CONNECTOR_SEND_TO_DIRECTORY_RETENTION=1
    restart: always
    volumes:
      - type: bind
        source: "/mnt/connector/8.Test-Backup/c.mitre"
        target: /data

Each saved folder for each feed will have its own Diode connector. This is for two reasons:

1. I am able to see the creator of the feed and know where the data comes from as the author is sometimes wrong or blank. See example image

2. To control the data ingest flow if needed or refeed data etc etc

Question 1: As per the attached photo, I am wondering why the creator is admin when I have set my user token on the diode and also attempted to map the DIODE_IMPORT_APPLICANT_MAPPINGS. Ultimately, setting the user OPENCTI_TOKEN=${OPENCTI_MITRE_USER_TOKEN} should ingest it as that user, shouldn't it?

  connector-diode-import-mitre:
    image: opencti/connector-diode-import:${OPENCTI_VERSION}
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_MITRE_USER_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_MITRE_ID}
      - CONNECTOR_NAME=Diode Import
      - CONNECTOR_RUN_AND_TERMINATE=false
      - CONNECTOR_LOG_LEVEL=info
      - DIODE_IMPORT_APPLICANT_MAPPINGS=${CONNECTOR_MITRE_ID_LOW}:${CONNECTOR_MITRE_ID}
      - DIODE_IMPORT_GET_FROM_DIRECTORY_PATH=/data
      - DIODE_IMPORT_GET_FROM_DIRECTORY_RETENTION=7
    restart: always
    volumes:
      - type: bind
        source: "/mnt/connector/8.Test-Backup/c.mitre_IN"
        target: /data

Question 2: Does the diode connector support the ability to CONNECTOR_UPDATE_EXISTING_DATA?

Question 3 Once the data has been ingested into OpenCTI, does the diode connector recognise the .JSON files it has already processed and avoid re-ingesting them again? Or do I need to restart the connector and remove the old .JSON files, then add new or updated .JSON files for re-ingestion? Additionally, will the diode connector need to be restarted in order to find new files?

Environment OS - Ubuntu 22 and Red Hat 8.9 OpenCTI version: 6.1.4 OpenCTI client: frontend Other environment details: N/A Reproducible Steps N/A

[PeeBee66]

I have been testing the diode on the air-gapped system to ensure the .JSON file captures all the data collected from the public system. So far, MITRE is spot on, and further feeds will be tested as I get through them.

Following up on the CREATORS issue, I'm still getting "admin" as the creator.

As per the image for my example with CVE, I have come across two different sets of examples and have been trying to test the ability to change the creator to a user in the system?

The following is from the air-gapped article: https://blog.filigran.io/opencti-6-0-10-in-air-gap-diode-environments-61c58b33559c

The following is from the GitHub page: https://github.com/OpenCTI-Platform/connectors/tree/master/external-import/diode-import

After testing multiple UUIDs from the user's ID token within the config, I finally came across an applicant ID as stated in the GitHub within the .JSON file (see image). This applicant ID was not visible anywhere else, so I am presuming it is generated or hard-coded in the container. However, the creator still appears as "admin, even if I link that APP_ID to a User_ID"

Any guidance or suggestions on resolving these issues would be greatly appreciated. Thank you!