CSV Mapper configuration not exposing required fields for entities with customized mandatory attributes

[akshaysth]

Description

For entity types that were customized for requiring mandatory attributes, CSV Mapper fails to display the fields appropriately in the configuration panel.

Environment

1. OS (where OpenCTI server runs): docker
2. OpenCTI version: 5.12.29
3. OpenCTI client: frontend
4. Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Customize Relationships in Settings > Customization > Entity Types and toggle Markings to be mandatory.
2. Create a CSV Mapper configuration that includes a relationship.
3. Use the Test button with a sample CSV file.

Expected Output

The CSV Mapper configuration panel should expose all attributes for each entity type. If any attributes have been customized as mandatory, there should be validation testing done to ensure the required fields are populated.

Actual Output

When testing the CSV Mapper configuration, the platform throws the below error:

{
    "errors": [
        {
            "message": "Validation error",
            "name": "VALIDATION_ERROR",
            "time_thrown": "2024-02-02T22:30:26.397Z",
            "data": {
                "http_status": 500,
                "genre": "BUSINESS",
                "field": "objectMarking",
                "message": "This attribute is mandatory",
                "attribute": "objectMarking"
            }
        }
    ],
    "data": {
        "csvMapperTest": null
    }
}

Below is the payload being sent as part of the mapper testing request:

{
  "configuration": "{\"id\":\"d1a73386-7810-4e0a-aa0e-5b415a7a15fe\",\"name\":\"CSV Mapper Bug\",\"has_header\":true,\"separator\":\",\",\"skipLineChar\":\"\",\"representations\":[{\"id\":\"8e63cc86-b906-450c-b42e-5a2eb7f8c435\",\"type\":\"entity\",\"target\":{\"entity_type\":\"Incident\"},\"attributes\":[{\"key\":\"name\",\"column\":{\"column_name\":\"A\",\"configuration\":null},\"default_values\":null,\"based_on\":null},{\"key\":\"incident_type\",\"column\":{\"column_name\":\"C\",\"configuration\":null},\"default_values\":null,\"based_on\":null},{\"key\":\"createdBy\",\"column\":null,\"default_values\":[\"a28b2f4e-d0f5-486f-884d-6409c9eceecd\"],\"based_on\":{\"representations\":[\"cc7f0e58-17cf-4f68-a096-ded7f14ed62a\"]}},{\"key\":\"objectMarking\",\"column\":null,\"default_values\":[\"user-choice\"],\"based_on\":{\"representations\":[\"1c4da4a8-81ab-4cae-a2e2-1751bb1e7618\"]}}]},{\"id\":\"8d31a091-12e7-4244-b185-ec18ad708656\",\"type\":\"entity\",\"target\":{\"entity_type\":\"Country\"},\"attributes\":[{\"key\":\"name\",\"column\":{\"column_name\":\"D\",\"configuration\":null},\"default_values\":null,\"based_on\":null}]},{\"id\":\"54c47d1d-330c-4155-b90c-f19580cdb46a\",\"type\":\"entity\",\"target\":{\"entity_type\":\"Sector\"},\"attributes\":[{\"key\":\"name\",\"column\":{\"column_name\":\"E\",\"configuration\":null},\"default_values\":null,\"based_on\":null}]},{\"id\":\"1c4da4a8-81ab-4cae-a2e2-1751bb1e7618\",\"type\":\"entity\",\"target\":{\"entity_type\":\"Marking-Definition\"},\"attributes\":[{\"key\":\"definition_type\",\"column\":{\"column_name\":\"G\",\"configuration\":null},\"default_values\":null,\"based_on\":null},{\"key\":\"definition\",\"column\":{\"column_name\":\"F\",\"configuration\":null},\"default_values\":null,\"based_on\":null},{\"key\":\"x_opencti_order\",\"column\":{\"column_name\":\"H\",\"configuration\":null},\"default_values\":null,\"based_on\":null}]},{\"id\":\"cc7f0e58-17cf-4f68-a096-ded7f14ed62a\",\"type\":\"entity\",\"target\":{\"entity_type\":\"Organization\"},\"attributes\":[{\"key\":\"name\",\"column\":{\"column_name\":\"I\",\"configuration\":null},\"default_values\":null,\"based_on\":null}]},{\"id\":\"223982d4-b347-4490-9495-5aef68f29bb5\",\"type\":\"entity\",\"target\":{\"entity_type\":\"Label\"},\"attributes\":[{\"key\":\"value\",\"column\":{\"column_name\":\"J\",\"configuration\":null},\"default_values\":null,\"based_on\":null},{\"key\":\"color\",\"column\":{\"column_name\":\"K\",\"configuration\":null},\"default_values\":null,\"based_on\":null}]},{\"id\":\"969f7313-1068-4b81-a146-3c08510fc061\",\"type\":\"entity\",\"target\":{\"entity_type\":\"Label\"},\"attributes\":[{\"key\":\"value\",\"column\":{\"column_name\":\"L\",\"configuration\":null},\"default_values\":null,\"based_on\":null},{\"key\":\"color\",\"column\":{\"column_name\":\"M\",\"configuration\":null},\"default_values\":null,\"based_on\":null}]},{\"id\":\"f94d0cda-84cd-4eaf-9044-1167f478c1f5\",\"type\":\"relationship\",\"target\":{\"entity_type\":\"targets\"},\"attributes\":[{\"key\":\"from\",\"column\":null,\"default_values\":null,\"based_on\":{\"representations\":[\"8e63cc86-b906-450c-b42e-5a2eb7f8c435\"]}},{\"key\":\"to\",\"column\":null,\"default_values\":null,\"based_on\":{\"representations\":[\"54c47d1d-330c-4155-b90c-f19580cdb46a\"]}}]},{\"id\":\"21534235-ae5f-42e6-8d14-2f617c15c221\",\"type\":\"relationship\",\"target\":{\"entity_type\":\"targets\"},\"attributes\":[{\"key\":\"from\",\"column\":null,\"default_values\":null,\"based_on\":{\"representations\":[\"8e63cc86-b906-450c-b42e-5a2eb7f8c435\"]}},{\"key\":\"to\",\"column\":null,\"default_values\":null,\"based_on\":{\"representations\":[\"54c47d1d-330c-4155-b90c-f19580cdb46a\"]}}]}]}",
  "content": "Case ID,Case Created Date,Incident Type,Country,Sector,Label,Label Colour,Label,Label Colour,Marking Definition,MD Type,MD Order,Author\r\nINC-1,2024-01-06,compromise ,United States,Transportation,,,cve-2023-20269,#D2AFFF,TLP:AMBER,TLP,3,SOC\r\nINC-2,2024-01-06,ransomware,United States,Agriculture,,,lockbit,#00FFFF,TLP:AMBER,TLP,3,SOC\r\nINC-3,2024-01-09,compromise ,United States,Communications,,,,,TLP:AMBER,TLP,3,SOC\r\nINC-4,2024-01-12,compromise,United States,Education,,,,,TLP:AMBER,TLP,3,SOC\r\nINC-5,2024-01-16,compromise,United States,Government,citrixbleed,#D2AFFF,cve-2023-4966,#D2AFFF,TLP:AMBER,TLP,3,SOC\r\nINC-6,2024-01-17,compromise,United States,Education,,,,,TLP:AMBER,TLP,3,SOC\r\nINC-7,2024-01-17,Compromise,United States,Insurance,,,,,TLP:AMBER,TLP,3,SOC\r\nINC-5,2024-01-16,compromise,United States,Government,,,citrixbleed,#D2AFFF,TLP:AMBER,TLP,3,SOC"
}

The screenshot below also illustrates that the Relationships entity that is customized to require Markings isn't exposing the attribute for configuration in the Mapper's details panel.

Additional information

Screenshots (optional)

[lndrtrbn]

Thanks you for reporting your issue.

The payload you sent is the one used when you had the Validation error that's right? It could be interesting for us to know if some of the other entities mapped in this mapper also had markings as required field?

[akshaysth]

@lndrtrbn the following entities have markings as a required field:

• Incident
• Relationships

I did further testing and was able to replicate the same behaviour on a fresh install as well. The ability to not specify markings for Relationships seems to be the root. Ideally all entities & relationships should have their respective properties all available in the CSV Mapper configuration UI with the required fields highlighted appropriately.

[akshaysth]

@lndrtrbn upon further testing, I've updated the description and title of the issue. Simply requiring the Markings in the Relationships customization is able to replicate the issue with the CSV Mapper.

[lndrtrbn]

Thank you for your feedbacks. Indeed some attributes are missing for some representations in the CSV Mapper and that's an issue if you can't map them. @Jipegien this issue is related with what we discussed about the list of attributes for each representation.

[lndrtrbn]

Related to #5838

[lndrtrbn]

@akshaysth you should be able to fill your required fields for relationships with the last release we have done

[akshaysth]

@lndrtrbn thank you! I don't see it on 5.12.31 yet. I'll have to wait for the next release that contains the fix.

[lndrtrbn]

The fix has been released in 5.12.31 so you should be able to specify markings in relationships representations in the CSV mapper. What issue do you have ?

[akshaysth]

@lndrtrbn I'm on 5.12.32 and not seeing the additional fields being displayed.

[akshaysth]

I can confirm that it's available in the rolling release. Just not the bug fix releases i.e. 5.12.31 or 5.12.32.

[SamuelHassine]

@lndrtrbn Confirmed in the master branch, also saw one other issue.

https://github.com/OpenCTI-Platform/opencti/assets/1334279/7abaa008-f1a1-43a7-9db5-3bb9cee66b20

1. Mark relationships marking as mandatory
2. Whether in entities, the mandatory marking are red if any, in relationships it is not the case.

Also, the "relation connection" cannot be mapped and need to be removed from the list. If any doubt, please talk with @richard-julien .

[lndrtrbn]

Indeed there is an other issue where required fields are not highlighted in the relationship representation. I'll create an other github issue for this one.

For the original issue. I'm not sure to understand what is confirmed 😅 You confirm that the issue is solved right?

EDIT

I can confirm that it's available in the rolling release. Just not the bug fix releases i.e. 5.12.31 or 5.12.32.

Indeed, you will see this fix in the next feature release

[richard-julien]

So we can close it?

[Kedae]

Fixed in rolling

[brianyschae]

Hi we've tried applying the commit that was potentially supposed to fix this issue: https://github.com/OpenCTI-Platform/opencti/commit/49ecb647c97951e478c30df0c3d7e454cd876a16 However, that did not solve it. Any insights if there were commits more recently that fixes this issue?