[Playbooks] Error when creating a container

[NiQuintin]

Description

When creating a playbook to automate the creation of containers (such as Case-Incident), the process is displayed as successful in the platform, but is in fact unsuccessful (error visible in Grafana).

Environment

1. OCTI testing (6.0.5)

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Creating a playbook,
2. Adding a step: "Container wrapper",
3. Choose any type of container (Case-Incident, Groupings, etc.),
4. Start the playbook.

Expected Output

The playbook should create a container with the items listened to in it.

Actual Output

The playbook displays that the steps have been successfully completed. In reality, the container has not been created and the error is visible in Grafana (see below).

Additional information

Here's an example of a playbook in the testing platform.

Error displayed in Grafana:

Traceback (most recent call last): File "/opt/opencti-worker/worker.py", line 268, in data_handler self.api.stix2.import_bundle_from_json( File "/usr/local/lib/python3.12/site-packages/pycti/utils/opencti_stix2.py", line 215, in import_bundle_from_json return self.import_bundle( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/pycti/utils/opencti_stix2.py", line 2453, in import_bundle self.import_object(item, update, types) File "/usr/local/lib/python3.12/site-packages/pycti/utils/opencti_stix2.py", line 920, in import_object stix_object_results = do_import( ^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/pycti/entities/opencti_case_incident.py", line 865, in import_from_stix2 return self.create( ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/pycti/entities/opencti_case_incident.py", line 706, in create result = self.opencti.query( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/pycti/api/opencti_api_client.py", line 351, in query raise ValueError( ValueError: {'name': 'VALIDATION_ERROR', 'message': 'Validation error'}

Screenshots (optional)

[NiQuintin]

BTW in the panel dedicated to choosing container types we don't have the same names as in the user interface (like "case" in the panel which doesn't exist in the user interface).

[nino-filigran]

Happens on TESTING only.

[labo-flg]

when I try with a grouping I get another error in opencti_grouping.py, accessing non-existing key "context" for the stix object

ERROR:worker:'context'
Traceback (most recent call last):
  File "C:\Users\LaurentBonnet\IdeaProjects\opencti\opencti-worker\src\worker.py", line 268, in data_handler
    self.api.stix2.import_bundle_from_json(
  File "C:\Users\LaurentBonnet\IdeaProjects\client-python\pycti\utils\opencti_stix2.py", line 215, in import_bundle_from_json
    return self.import_bundle(
  File "C:\Users\LaurentBonnet\IdeaProjects\client-python\pycti\utils\opencti_stix2.py", line 2453, in import_bundle
    self.import_object(item, update, types)
  File "C:\Users\LaurentBonnet\IdeaProjects\client-python\pycti\utils\opencti_stix2.py", line 920, in import_object
    stix_object_results = do_import(
  File "C:\Users\LaurentBonnet\IdeaProjects\client-python\pycti\entities\opencti_grouping.py", line 807, in import_from_stix2
    context=stix_object["context"],
KeyError: 'context'

[labo-flg]

@nino-filigran to be clear on the subject: when a playbook sends for ingestion and this steps is reported OK, it does not mean that the objects are created successfully. It means that they are sent successfully to the worker queue.

This difference is important, if something bad happens on the worker side (like we have in this case, because there is a bug) then we are not aware of it.

To be able to see errors when consuming the playbook ingestion queue, we would need a dedicated feature (a bit like the Tasks list).

[nino-filigran]

Ok so if I undertstand there is a bug at the processing level (this bug), and we need a feautre to display any error that would happen during the processing. I've created it here: https://github.com/OpenCTI-Platform/opencti/issues/6668

[labo-flg]

Thanks @nino-filigran ! Yes this feature would help in debugging such issue. As for the initial issue here, I've done some digging.

Right now, Container Wrapper works only if you select a container type that requires only a name (like a report). If there is other required field, such as the 'context' for a grouping, worker will fail when trying to ingest the bundle created within the playbook, because of the missing fields.

Ultimately, I think we want to have a dynamic form in the Container Wrapper depending on the selected container type. For instance if you select Grouping, then it shows a selector for the grouping context. This is significant work.

If we want quick fix, here are some ideas:

Option 1 Add required field, generating values if needed, directly in the playbook manager backend side. This is already the case for labels, markings, assignees, participants, created_by_ref. If these fields exist in the base object being listened to, they are copied to the container. This is quite hard for some types of cotnainer... Grouping context can be unspecified by default for instance, but what about the content of a note, or the value of an Opinion ? Observed-Data need observation dates, number of observations...

Option 2 Limit the available container types in the list, right now we're only compatible with Report, Incident Response, RFI, RFT, Feedback, Tasks.

... and of course a mix of the 2 options.

[nino-filigran]

@labo-flg thanks for your answer. Indeed, your idea of having a selection of the value for the mandatory fields seems a good solution, if we want to support all types of containers on the long term.

Just to clarify, these are the fields that are needed to create an entity (at least in UI):

notes	observed data	opinion	report	grouping	case incident	RFI	RFT	Feedback	Task
publication date, content	entity, first seen, last seen	value of the opinions defined in an open vocab	name, publication date	name, context (open vocab), number_observed (set to 1 by default)	name	name	name	none (not even an entity)	name (but must be linked to a case)

However, currently, if I have followed you, even though we propose all these containers, we only support Report, Incident Response, RFI, RFT, Feedback, Tasks. Therefore only Observed Data, Opinions, Grouping, Notes do not work at the moment.

As a result, your solution 2 seems the more appropriate if we want to remain coherent with the current behavior: we simply need to remove the following entities from the list: Observed Data, Opinions, Grouping, Notes, Cases.

The easiest to add would be grouping, since we could pick the first value that we found (given that explicity mentionning a value could be dangerous if it gets deleted). Same applies for opinions. But this could be misleading for users & we have not received any feedback so far that creatng a grouping from automation does not work.

@Jipegien and @richard-julien please let us know if you disagree with this.

[labo-flg]

Thanks for the recap ! Quick comment: observed data also needs a number_observed (when you create from UI it's sent as 1) Also, the list mentions "Case" but it cannot be created at all in opencti (it's a parent type of Incident Response for instance).

[nino-filigran]

Thanks I've updated it. Yes, I saw "case" in the list and did not bother to mention it, since it seemed obvious to me. But just for sake of clarity, I'll also update my comment to have it 100% clear and written.

[labo-flg]

After discussing with @richard-julien, we will definitely go for option 2. I'll remove all incompatible types.

The easiest to add would be grouping, since we could pick the first value that we found (given that explicity mentionning a value could be dangerous if it gets deleted). Same applies for opinions. But this could be misleading for users & we have not received any feedback so far that creatng a grouping from automation does not work.

Agreed, let's remove grouping.