Open eliataylor opened 2 years ago
Hello @eliataylor,
Can you please try to put a confidence level to your observed data greater or equel to 15 because this is the default one in the platform?
If your confidence level is greater or equal than the existing one (so you can put always the same), the entity will be updated. Update=true is now used only in some merging circumstances.
Kind regards, Samuel
@SamuelHassine , I wish i could say that worked. I tried all variations of
fields = {}
fields["stix_id"] = observed["standard_id"]
fields["id"] = observed["id"]
fields['number_observed'] = observed["number_observed"] + 1
# fields["createdBy"] = observed["createdBy"]
# fields["objects"] = observed["objectsIds"]
fields["objects"] = observed["objects"]
fields["confidence"] = 100
fields["last_observed"] = observed["last_observed"]
fields["first_observed"] = observed["first_observed"]
# fields["objectMarking"] = observed["objectMarkingIds"]
# fields["objectLabel"] = observed["objectLabelIds"]
# fields['update'] = True
# self.helper.api.observed_data.update_field(**fields)
self.helper.api.observed_data.create(**fields)
Ok @eliataylor, you're right, we have a special portion of the API code that prevent this update if the first_observed / last_observed are the same. We will fix this asap. In the mean time:
If you change (a little) the first_observed or the last_observed, you will enter in this condition:
const timePatch = handleRelationTimeUpdate(updatePatch, element, 'first_observed', 'last_observed');
// Upsert the count only if a time patch is applied.
if (isNotEmptyField(timePatch)) {
const basePatch = { number_observed: element.number_observed + updatePatch.number_observed };
const patch = { ...basePatch, ...timePatch };
const patched = await patchAttributeRaw(element, patch, { upsert: true });
impactedInputs.push(...patched.impactedInputs);
patchInputs.push(...patched.updatedInputs);
}
So this means the number_observed will be updated with "existing number_observed" + "passed number_observed". In your case, if you would like to increment, just pass "1".
Kind regards, Samuel
@SamuelHassine , unfortunately it's still not budging even when incrementing last_observed
by 1 second each run.
fields = {}
fields["stix_id"] = observed["standard_id"]
fields["id"] = observed["id"]
fields['number_observed'] = 1
# fields['number_observed'] = observed["number_observed"] + 1
fields["objects"] = observed["objects"]
fields["confidence"] = 100
fields["first_observed"] = observed["first_observed"]
date = datetime.strptime(observed["last_observed"], self.dateFormat)
date += timedelta(seconds=1)
fields["last_observed"] = datetime.strftime(date, self.dateFormat)
fields['update'] = True
self.helper.api.observed_data.create(**fields)
However, this idea does match up with the fact my runtime Connector is setting greater than 1. Of course, I would like to ingest out of chronological order so those dates may not change despite the number_observed.
We have another customer where this is working, can you display the "fields" at each run and verify the last_observed is actually incremented?
on 5.3.10?
Yes this should work since a while :)
@SamuelHassine , I've verified through breakpoints neither last_observed
nor number_observed
are updated, despite sending the updated fields object.
Both runs show:
The only difference in my queried object and update fields is '2022-08-05T20:16:34.000000Z'
vs '2022-08-05T20:16:34.000Z'
in my code above I'm using self.dateFormat = "%Y-%m-%dT%H:%M:%S.%fZ"
@SamuelHassine , i now see i'm getting this error:
{'name': 'Variable "$input" got invalid value { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null } at "input.objects[0]"; String cannot represent a non string value: { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null }', 'message': 'Variable "$input" got invalid value { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null } at "input.objects[0]"; String cannot represent a non string value: { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null }'}
ERROR:root:Variable "$input" got invalid value { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null } at "input.objects[0]"; String cannot represent a non string value: { id: "3936c937-8315-49e3-9532-efbed95c8d07", entity_type: "Url", parent_types: ["Basic-Object", "Stix-Object", "Stix-Core-Object", "Stix-Cyber-Observable"], standard_id: "url--e9d97d97-6e83-501a-bed5-91786286d4f9", spec_version: "2.1", created_at: "2022-09-13T17:55:58.610Z", updated_at: "2022-09-13T17:55:58.635Z", createdById: null }
obviously observed_data.create(**fields)
shouldn't be touching the contained objects when only updating the Observed Data fields.
@SamuelHassine , resolved. in the end. confidence
and update
didn't matter. I do need to increment the last_observed
by a millisecond and only pass number_observed=1
to increment it's value. My issue otherwise was that objects
needs to just be an array of Ids, not the whole stix objects.
fields = {}
fields["stix_id"] = observed["standard_id"]
fields['number_observed'] = 1
# fields['number_observed'] = observed["number_observed"] + 1 #this seems more appropriate.
fields["objects"] = observed["objectsIds"]
fields["first_observed"] = observed["first_observed"]
date = datetime.strptime(observed["last_observed"], self.dateFormat)
date += timedelta(milliseconds=1)
fields["last_observed"] = datetime.strftime(date, self.dateFormat)
# fields["confidence"] = observed["confidence"]
# fields['update'] = True
self.helper.api.observed_data.create(**fields)
Not sure if you want to close this given the necessary patches on these upsert conditions.
Description
Our connect needs to increment the
number_observed
property. From the active connector, I am getting some number_observed greater than 1, but it's not clear the exact context of those updates. Usingpycti
I've tried many combinations of fields but never getnumber_observed
to increment.Environment
pycti
Reproducible Steps
Expected Output
The Observed Data's
number_observed
property increments by 1 each run.Actual Output
No change.
Additional information
I found this related Issue - https://github.com/OpenCTI-Platform/opencti/issues/1444 - but the fix is already applied to 5.3.10
Screenshots (optional)