search

MISP / PyMISP

Python library using the MISP Rest API
Other
445 stars 280 forks source link

Bug - Exported event schema validation fails #1124

Open imranmmd17 opened 10 months ago

imranmmd17 commented 10 months ago

PyMISP exported event fails schema validation.

event.to_json() returned event is when passed through the standard MISP schema (https://github.com/MISP/MISP/blob/2.4/format/2.4/schema.json), the schema validation fails for:

  1. publish_timestamp - is expected to be str
  2. tags - Event and Attribute tag's local attribute type is expected to be int
imranmmd17 commented 10 months ago

@Rafiot - Can you please look into it. Thanks.

Rafiot commented 10 months ago

The events generated by PyMISP work just fine on MISP, so it is probably more an issue with the schema itself.

Rafiot commented 10 months ago

ok, update on that: PyMISP to_json returns a blob that is compatible with what MISP expects (float or int) when the schema says string.

We could consider to add a method that casts the int/float to str, but the default will stay the same for backward compatibility reasons. What is your use-case?

imranmmd17 commented 10 months ago

Thanks @Rafiot. Our requirement is to export the MISP event and ensure it passes the MISP standard schema before passing it on to the data pipeline for subsequent processing. At every stage of the pipeline, the very thing they do is schema validation again.

Rafiot commented 10 months ago

Alright, I started to look at that:

  1. will require more detailed investigation as it is converted to an integer at a place that makes it somewhat hard to change. nevermind, fixed: https://github.com/MISP/PyMISP/commit/1d24c1b3cee00ac887dba1a77be81aaaaceed2b2
  2. the local key in tag isn't in the schema at all, can you give more details on how you got the warning, please?
imranmmd17 commented 10 months ago
  1. Thanks @Rafiot for fixing the timestamp. LGTM.
  2. When an event is downloaded using MISP WebUI, the tag's local attribute is number (i.e. int). Whilst the API event.to_json() returns it as str. Yes the local key isn't in the schema, however we've added it for our use case to match with the WebUI downloaded event. We expect the outcome to be the same whether an event is d/l using WebUI or using API. It's not a show stopper, but good to have consistency.
dragsu commented 8 months ago

@Rafiot I want to extend this. What is the reason for the blanket _int_to_str(to_return) call inside to_dict method in AbstractMISP.py? As a result, Tag's local values (0 and 1) get changed into string values ("0" and "1") , which is unexpected.

P.S I have added a fix for this https://github.com/MISP/PyMISP/pull/1171

Rafiot commented 8 months ago

It is a bug on MISP side, that should be handled as a boolean, and not 0/1. It will be fixed in MISP asap.

iglocska commented 8 months ago

FYI, the fix is now on develop and will go into the release later today.

dragsu commented 8 months ago

MISP v2.4.187 got an upstream fix for this. Now local values are represented as true and false.