Event Retrieval With PyMISP.search Returns Empty List

[kevin3567]

Hi,

I am having an issue with the retrieving events with PyMISP. When I use the search() function from PyMISP class with the _dateto argument, the returned list of events is always empty even though the queried events (those created before _dateto) are present on MISP. Specifically, here is my implmentation:

misp = ExpandedPyMISP(misp_url, misp_key, misp_verifycert) results_a = misp.search(date_to=datetime.now()+timedelta(days=1))

Theoretically, all events present in MISP should be returned, as all events are guranteed to be created before the _dateto time. However, on execution, _resultsa is [].

Is this a potential bug? Alternatively, am I doing something wrong with this search?

[Rafiot]

Out of curiosity, have you tried to use date_from too? I wouldn't be surprised MISP acts weirdly if you only pas one of the two.

Also, if you want to get a complete MISP database, you definitely also want to use limit and page so MISP doesn't returns thousands of entries and run out of memory.

If you really want to get all the events from your MISP instance, you can also use the events method.

[kevin3567]

Hi,

Yes I have also tried _datefrom, which seems to work. I have also tried using both _datefrom and _dateto, which returned no events. Thus, I suspect that _dateto is the issue. I have also tried limit, although that does not appear to help.

Ultimately, the objective is for my program to retrieve all events created (not updated with additional attributes, just created) between time X and Y. So, it is necessary to have the _dateto argument working.

[Rafiot]

I just checked, and we have a test case for that (date_from only and date_from + date_to): https://github.com/MISP/PyMISP/blob/main/tests/testlive_comprehensive.py#L819 And it works as expected so I'm not sure why it's not working for you.

But anyway, I'm not sure you can do that: date_from and date_to are set by the user, and it can be anything: it is possible to create an event today and have a date field set to months ago. And afaik, there is no way to search for the creation timestamp (please tell me if I'm wrong @iglocska @mokaddem).

Maybe you want to use the publish timestamp instead? If an event is re-published, it will come back in your list, but it may be better than nothing?

[kevin3567]

I think I have found the issue, the in my previous code, I was passing a Datetime object, not a Date object. Once I replaced that with a string (2021-06-17), it seems to work.

I do have some follow up questions though:

1. I am wondering if I can query for events by time, rather than by date. For example, could I retrieve all the events between 6:30 AM to 9:30 AM on the 2021-06-16?
2. It appears that _datefrom and _dateto permits the input of int and float. Are these used to enter Unix time (in seconds)?
3. Is there a way to query for events by generation time, rather than last update time. I think this question is already answered. But, just in case you have thought of something else, please let me know.

Thanks

[Rafiot]

1. no, the date field in the event doesn't have more precision than the day
2. you can pass a Unix time in seconds, but it will converted as a date
3. afaik, creation time isn't kept, but my colleagues might have an other idea

[JoePJisc]

It appears PyMISP doesn't support datetime objects for this field as the documentation suggest.

0 Results

UnpublishTo = datetime.now() - timedelta(days=365*3)
MISP = PyMISP(URL, AuthKey)
OldEvents = MISP.search(date_to=UnpublishTo)

Expected results

UnpublishTo = datetime.now() - timedelta(days=365*3)
UnpublishToStr = UnpublishTo.strftime("%F")
MISP = PyMISP(URL, AuthKey)
OldEvents = MISP.search(date_to=UnpublishToStr)

[Rafiot]

try with date_from, instead of date_to?

[JoePJisc]

date_from is not useful for my requirements unfortunately, however the .strftime("%F") solved the issue for me.