Open LFED-FP opened 3 years ago
Just making sure, you're incrementing the page
parameter in every loop. right?
Something like:
i = 1
while True:
old_events = client._search(controller='events', tags=old_tag_name, limit=500, page=i, date_to=now, published=True)
if len(old_events) < 500:
break
i += 1
If yes, the issue you opened on the MISP project (https://github.com/MISP/MISP/issues/6887) is where you will get an answer.
Yes I am incrementing the page variable :-)
Woops, sorry, I forgot a follow up on this issue. Do you still have the problem?
Work environment
In Dockerfile
In MISP ui
Expected behavior
I have purpose written code which locates events with specific tags, creates new tags, tags all found events with new tags, republishes those events, & then deletes the old tags. Essentially I am updating tags on events & removing old tags.
Actual behavior
Some tags have over 9,000+ events associated with them. When I encounter such tags, the code goes about its normal execution. As the code attempts to search for all the events for a particular tag with previously statement total, I get this error message
{"asctime": "2021-01-19 21:46:07,358", "timestamp": 1611092767.394494, "name": "misp-feeds", "filename": "search_and_destroy.py", "funcName": "_search", "lineno": 228, "level": "ERROR", "levelno": 40, "message": "Searching for events failed. kwargs: {'controller': 'events', 'tags': 'malware:GandCrab', 'limit': 500, 'date_to': 1611092482, 'published': True, 'page': 4} Reason: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))", "severity": "ERROR"}
I tried "streaming" events by limiting how many events are returned by MISP in a while loop, cycling through until there are no events returned
old_events = client._search(controller='events', tags=old_tag_name, limit=500, date_to=now, published=True)
However, I get the same error. I had a tag with 9,015 event. My code processed exactly 1,500 events and then refuses to continue even after I stop the script and run it again.Steps to reproduce the behavior
I looked through other issues related to this and found a similar issue
https://github.com/MISP/MISP/issues/6805
. Im not exactly sure how to reproduce this.Logs, screenshots, configuration dump, ...