Invalid isoformat string

[NickS-2022]

In my log file i have the following error: 2023-07-26 10:59:10,525 - misp2sentinel - ERROR - Error when processing data from MISP Invalid isoformat string: '2023-02-06T09:42:59.09'

When this happens it does not process any indicators.

The above string looks like a valid date to me, but not sure what format the script is expecting them to be in.

If the script could handle the exception and dump the offending event details that would be perfect.

[cudeso]

I don't immediately have an answer for the date (will look into it later); but the latest commit should

• continue after such an error
• print out the event id where the error occurred

[NickS-2022]

Thanks for the quick turnaround, but that hasn't made any difference for my. Here's the full log:

2023-07-26 20:17:58,915 - misp2sentinel - INFO - Received MISP events page 1 with 31 events 2023-07-26 20:17:58,927 - misp2sentinel - DEBUG - Add [url:value = 'http://xxxxxxxxxxxxxxx/'] to list of indicators to upload 2023-07-26 20:17:58,930 - misp2sentinel - DEBUG - Add [url:value = 'https://xxxxxxxxxxxxxxxge/'] to list of indicators to upload 2023-07-26 20:17:58,933 - misp2sentinel - DEBUG - Add [url:value = 'https://xxxxxxxxxxxxxxx'] to list of indicators to upload 2023-07-26 20:17:58,936 - misp2sentinel - DEBUG - Add [url:value = 'https://xxxxxxxxxxxxxxx'] to list of indicators to upload 2023-07-26 20:17:58,939 - misp2sentinel - DEBUG - Add [url:value = 'https://xxxxxxxxxxxxxxx'] to list of indicators to upload 2023-07-26 20:17:58,941 - misp2sentinel - DEBUG - Add [url:value = 'https://xxxxxxxxxxxxxxx'] to list of indicators to upload 2023-07-26 20:17:58,944 - misp2sentinel - DEBUG - Add [url:value = 'https://xxxxxxxxxxxxxxx'] to list of indicators to upload 2023-07-26 20:17:58,947 - misp2sentinel - DEBUG - Add [url:value = 'https://xxxxxxxxxxxxxxx'] to list of indicators to upload 2023-07-26 20:17:58,950 - misp2sentinel - ERROR - Error when processing data from MISP Invalid isoformat string: '2023-09-11T12:58:29.72' 2023-07-26 20:17:58,950 - misp2sentinel - INFO - Received 8 indicators in MISP 2023-07-26 20:17:59,158 - misp2sentinel - INFO - Start uploading indicators 2023-07-26 20:18:00,676 - misp2sentinel - INFO - Indicators sent - request number: 0 / indicators: 8 / remaining: 0 2023-07-26 20:18:00,677 - misp2sentinel - INFO - Finished uploading indicators 2023-07-26 20:18:00,678 - misp2sentinel - INFO - End MISP2Sentinel

No matter how many days worth I try to retrieve it stops processing too early with that error.

When I tried 30 days i got a log like this:

I can see that I have pulled down the updated code as mine looks like this in script.py

try: parser = MISPtoSTIX21Parser() parser.parse_misp_event(event) stix_objects = parser.stix_objects except Exception as e: logger.error("Error when processing data in event {} from MISP {}".format(misp_event.id, e)) continue

[cudeso]

Could you check with the latest commit? I added an "unclean" way of dealing with the fromisoformat errors. In my tests it worked, but the approach should be done a little bit more cleaner.

[NickS-2022]

That still doesn't show me any more details, but I added some lines into my copy and found an example event. It's not just this one event that it is having trouble with.

The message in the log file is: 2023-07-27 09:41:28,554 - misp2sentinel - ERROR - Error when processing data from MISP Invalid isoformat string: '2023-09-11T12:58:29.72'

There is one indicator in that event and it is a URL with a date of 2023-06-13.

13th June + 90 days = 11th September which is correct.

Can't see anything wrong in the source data.

[cudeso]

That's strange; the line https://github.com/cudeso/misp2sentinel/blob/upload_indicators_api/script.py#L115 should split the datetime on . and drop everything after it (agreed, that's not a 'clean' way to deal with it).

I'll check later tonight.

[cudeso]

Related to https://github.com/cudeso/misp2sentinel/issues/56 ?

[cudeso]

@NickS-2022 can you check if the latest commit https://github.com/cudeso/misp2sentinel/commit/75acd1eeb79921778c6aa826a7826bfd037b00c4 fixes your problem?

[cudeso]

Closing pending feedback

[aliman53]

@cudeso this issue seems to still be alive

I have an event with 2 indicators - one with a last seen time of 14:37:57.330000+00:00

and another with the last seen time of 14:37:57.336000+00:00

The attribute with the last seen time of 14:37:57.330000+00:00 will give the isoformatstring error but the attribute with last seen time of 14:37:57.336000+00:00 will NOT give the error and successfully send through.

As can be seen it appears that the script does not like it when the last seen time contains a '0' in the 3rd decimal place of the microseconds.

[aliman53]

hi @cudeso

Any possible update on a solution or work around for this ?

[NickS-2022]

I'm still having issues with this. If one of the indicators has this issue then none are uploaded. The script needs to validate the data format before making the API call.

[NickS-2022]

That's strange; the line https://github.com/cudeso/misp2sentinel/blob/upload_indicators_api/script.py#L115 should split the datetime on . and drop everything after it (agreed, that's not a 'clean' way to deal with it).

I'll check later tonight.

I added some debug lines to work out what was happening and the valid_until entry looked like "2024-08-01T23:26:55.589Z", so it was never reaching the line you added at 115. As a quick fix I add the following code before line 113 to strip off anything from a dot onwards: if "." in valid_until: valid_until = valid_until.split(".")[0]

Once I did that I was able to import 3800 indicators without a problem.

[cudeso]

Hi @NickS-2022 @aliman53 ; I foresee time end of October to work on it and adjust the code

[aliman53]

@NickS-2022 that fixed the problem for me also, thanks!

[cudeso]

I'll include the fix from @NickS-2022 in an update