lambda es-loader datetime.fromisoformat does not work with most ISO strings

[PascalArevalo]

Issue

I am trying to ingest a non-AWS service log as mentioned in the documentation. The log contains an ISO 8601 timestamp such as this one: 2023-10-06T22:03:35.2807476Z. Unfortunately, the ingestion of such logs has not been successful.

I tried to debug the lambda es-loader and I have found that Python returns ValueError: Invalid isoformat string: 2023-10-06T22:03:35.2807476Z

In the Python documentation it specifies that this method only works for certain specific strings up to Python 3.11

Changed in version 3.11: Previously, this method only supported formats that could be emitted by date.isoformat() or datetime.isoformat().

Furthermore, I have tried to use a custom timestamp_format but as before, it doesn't work. I used the following timestamp format: %Y-%m-%dT%H:%M:%S.%f%z. Unfortunately, Python datetime.strptime function is only capable of parsing 6 microseconds digits in a string. Trying to parse the datetime I already mentioned just result in another ValueError: time data does not match format.

Possible Solution

I have created a Pull Request that fixes the ISO 8601 issue that I mentioned using the dateutil-python PyPI package: https://github.com/aws-samples/siem-on-amazon-opensearch-service/pull/421

[PascalArevalo]

P.s. I forgot to mention that your work is incredible!😄 I really like it

[nakajiak]

Thanks for the feedback and PR. We will investigate it.

[nakajiak]

Hi, when you use nano seconds, please use timestamp_nano = True. https://github.com/aws-samples/siem-on-amazon-opensearch-service/blob/v2.10.2/source/lambda/es_loader/aws.ini#L205

[nakajiak]

Thanks so much