Support RFC3339 Timestamps with Nano precision and Timezones (ie: 2023-11-28T01:55:24.811939-07:00 )

[veqryn]

What?

I attempted to set the field timestamp equal to 2023-11-28T01:55:24.811939-07:00, but got the following error in my log line: Replaced invalid timestamp value in message <e9f180a0-8dcb-11ee-9f35-0242ac160004> with current time - Value <2023-11-28T01:55:24.811939-07:00> caused exception: Invalid format: "2023-11-28T01:55:24.811939-07:00" is malformed at "T01:55:24.811939-07:00".

Why?

Not accepting timezones isn't great. Not accepting more precision isn't great. Not accepting the strict rfc-3339 format isn't great. It would be great if graylog could accept precision down to 9 decimal places, but even just 6 would be amazing. Sometimes logs are streaming very fast, and we have multiple with the same first 3 digits (which is what graylog currently accepts).

Examples are valid timestamps: 2023-11-28T01:55:24Z 2023-11-28T01:55:24-07:00 2023-11-28T01:55:24.811939Z 2023-11-28T01:55:24.811939-07:00 2023-11-28T01:55:24.811939123Z 2023-11-28T01:55:24.811939123-07:00

Your Environment

• Graylog Version: 5.2
• Datanode Version: 5.2
• MongoDB Version: 5.0
• Operating System: Docker, on Mac
• Browser version: Chrome

[kroepke]

Adding nanosecond support is pretty difficult and involved, but timezones should work, we'll investigate. If you have test cases available that would be helpful.

[veqryn]

The only test cases I have on hand are the examples I gave of RFC-3339 timestamps, such as:

2023-11-28T01:55:24Z 2023-11-28T01:55:24-07:00 2023-11-28T01:55:24.811939Z 2023-11-28T01:55:24.811939-07:00 2023-11-28T01:55:24.811939123Z 2023-11-28T01:55:24.811939123-07:00

If nanoseconds can't be supported, it would be great if Graylog could accept the above input, and translate it into what it can support, without erroring or forcing the creation of pipelines/extractors. RFC-3339 is very common and very strict.