gchq / stroom

Stroom is a highly scalable data storage, processing and analysis platform.
https://gchq.github.io/stroom-docs/
Apache License 2.0
424 stars 58 forks source link

EffectiveTime header variable/stream attribute should allow for missing milli seconds #4131

Closed burnalting closed 4 months ago

burnalting commented 4 months ago

Currently, the EffectiveTime stream attribute, gained from a posted header variable or a UI entry box, must be of the form YYYY-MM-DDThh:mm:ss.SSSZ.

We should allow for YYYY-MM-DDThh:mm:ssZ as most non-stroom initiated users don't think about milli seconds in a timestamp.

at055612 commented 4 months ago

In principle yes this makes a lot of sense, as would accepting times with an offset different to UTC. However, if we allow different data formats to creep into stroom then we have to be very sure that all the code and content and cope with those formats. E.g. anything that reads that date from the meta needs to be able to parse it.

at055612 commented 4 months ago

The code change will allow the internal date parsing function (as used for parsing datatime header values) to be more lenient with ISO 8601 datetimes. They still need to be zoned however. The following are examples of data time strings that it will now handle.

"2010-01-01T23:59:59.1Z"
"2010-01-01T23:59:59.12Z"
"2010-01-01T23:59:59.123Z"
"2010-01-01T23:59:59.1234Z"
"2010-01-01T23:59:59.12345Z"
"2010-01-01T23:59:59.123456Z"
"2010-01-01T23:59:59.000123Z"
"2010-01-01T23:59:59.0Z"
"2010-01-01T23:59:59.00Z"
"2010-01-01T23:59:59.000Z"
"2010-01-01T23:59Z"
"2010-01-01T23:59:59Z"
"2010-01-01T23:59:59+02:00"
"2010-01-01T23:59:59.123+02"
"2010-01-01T23:59:59.123+00:00"
"2010-01-01T23:59:59.123+0200"
"2010-01-01T23:59:59.123+02:00"
"2010-01-01T23:59:59.123-03:00"
"2010-01-01T23:59:59.123+02:00:00" 

When it comes to handling and storing header values for keys that are known to be a datetime, it will normalise the datetime string to our standard form yyyy-MM-dd'T'HH:mm:ss.SSSXX so that meta values are consistent in appearance.