search

quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
8.34k stars 340 forks source link

Support specifying a timezone manually with strptime parser #3171

Open guilload opened 1 year ago

guilload commented 1 year ago

For instance:

name: timestamp
type: datetime
description: Time at which the event was emitted
input_formats:
  - rfc3339
  - unix_timestamp
  - "%Y %m %d %H:%M:%S.%f %z"
  - strptime:
      format: "%Y %m %d %H:%M:%S.%f"
      default_timezone: America/New_York
fulmicoton commented 1 year ago

Maybe default_timezone?

guilload commented 1 year ago

default_timezone sounds better, indeed.

rdettai commented 4 months ago

I think we should raise an error during the parsing of the config if the user provides a format with a timezone and a default timezone (%z or %Z). It's better to fail early. Any opinion @fulmicoton ?

rdettai commented 4 months ago

Parsing a timezone like America/New_York and use it to compute a timestamp from a date string requires a TZ database. The most popular / better maintained one seems to be chrono-tz, but we use time almost everywhere in Quickwit (and Tantivy). There is a time-tz crate that would fit more nicely into the existing code, but it appears to be substantially less used.

fulmicoton commented 3 months ago

In #5248 I'm rewriting a tiny bit the code that parse dates. It will be possible to pass an utc offset which might simplify this issue. Getting the offset from the timezone will require a bit of work/wits however.

guilload commented 3 months ago

You may also have a look at https://github.com/quickwit-oss/quickwit/tree/guilload/strptime-timezone