Open sergiimk opened 10 months ago
This can be easily implemented now that datafusion supported chrono format specifiers e.g.:
select to_timestamp('2020/03/30 12:00:00+00', '%Y/%m/%d %H:%M:%S%#z') as t;
@sergiimk , is it a ticket to add a "custom" SQL function that will call our native code?
very interesting direction!
Currently the
timestampFormat
is not supported in our datafusion-based ingest readers.Datafusion and arrow timestamp support is currently lacking and only can parse
rfc3339
strings.Because of this, some datasets like
covid/quebec.case-details
have to rely on Spark to parse complex time formats.This ticket is to investigate the support for advanced timestamp parsing.