jorgecarleitao / arrow2

Transmute-free Rust library to work with the Arrow format
Apache License 2.0
1.07k stars 220 forks source link

Error when timestamp casting for time unit millisecond or microsecond #1541

Closed brayanjuls closed 9 months ago

brayanjuls commented 11 months ago

I am implementing the support for the following expression in polars using the cast function under the compute module of this library, but I am getting the following error when using millisecond or microsecond time units.


import polars as pl

 df = pl.DataFrame(
        {"x1": ["2021-12-19T16:39:57-02:00"]}
    ).with_columns(
        **{
            "x1-datetime-ms": pl.col("x1").cast(pl.Datetime(time_unit="ms")),
            "x1-datetime-us": pl.col("x1").cast(pl.Datetime(time_unit="us")),
        }
    )

Output:

cast failed: NotYetImplemented("Casting from Utf8 to Timestamp(Millisecond, Some(\"-02:00\")) not supported")

I would expect them to cast the date time without issues. I can provide a proposal PR to support this if you are ok with it.

Also the following date time format is not supported '2021-12-19 16:39:57', When I try to cast using it returns None. Should I open a new issue for this or could we handle it here?

brayanjuls commented 10 months ago

Already provided a PR. I Also need to support different timestamp formats cause the only one supported right now is %Y-%m-%dT%H:%M:%S%.f%:z but I guest I have to put those changes on a different PR.