kujaku11 / mth5

Exchangeable and archivable format for magnetotelluric time series to better serve the community through FAIR principles.
https://mth5.readthedocs.io/en/latest/index.html
MIT License
16 stars 4 forks source link

ValueError: time data does not match format 'mixed' (match) #197

Closed kkappler closed 1 month ago

kkappler commented 6 months ago

A recent change was made regarding time formats, adding the keyword argument format="mixed" to channel_table.py.

While this may have fixed the specific instance of data, aurora is failing a lot of tests with messages like:

ValueError: time data '1980-01-01T00:00:00+00:00' does not match format 'mixed' (match)

Another tool that uses mth5 to build datasets is hitting the same issue.

I will try wrapping this in a try/except so it uses both the old and the new command on patches branch and see if that fixes it.

image

kkappler commented 6 months ago

@kujaku11 when we merge this in, we may want to invert the try/except from

        try:
            df.start = pd.to_datetime(df.start.str.decode("utf-8"), format="mixed")
            df.end = pd.to_datetime(df.end.str.decode("utf-8"), format="mixed")
        except ValueError:
            df.start = pd.to_datetime(df.start.str.decode("utf-8"))
            df.end = pd.to_datetime(df.end.str.decode("utf-8"))

to

        try:
            df.start = pd.to_datetime(df.start.str.decode("utf-8"))
            df.end = pd.to_datetime(df.end.str.decode("utf-8"))
        except UNKNOWN_EXCEPTION:
            df.start = pd.to_datetime(df.start.str.decode("utf-8"), format="mixed")
            df.end = pd.to_datetime(df.end.str.decode("utf-8"), format="mixed")

but I do not know what the Exception was that you encountered, nor do I have an example dataset with the error.

Since we ran fine for years the "mixed" option, then probably most data will not need it, so it might be slightly faster to invert. That said, these timestamps are in the channel_summary table, which is never very big so maybe it doesn't matter.