search

Deltares / Ribasim

Water resources modeling
https://ribasim.org/
MIT License
39 stars 5 forks source link

Use pyarrow dtype_backend #1781

Closed visr closed 2 days ago

visr commented 4 weeks ago

Fixes #1721 Fixes #1394

The dtype_backend part of this is not breaking. The only part that is technically breaking is that we specify a unit of milliseconds for the Arrow time type. Previously we used the default nanosecond precision, which was then truncated to milliseconds in the core. I think it is better to disallow precision higher than milliseconds if we cannot distinguish them in the core.

evetion commented 4 days ago

I'd like to see some extra tests with adding/setting a default pandas dataframe, without the correct dtypte and time. edit: Basically trying to break it ;)

DanielTollenaar commented 4 days ago

Tested it on De Dommel workflow and get a model that gets written with the Python-API and read and simulated without ValidationErrors or Exceptions. Didn't test/check any results, but the model + results are here: https://we.tl/t-Uxk9xra8i1

However, I do repeatedly get this FutureWarning which is messing up my logging quite alot:

(file:///D:/repositories/Ribasim/python/ribasim/ribasim/model.py:239): FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  pd.concat(df_chunks)