search

Deltares / Ribasim

Water resources modeling
https://ribasim.org/
MIT License
42 stars 5 forks source link

Support Arrow input tables #170

Closed visr closed 12 months ago

visr commented 1 year ago

It should work already in the core, but since there are no tests let's assume it's not working. Only the output tables are written to Arrow.

The example models in conftest.py currently write all input tables to the GeoPackage file. This is what model.write(path) does. Usually Arrow input tables are mainly interesting for large (time-dependent) tables, but we need to make sure it is working in CI.

I see the class InputMixin has _write_arrow and _write_geopackage that are both not used. Instead write loops over the underlying tables and writes them one-by-one to GeoPackage. Perhaps we need something like an attribute in all tables (e.g. classes that inherit from pa.SchemaModel, that determines if it should be written to GeoPackage (the default) or Arrow.

A minimum first test is added here, which contains a utility function to convert GeoPackage tables to Arrow. https://github.com/Deltares/Ribasim/pull/187/files#diff-dd42657be5ede608d919dd58408933bac74a4661098f89d8b48dc53adf0478d7

evetion commented 1 year ago

Will add/update a testmodel that writes an arrow file in #731, which will implicitly test it in Julia as well.

visr commented 1 year ago

Ok, nice. It'd say fixing the Julia side of things is out of scope of #731 though, it's already large enough.