artificialwisdomai / origin

Artificial Wisdomâ„¢ Cloud Platform
Apache License 2.0
3 stars 4 forks source link

wisdomctl: Use Apache Feather format in OCI buckets #118

Closed MostAwesomeDude closed 10 months ago

MostAwesomeDude commented 10 months ago

Fixes #108.

As requested, ask for zstd compression.

Poetry/poetry2nix pulls a wheel with incorrect linkage for me (wrong C++ libraries?), so I've enumerated the actual dependencies in the Nix flake. This means that I don't get to share exactly the same Python package versions between the flake and poetry.lock, but I'm willing to make that sacrifice in order to actually test my changes!

Note that there is a slight asymmetry in how JSON Lines is converted to Feather; Apache Arrow was not designed to have full interchange with JSON Lines, but we can ask Pandas to make up the difference.