search

aljazerzen / connector_arrow

Apache Arrow database client for many databases.
https://docs.rs/connector_arrow
MIT License
37 stars 2 forks source link

Stale dependency on arrow-rs, because it also depends on duckdb-rs #32

Open tripokey opened 2 months ago

tripokey commented 2 months ago

The latest release of this crate uses arrow 49, as of writing the latest arrow release is at version 52. Could we get a new release with bumped arrow?

aljazerzen commented 2 months ago

We cannot update to 52, we are tied to the version of arrow that duckdb-rs uses.

That's because this crate (actually only some of its features) depend on duckdb-rs, which currently depends on arrow 51. The code in this crate assumes that "our" arrow versions matches with the duckdb's version.

This problem is made much much worse by the versioning cadence of arrow-rs. Read more about that here.

Maybe it would be possible to have two versions of arrow in the project and then convert duckdb arrow objects into "our" objects?

tripokey commented 2 months ago

Maybe a release using arrow 51 would be possible?

Two arrow versions would increase compile times as well as the final binary size.

aljazerzen commented 2 months ago

Oh yes, release - that's easy :D

Two arrow versions would increase compile times as well as the final binary size.

Yes, probably. Although that would only duckdb connector, while current state is holding back all and each connectors.

The real solution is for arrow-rs to stop publishing major versions of crates without breaking changes. But that requires a bit of coordination and increased release management on their side.

aljazerzen commented 2 months ago

Even though a new version of connector_arrow is now released, let's keep this issue open because arrow versioning is a problem for this crate.

aljazerzen commented 2 months ago

I now also realize that updating arrow version in connector_arrow is a breaking change, as it is exposed in the public API. Which also means that the crazy release cadence of arrow-rs is also forced on connector_arrow.

Fortunately, this crate has not yet published 1.0 version, so having 0.15 in a few years will not be as damming as arrow-rs's 51.0 :D

tripokey commented 1 month ago

Could you make a new release now that arrow 52 has been merged? :pleading_face:

aljazerzen commented 1 month ago

Done