search

datafusion-contrib / datafusion-objectstore-hdfs

HDFS based on Java implementation as a remote ObjectStore for DataFusion
Apache License 2.0
8 stars 8 forks source link

Upgrade object_store dependency and bump version #18

Open tlovell-sxt opened 8 months ago

tlovell-sxt commented 8 months ago

Currently running into an issue where some crates are upgrading to newer versions of arrow/parquet, the latter of which depends on object_store version 0.7. There's some versioning discrepancy happening if this crate is also depended on. In particular, I ran into this with the delta-rs crates: https://github.com/delta-io/delta-rs/issues/1793

As you can tell, the error is similar to #15, but that seems to have been a different incident with users getting automatic upgrades to datafusion-objectstore-hdfs version 0.1.4 and then having conflicting versions of object_store since 0.1.4 depended on 0.6 while some of their other, older crates they were depending on still depended on 0.5 or whatever. So, I think if we bump the object_store dependency here, it warrants a breaking change semver bump to 0.2.0 to avoid a similar issue occurring again.

After which, delta-rs would need to upgrade to the new version of this crate for the issue to be fully resolved

Jacarte commented 2 months ago

Hi all ! Any progress on this?