search

lancedb / lance

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..
https://lancedb.github.io/lance/
Apache License 2.0
3.84k stars 212 forks source link

aws-* dependency maintenance #2511

Open NickDarvey opened 3 months ago

NickDarvey commented 3 months ago

Building my Rust project with lancedb@0.5 with aarch64-pc-windows fails because this architecture isn't supported with ring@0.16 ^1, which is a dependency because lancedb | lance -> lance-index -> lance-io -> aws-config@0.56 -> ring@0.16. Support for aarch64-pc-windows is available in ring@0.17 ^2.

Potential fixes

  1. Updating aws-config

    2510

    This PR bumps a minor version. However, the aws-* dependencies are depending on pre-release versions. The production-ready versions (1+) are now available. Maybe this is worth scheduling in #2079? (However, I'm guessing there are plenty of breaking changes.)

  2. Making the aws libs optional

    Could we make these dependencies optional by making them part of cargo feature named 'aws' for lance-io, lance-index, lance, and lancedb? I am not using AWS for my project, perhaps others aren't either.

    This feature could be enabled by default in lance and lancedb for backwards compatibility.

Workaround

In your Cargo.toml, add:

[patch.crates-io]
lance-io = { git = "https://github.com/nickdarvey/lance", branch = "patch-2" }
wjones127 commented 3 months ago

I agree we should make aws libs optional.