search

datafusion-contrib / datafusion-federation

Allow DataFusion to resolve queries across remote query engines while pushing down as much compute as possible down.
Apache License 2.0
77 stars 18 forks source link

upgrade datafusion & arrow #57

Closed xhwhis closed 2 months ago

xhwhis commented 2 months ago

upgrade datafusion to 42 and arrow to 53. the .taplo.toml file is copied from dadafusion. BTW, I think we don't need datafusion and arrow dependencies, just need subcrates like dadafusion-expr and arrow-array. If you agree, I would do this.

hozan23 commented 2 months ago

BTW, I think we don't need datafusion and arrow dependencies, just need subcrates like dadafusion-expr and arrow-array. If you agree, I would do this.

@xhwhis Indeed, that would be useful to minimize the number of dependencies and reduce compile time. You can go ahead with this.

Thanks

xhwhis commented 2 months ago

@xhwhis Indeed, that would be useful to minimize the number of dependencies and reduce compile time. You can go ahead with this.

@hozan23 done

hozan23 commented 2 months ago

Hey @xhwhis

I am sorry but I believe this is not the ideal way to manage dependencies in rust. I thought you intended to use Cargo features to minimize the dependencies. Your current approach, where you're importing the subcrates, won't make a difference since you're also importing the main crate datafusion, It will also make it much harder to manage dependencies. It's best to reduce dependencies using Cargo features for the main crate. You can find the available features for datafusion here, which allows you to include only the necessary features (crates) for each crate in our repository.

xhwhis commented 2 months ago

merge PR #59, close it.