search

datafusion-contrib / datafusion-objectstore-s3

S3 as an ObjectStore for DataFusion
Apache License 2.0
59 stars 13 forks source link

basic implementation #2

Closed seddonm1 closed 2 years ago

seddonm1 commented 2 years ago

ObjectStore implementation for the Amazon S3 API

this takes the feedback from the gist: https://gist.github.com/seddonm1/2fb5a6892989fe7bf246022a7bd586ee and adds tests against the standard test suite.

CI could be set up this way.

matthewmturner commented 2 years ago

Good for me. I was able to test locally and run sql queries.

This gives us great baseline for now adding CI, which ill work on, and creating new issues.

matthewmturner commented 2 years ago

@alamb do you think youll have the chance to check this out today? I was hoping to just use this as a baseline for subsequent improvements which could be discussed in issues / other PRs.

alamb commented 2 years ago

@alamb do you think youll have the chance to check this out today? I was hoping to just use this as a baseline for subsequent improvements which could be discussed in issues / other PRs.

@matthewmturner I wasn't really planning on reviewing this one (was focusing on getting arrow 7.0.0 ready to go and other stuff for IOx). Do you want me to review it (or are you waiting on someone to merge the PR)?

If you plan to work heavily in this repo, perhaps @houqp would be cool with giving you owner rights (we brought it outside the arrow / apache landscape to lower the bar for getting people access I think)

matthewmturner commented 2 years ago

@alamb was just pinging you as @houqp had requested review from you. I'm definitely happy to run with this I've already merged a couple commits.

@houqp anything in particular you wanted to be reviewed?

alamb commented 2 years ago

I'll try and check it out over the next few days -- but please don't wait for me to merge 🚀 !

matthewmturner commented 2 years ago

I'll try and check it out over the next few days -- but please don't wait for me to merge 🚀 !

thank you @alamb 🙏

houqp commented 2 years ago

I have already given @matthewmturner and @seddonm1 owner access to this repo. Also tagged Andrew just in case he has any blocker comments. I think the repo right now is in a highly experimental state, so we can merge suboptimal solutions and iterate from there.