search

Miserlou / NoDB

NoDB isn't a database.. but it sort of looks like one.
https://blog.zappa.io/posts/introducing-nodb-pythonic-data-store-s3
380 stars 45 forks source link

S3 Select support #16

Closed lormayna closed 5 years ago

lormayna commented 6 years ago

Hi, do you think to support S3 Select in the future?

bendog commented 5 years ago

could you flesh this out? maybe give an example?

lormayna commented 5 years ago

I mean taking advantage of this feature: https://medium.com/@zxlee618/s3-select-s3-is-actually-more-capable-than-what-i-thought-83251985e540

bendog commented 5 years ago

I read up on it

Limitations
Currently, we cannot run aggregation queries on complete folder level, 
because the queries that run are on individual file on S3.
This limitation is on S3 Select.

Since NoDB stores each "row" as a different file I'm not sure how this new feature could be leveraged.

lormayna commented 5 years ago

Okay, nevermind then. You can close the issue.

Thank you for spending time on this and clarify my doubts.

Il giorno gio 12 set 2019 alle ore 09:56 Ben Fitzhardinge < notifications@github.com> ha scritto:

I read up on it

Limitations Currently, we cannot run aggregation queries on complete folder level, because the queries that run are on individual file on S3.This limitation is on S3 Select.

Since NoDB stores each "row" as a different file I'm not sure how this new feature could be leveraged.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Miserlou/NoDB/issues/16?email_source=notifications&email_token=ABMUJC2LR6APJFS3KDH7DJDQJHY23A5CNFSM4E3GC472YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6RAHQY#issuecomment-530711491, or mute the thread https://github.com/notifications/unsubscribe-auth/ABMUJC33HDZAXOZYSGBT5Q3QJHY23ANCNFSM4E3GC47Q .

-- LORENZO MAINARDI

bendog commented 5 years ago

all good! it's a good concept to consider so thankyou, and if AWS ever allowed S3 Select to run over multiple files, it would be very useful for doing .filter() style queries with NoDB.