search

nelhage / llama

Apache License 2.0
589 stars 24 forks source link

Other clouds? #20

Open yjftsjthsd-g opened 3 years ago

yjftsjthsd-g commented 3 years ago

It looks like the current implementation is relatively tied to AWS, but in the long term is there any chance that llama could be extended to support GCP (Google Cloud Run + Google Cloud Storage look like a nearly drop-in replacement for lambda/s3) or even an arbitrary docker host/hosts and arbitrary s3-compatible object store?

chadbrewbaker commented 3 years ago

You would have to do something like SQLite does and have a virtual file system. AWS S3 supports HTTP range queries, https://github.com/dacort/athena-sqlite/blob/master/lambda-function/vfs.py which can be a big performance win by not having to download entire files.

nelhage commented 3 years ago

It's totally possible. S3 is already mostly hidden behind the store.Store interface, but you'd need to add a few shims/interfaces in other places. The actual surface area for interacting with AWS -- other than the initial configuration bootstrap -- is fairly small, so I don't think it would be a huge project.

gg reported substantially better results on Lambda than on Google Cloud Run, though, which is the main reason I chose to target it first.