michalc / sqlite-s3-query

Python functions to query SQLite files stored on S3
MIT License
251 stars 15 forks source link

docs: more examples needed #2

Closed souzaramon closed 3 years ago

souzaramon commented 3 years ago

I'm planing to use this in a aws-lambda function, what is the best aproach to do this? Cause the result from a query is a interator (as showed), and it can't be returned directly (or it can be) from the function

btw, awesome job 💯

michalc commented 3 years ago

The iterable part I suspect is solvable by passing it through list

results_iter = sqlite_s3_query(
    'SELECT * FROM my_table WHERE my_column = ?', params=('my-value',),
    url='https://my-bucket.s3.eu-west-2.amazonaws.com/my-db.sqlite',
)
results_list = list(results_iter)

and then I suspect it can be returned from an AWS Lambda function either directly, or by wrapping in a dictionary. So something like:

def my_handler(event, context):
    return {
      'results': list(sqlite_s3_query(
         'SELECT * FROM my_table WHERE my_column = ?', params=('my-value',),
          url='https://my-bucket.s3.eu-west-2.amazonaws.com/my-db.sqlite',
       ))
     }

(Untested)

One other thing I realise I'm not sure on in AWS Lambda is the APSW dependency... it's crossed my mind that could be a tricky one since AFAIK it involves compiling C code, and I'm not sure how friendly Lambda is to Python packages that need that...

souzaramon commented 3 years ago

Thanks for your answer,

kinda sad cast this in a list, cause it will go directly to memory right? Really good point in the APSW btw. I'm current searching for a sqlite vfs solution to use in lambda functions. I'll probably make some tests with this repo, if I find something usefull, I'll try to contribute.

Thanks, Cheers

michalc commented 3 years ago

@souzaramon In case it's helpful, have now removed the dependency on APSW