search

lancedb / lancedb

Developer-friendly, serverless vector database for AI applications. Easily add long-term memory to your LLM apps!
https://lancedb.github.io/lancedb/
Apache License 2.0
3.5k stars 234 forks source link

Feature(Node): Lance Lambda Layer in AWS Serverless Application Repository #1010

Open LincolnCannon opened 4 months ago

LincolnCannon commented 4 months ago

SDK

Node

Description

Deploy and maintain a Lance lambda layer for use by Node.js Lambda functions in the AWS Serverless Application Repository. This will greatly facilitate onboarding.

LincolnCannon commented 2 months ago

Any chance of this getting bumped up in priority? I'd love to use LanceDB more often and share it with more developers, but setup on AWS is still kind of a pain. It would be really nice to be able to tell people that setup and use is EASY. Thanks for your consideration.

wjones127 commented 2 months ago

I'm not sure any of the devs at LanceDB have bandwidth to work on this right now. However, we'd welcome a contribution if someone wanted to set up whatever templates and CI jobs are needed to do this. Even if you didn't want to do that yourself, writing down in this issue what the necessary steps would be to complete this would be a useful step in the right direction.

rpgreen commented 2 months ago

HI @LincolnCannon - can you share some information on how using a layer would make things easier for your use-case vs. using a package manager like NPM?

westonpace commented 2 months ago

@LincolnCannon do you know of any open repositories / projects that have good existing CI for building / packaging / releasing lambda layers that we can use as an example?

I assume we can use something like aws-sdk-pandas-layer-py3-11 as the base layer? Is there any other pyarrow base layer that might be more standard?

LincolnCannon commented 1 month ago

@LincolnCannon do you know of any open repositories / projects that have good existing CI for building / packaging / releasing lambda layers that we can use as an example?

@westonpace I recommend Serverless for building and packaging AWS lambda layers. https://www.serverless.com/

I assume we can use something like aws-sdk-pandas-layer-py3-11 as the base layer? Is there any other pyarrow base layer that might be more standard?

I also recommend that you start with a Nodejs lambda layer, rather than Python, to accelerate adoption.