Amazon Bedrock embedding support and reduce the package size to pack it as AWS Lambda layer

[debdayal-aws]

I am unable to add nemoguardrails in a AWS Lambda layer as it pulls all sorts of dependencies. Also version 0.5.0 does not support langchain > 0.0.251 which becomes a blocker for using new features in langchain. How do i get a slimed down version of this package with langchain support?

[drazvan]

Hi @debdayal-aws ! The list of dependencies is kept to the minimum, there isn't much that can be stripped you. However, if, for example, you don't want to use sentence-transformers/annoy and implement your own custom embedding search provider (which may call an API), you can skip those dependencies. Also, if don't need the server, you can remove fastapi.

Regarding langchain, it is not incompatible with newer versions, it's just that version 0.5.0 points to langchain==0.0.251. If you manually update langchain after installing nemoguardrails it should still work.

[debdayal-aws]

@drazvan Thank you so much for your response, ok then I can build a wheel excluding those packages, could you point me to an example how i can configure an API based embedding search e.g. with Amazon Bedrock or any other SageMaker hosted (JumpStart or HuggingFace) open source embedding searches?

[trebedea]

@debdayal-aws You can easily use OpenAI embeddings, as shown here: https://github.com/NVIDIA/NeMo-Guardrails/blob/main/docs/user_guide/advanced/embedding-search-providers.md Or take a look at the documentation and the code to use OpenAI embeddings and change it to use another API-based embeddings provider. You can even contribute your code in a MR.

On another hand, the main bulk of the nemoguardrails size comes from pytorch with GPU support required by a default installation of pytorch (required from sentence-transformers). However all-MiniLM-L6-v2 is very fast on CPU (most people use it on CPU) so the bulk of the pytorch GPU installation is actually not used. This is mainly a sentence-transformers issue and there are fixes to only install the CPU support for pytorch and sentence-transformers: https://github.com/UKPLab/sentence-transformers/issues/1409

Hope this helps you and other people that want to use sentence-transformers and NeMo Guardrails together but have a slimmer install.

[debdayal-aws]

@trebedea Really appreciate your prompt and detailed response. I'll start working on the customization and get back to you.

[debdayal-aws]

@trebedea @drazvan I have customized the lib to remove the hard dependency with sentence-transformers, and now we are able to package the lib with AWS Lambda. My recommendation would be to remove the hard dependency with sentence-transformers to make this lib packable with Lambdas and Containers. It can be an additional dependency if someone needs to use sentence-transformers, i.e. non-api based embedding model.
Also integrated with Amazon Bedrock Embedding Model. Now this library will work with Amazon Bedrock like OpenAI. Please let me know the process of MR for contributing the code to the repo. One unrelated issue - this lib is not compatible with latest version of pydantic which is causing conflict while packaging this lib with others. Let me know if there is any plan to handle this in the next version.

[EthanDev]

@trebedea @drazvan I have customized the lib to remove the hard dependency with sentence-transformers, and now we are able to package the lib with AWS Lambda. My recommendation would be to remove the hard dependency with sentence-transformers to make this lib packable with Lambdas and Containers. It can be an additional dependency if someone needs to use sentence-transformers, i.e. non-api based embedding model. Also integrated with Amazon Bedrock Embedding Model. Now this library will work with Amazon Bedrock like OpenAI. Please let me know the process of MR for contributing the code to the repo. Meanwhile if anyone is facing this issue please direct towards me for the solution. One unrelated issue - this lib is not compatible with latest version of pydantic which is causing conflict while packaging this lib with others. Let me know if there is any plan to handle this in the next version.

Hi, I am in the process of implementing this with Langchain RetrievalQA but am struggling to use Bedrock for embeddings. @debdayal-aws could you guide to me some resources on how to incoporate NeMo Guarrdrails and AWS Bedrock?

[juanrincong]

@debdayal-aws It should be great if you can help the people who are trying to do the same (aws lambda integration + bedrock support). Could you please provide guidance to reproduce your results and being able to customize the lib in the meantime or give us more information as a starting point to do so? Thank you in advance.

[debdayal-aws]

Amazon Bedrock team has provided NeMo integration example here. Please refer to this workshop example if anyone is looking for Bedrock integration with NeMo