Feature request: Bedrock Knowledge Bases Connector

[HQarroum]

Use case

Integrate Bedrock Knowledge Bases with Lakechain to allow customers to publish their documents in a Bedrock Knowledge Base

Solution/User Experience

No response

Alternative solutions

No response

[moltar]

While reading the docs I expected this to be one of the integrations and was quite surprised that it was not there. In fact, I kept looking for it, and even checked the code base to see if perhaps it was just undocumented.

[HQarroum]

Bedrock Knowledge Base takes its input documents from S3, it is a managed end-to-end service. We created this feature request as a mean to later integrate with Bedrock KB, but I don't think it is readily possible today.

If you need to process documents before they get indexed into Bedrock KB, you can do so with Lakechain using the S3 Connector and store your processed documents into the S3 bucket associated with your knowledge base.

But if we're missing something, let us know your exact use-case and the Bedrock KB API you'd like Lakechain to create an integration with.

[moltar]

Well, in my mind KB has two main components:

1. Ingestion
2. Retrieval

I understand that the scope of this project is similar to (1) Ingestion. Which I think Bedrock KB covers ok. But there's still certainly room for improvement. I'd love to be able to augment KB database with custom pipelines, that I could define using this project.

But there's also the retrieval part.

Yes, I could also preprocess using this project, and then land the files in the KB sourced S3. I think that would work fine too.

Perhaps a documented use case and an example of this integration would be sufficient to begin with?

[HQarroum]

Makes sense. We will provide an example for an integration with Bedrock Knowledge Bases.