End-to-end RAG tutorial using Amazon SQS, PyAirbyte and Milvus (using Langchain)

airbytehq / PyAirbyte-Hackathon

Tasks for PyAirbyte Hackathon June 2024

0 stars 2 forks source link

End-to-end RAG tutorial using Amazon SQS, PyAirbyte and Milvus (using Langchain) #21

Open aaronsteers opened 1 month ago

aaronsteers commented 1 month ago

Summary

The goal is to show users how to use PyAirbyte to load data from Amazon SQS source, store the data in Milvus and perform a short RAG demo (using OpenAI/LangChain) on the loaded data in Milvus.

Deliverable

Python notebook illustrating end to end RAG using PyAirbyte, Amazon SQS source,LangChain & Milvus. Please add a "What / Why / How" blurb at the top of the notebook to explain what the code is doing.

Resources to Assist

PyAirbyte tutorials: https://github.com/airbytehq/quickstarts/tree/main/pyairbyte_notebooks
PyAirbyte Documentaton: https://docs.airbyte.com/using-airbyte/pyairbyte/getting-started
PyAirbyte API Reference: https://docs.airbyte.com/using-airbyte/pyairbyte/getting-started#api-reference
Amazon SQS documentation: https://docs.airbyte.com/integrations/sources/amazon-sqs
Milvus (Zilliz) langchain integration: https://python.langchain.com/v0.1/docs/integrations/vectorstores/zilliz/

aazam-gh commented 3 weeks ago

I'll take this up :)

aazam-gh commented 3 weeks ago

Here do we need to use Vectara or Milvus vector database?

marcosmarxm commented 3 weeks ago

Milvus, probably a typo from other issue.

marcosmarxm commented 3 weeks ago

@aaronsteers fyi

marcosmarxm commented 1 week ago

@aazam-gh are you working on this one? Need any help?

aaronsteers commented 1 week ago

@aazam-gh - Yes, @marcosmarxm is correct: this should be "Milvus". I see @marcosmarxm has already updated the description. 🙏

@aazam-gh - Let us know if you need any further assistance on this. Thanks!

15vaibhav2003 commented 1 week ago

Hey, Just in case @aazam-gh is no more working on this or planning to quit, I am quite eager to work on this! Please assign it to me if thats the case!

aazam-gh commented 1 week ago

Hi, I won't have time to work on this, you can unassign me :)

15vaibhav2003 commented 1 week ago

@aazam-gh Thanks for the quick reply! @marcosmarxm Please assign it to me!

15vaibhav2003 commented 1 week ago

Hey Team Airbyte @marcosmarxm @aaronsteers I have started working on the issue. Could you please provide me an AWS IAM Access Key,AWS IAM Secret Key,AWS SQS Queue so that I can test my code?🙇‍♂️ Thank You 🙏

15vaibhav2003 commented 1 week ago

@aaronsteers Before this issue is resolved another issue needs your attention. The Amazon SQS source connector must be updated to remove the legacy Logger , as told by @marcosmarxm on Slack. Therefore I request you to create a new issue for this task in this hackathon and assign it to me! 🙇

15vaibhav2003 commented 6 days ago

Hey Team @marcosmarxm @aaronsteers @bindipankhudi any update on this?

marcosmarxm commented 5 days ago

@15vaibhav2003 I created https://github.com/orgs/airbytehq/projects/75/views/3?pane=issue&itemId=68752114 please comment in the isssue so I can assigned to you.