search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.52k stars 4k forks source link

Milvus #29008

Open RealChrisSean opened 1 year ago

RealChrisSean commented 1 year ago

https://milvus.io/

Total swag credits for this connector: 75

paideia1200bce commented 1 year ago

@RealChrisSean I'd like to take this up!

RealChrisSean commented 1 year ago

Awesome! This has been assigned to you! Good luck!

marcosmarxm commented 1 year ago

This connector doesn't exist right now. @RealChrisSean why this is in the hackathon?

RealChrisSean commented 1 year ago

It's a destination - vector database

RealChrisSean commented 1 year ago

Hi @amogh-opsverse hows it going? Just fyi, this is for a vector db destination so you won't be able to use our connector builder or low-code cdk to build this.

Here's a guide from our engineering team. Hopefully, this helps :D

https://transparent-trout-f2f.notion.site/How-to-implement-vector-database-destinations-dd3f5f020e0943ee93ca76e42d0a494a

paideia1200bce commented 1 year ago

@RealChrisSean, I was able to create a Milvus cloud account but didnt get a chance to play around with their docs, API endpoints etc. I wasn't fully aware of not being able to use the connector-builder for this. Thanks for the resource, I'm looking into it right now!

RealChrisSean commented 1 year ago

Hopefully it helps!

KarthikRaju391 commented 3 months ago

I am using airbyte to move data from mongodb to milvus, when setting up milvus, there are options to choose the embedding model. The only options are to use openai or cohere models. Is there no option to use other providers like huggingface for example?