raid-guild / gaianet-rag-api-pipeline

Supercharge your Gaianet node by generating a vector knowledge base from any API. Demo slides: https://hackmd.io/@santteegt/ByoykY4nC#/ Link to Docs below
https://raid-guild.github.io/gaianet-rag-api-pipeline/
MIT License
1 stars 0 forks source link

1.6 Transfer Data Chunks to Embeddings #9

Closed earth2travis closed 3 months ago

earth2travis commented 4 months ago

Connect outputs from the custom parser module to the next stage in the pipeline. The parsed and structured data needs to be properly linked to the embedding system to ensure a smooth transition from raw data to vector embeddings. This connection is crucial for maintaining data integrity and ensuring that all relevant information is accurately processed in subsequent steps.

Transfer normalized data chunks to the embedding system for vectorization. This step involves feeding the processed data into the embedding model to generate vector representations, which are essential for creating a searchable knowledge base. Ensuring the accurate and efficient transfer of data chunks is crucial for the performance of the embedding system.