0xPlaygrounds / rig

⚙️🦀 Build portable, modular & lightweight Fullstack Agents
https://rig.rs
MIT License
153 stars 9 forks source link

feat(loaders): CSV Loader to Document Loaders #30

Closed Tachikoma000 closed 4 weeks ago

Tachikoma000 commented 1 month ago

This PR implements a CSV loader as part of the document loaders module in Rig. It allows users to easily load and process CSV documents for use in RAG systems and other document processing tasks.

Changes

Implementation Details The CsvLoader uses the csv crate to parse CSV files and extract content. It handles potential errors such as file not found or parsing errors. The extracted content is converted into a single DocumentEmbeddings object for further processing in Rig. Each row of the CSV is formatted as "header: value" pairs, separated by newlines.

Testing Ran tests to ensure the CsvLoader correctly loads CSV files and handles various edge cases. The tests covered:

Documentation Code files are commented, and usage examples have been added to the documentation.

Related Issue Closes #29

Checklist

Additional Notes This implementation focuses on converting CSV data into a single document for embedding. Future enhancements could include options for creating separate embeddings for each row or handling more complex CSV structures.

cvauclair commented 4 weeks ago

This is being (re-)implemented in #55