search

deepset-ai / haystack

:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
16.66k stars 1.83k forks source link

Support for Crawler in Haystack 2.x #6609

Open tewnut opened 8 months ago

tewnut commented 8 months ago

Haystack 2.0 should support user intention to add the Crawler in a similar way that was supported in version 1.x

Additional Information: Discord Discussion Link: Discord Discussion

mohitlal31 commented 6 months ago

Can I work on this? I recently fixed a bug in the crawler class, so I am familiar with the code. I can create a draft PR and ask for clarifications/feedback along the way.

mohitlal31 commented 6 months ago

Can I work on this? I recently fixed a bug in the crawler class, so I am familiar with the code. I can create a draft PR and ask for clarifications/feedback along the way.

@anakin87 Are you'll planning to take this up on your own? I'd love to work on this if it's not already in your pipeline.

PGryllos commented 4 months ago

hi folks, is someone working on porting the crawler?

sachinsachdeva commented 1 month ago

Hi @anakin87,

Any plans on picking this up ? I noticed @mohitlal31 already offered to help.