search

deepset-ai / haystack

:mag: AI orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots.
https://haystack.deepset.ai
Apache License 2.0
17.36k stars 1.89k forks source link

MarkupConverter - convert HTML documents into plain text #3988

Open sebjwallace opened 1 year ago

sebjwallace commented 1 year ago

Is your feature request related to a problem? Please describe. The Crawler component works great. However, once I've saved the HTML documents to disc I'd like to convert the markup into plain text. This is a similar feature request, however I believe it'll be useful to have a dedicated node to clean up markup in-memory. Typically we'd want to keep our source data in its raw state on disc. This allows a user to change the parameters of the converter to yield different conversions without mutating the source.

Describe the solution you'd like MarkupConverter can use BeautifulSoup to exclude and/or include HTML tags. Given an input HTML document, a user can define that they want to exclude (link, style, script, etc.) Or they can define they'd only like paragraph tags. Allowing the user to define inclusive/exclusive sets allows flexibility of what kind of text they'd like to extract for their particular task.

Like the Markdown converter, the Markup converter can extract metadata from the header element.

sebjwallace commented 1 year ago

If this logically fits in as a Converter then I can place a PR soon enough :)

bogdankostic commented 1 year ago

Hi @sebjwallace! This seems to be an interesting feature request and is somehow related to #3838. To make sure we're all on the same page before investing time into the actual implementation, we would ask you to file a design proposal detailing the changes you would like to see in Haystack. Once approved, you could implement the proposal yourself, wait for a contributor to pick it up, or ask the core developers to prioritise it.