Config - Extend functionality for more scraping tasks

[tubamos]

Item Type: feature request

Description: The implemented config functionality to support the youtube and podcast scraping should be extended to support Pubmed, Arxiv, NHS, recipe, Allrecipes, nutritionfacts, and other relevant website scraping.

User story

• As an industry partner
• I need to have a comprehensive config file that accomodate all sources to be scrapped.
• so that all the relevant sources are transcriped in one go.

Acceptance criteria

• The existing config functionality is extended to support Pubmed, Arxiv, NHS, recipe, Allrecipes, nutrionfacts.
• A User can specify the URLs, HTML elements, or patterns specific to each website for scraping content.
• It allow users to define scraping rules, such as selecting specific sections or fields to extract.

[tubamos]

@eloinoel @EnescuDavid

I suppose this ticket is also obsolete right not and can be deleted entirely, correct?