Need to consolidate to a single HTML ingest

[mpgreg]

Please describe the feature you'd like to see Multiple extract functions use almost identical HTML extract logic.

Describe the solution you'd like Should consolidate to a single function if possible and use dynamic task mapping like github extract.

Are there any alternatives to this feature?

Additional context

Acceptance Criteria

• [ ] All checks and tests in the CI should pass
• [ ] Unit tests
• [ ] Integration tests (if the feature relates to a new database or external service)
• [ ] Example DAG
• [ ] Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
• [ ] Exception handling in case of errors
• [ ] Logging (are we exposing useful information to the user? e.g. source and destination)
• [ ] Improve the documentation (README, Sphinx, and any other relevant)
• [ ] How to use Guide for the feature (example)

Note:

• [ ] After the implementation is complete, the data should be ingested to the dev database and the dev slackbot should be deployed.
• [ ] This change should be tested by @vatsrahul1001 and after his take on quality of response, this should be merged

[sunank200]

Note:

• [ ] After the implementation is complete, the data should be ingested to the dev database and the dev slackbot should be deployed.
• [ ] This change should be tested by @vatsrahul1001 and after his take on quality of response, this should be merged

[sunank200]

Discussed with @pankajastro - a single HTML extractor makes sense.

[pankajastro]

just adding more here a single extractor makes sense but still we require a thin layer over it for the different sources because we need some different cleanup approaches for different sources for example in Astro SDK I'm excluding if the docs URL has "autoapi", "genindex.html", "py-modindex.html", ".md", ".py" but for provider excluding "_api", "_modules", "_sources", "changelog.html", "genindex.html", "py-modindex.html", "#". So at least a different task per source makes sense to me and if we keep a different dag per source then maybe it will be easy to run i.e we can upsert only the source we want

[phanikumv]

Created draft PR on this one

[pankajastro]

just marked PR as ready for review, would appreciate a review