Feature: Support for regular expression pattern for url crawling

[indrajithi]

• Only crawl the websites which matches the regular expression pattern

[indrajithi]

@Mews Let me know if I can assign this to you. Thanks!

[Mews]

Yep, go ahead.

[Mews]

Are there any pre-commit hooks I should install?

[indrajithi]

Yes, I have added a pre commit hook to check for lint errors. poetry install --with dev should trigger script.py and install the pre-commit hook. LMK if it does not work. If not you can manually install the hook

[indrajithi]

Done in this #PR. Closing