Url regex matching - Githubissues

indrajithi / tiny-web-crawler

A simple and easy to use web crawler for Python

MIT License

55 stars 11 forks source link

Closed Mews closed 1 week ago

Mews commented 1 week ago

This pr closes issue #13

Changes

Added url_regex argument to the Spider class.
Urls then get matched against this pattern inside the Spider.crawl method. If they don't match the pattern, they are skipped.
Also added a test case for this feature, in test_url_regex.

Please let me know if any changes need to be made to this pr.

indrajithi commented 1 week ago

@Mews Can you make a pull from master. The CI fails because of some outdated workflow in master.

Mews commented 1 week ago

I don't think that was the issue, it was because of the import re line, it should be before requests