Fix incorrect lowercasing of robots.txt Sitemap URLs

GateNLP / ultimate-sitemap-parser

Ultimate Website Sitemap Parser

https://mediacloud.org/

Other

178 stars 64 forks source link

Open ArthurMelin opened 2 years ago

ArthurMelin commented 2 years ago

This PR fixes an issue with Sitemap URLs in robots.txt that have uppercase characters.

The .lower() call is redundant with the re.IGNORECASE flag usage in the next line.