Parse sitemaps values with full urls correctly

BedrockStreaming / roboxt

:no_entry: DEPRECATED - simple parser for robots.txt

19 stars 7 forks source link

Closed benja-M-1 closed 9 years ago

benja-M-1 commented 9 years ago

The following directive was not parsed correctly

Sitemap: https://www.google.com/edu/sitemap.xml

status	name	value
✘	https
✔	Sitemap	https://www.google.com/edu/sitemap.xml