search

temoto / robotstxt

The robots.txt exclusion protocol implementation for Go language
MIT License
269 stars 55 forks source link

sitemaps? #28

Closed hackermondev closed 3 years ago

hackermondev commented 3 years ago

is there a way to find sitemap URLs using this tool

temoto commented 3 years ago

Yeah, please try reading RobotsData.Sitemaps

robots, err := robotstxt.FromBytes(...)
if err != nil { ... }
fmt.Printf("sitemaps:\n%#v\n", robots.Sitemaps)