parse robots.txt when there is no response from website

kapouer / url-inspector

Get metadata about any url

MIT License

28 stars 8 forks source link

parse robots.txt when there is no response from website #26

Open kapouer opened 2 years ago

kapouer commented 2 years ago

Some websites won't answer to "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" other websites will answer to "Mediapartners-Google" or to "curl/7.79.1" or even no User-Agent.