Using Googlebot as user agent by default causes issue on some links.
For example, instagram redirects to login page after few anonymous requests which causes all the preview to be blocked.
This PR uses the Googlebot as user agent as a backup ONLY if the default HTTP request result with no user-agent specified is empty.
Fixes #38.
Using
Googlebot
as user agent by default causes issue on some links. For example, instagram redirects to login page after few anonymous requests which causes all the preview to be blocked.This PR uses the
Googlebot
as user agent as a backup ONLY if the default HTTP request result with no user-agent specified is empty.