Open lahabana opened 1 year ago
Some websites (.e.g github) have aggressive throttling if we crawl unauthenticated. It would be nice to be able to set some headers on specific hosts.
That would make it possible to add a Github token auth to check links to github.
Maybe this would require to introduce a "per host" config where we could have a set of host specific config.
Some websites (.e.g github) have aggressive throttling if we crawl unauthenticated. It would be nice to be able to set some headers on specific hosts.
That would make it possible to add a Github token auth to check links to github.
Maybe this would require to introduce a "per host" config where we could have a set of host specific config.