Linkcheck reports cf-challenge protected pages as missing

filiph / linkcheck

Fast link checker

https://pub.dartlang.org/packages/linkcheck

MIT License

426 stars 51 forks source link

Linkcheck reports cf-challenge protected pages as missing #122

Open wom-bat opened 1 year ago

wom-bat commented 1 year ago

We run a website that has links to publications. Any publication hosted on a site protected by cloudflare challenges reports 403 instead of something useful.

For instance, a link to https://royalsocietypublishing.org/doi/10.1098/rsta.2015.0401 reports as 403 instead of passing.

I don't know what to do about this; I want to check that the links are valid, but the 403 response prevents this.

filiph commented 1 year ago

Unfortunately, I don't think there's anything an automated tool can do about this. If a website provides the wrong HTTP code, that's that. I've seen this happen with YouTube (I think) and github and other major websites.