A lot of connection failed

[drym3r]

Hi It seems that when executing linkcheck against https://daemons.it, some link won't be validated even when triying more than once. That link changes from one execution to other.

I'm using the dockerfile, if it helps. If I can add more information, just ask.

Thanks for your work, this program is awesome!

[filiph]

Sorry for not seeing this earlier. I think the issue is that linkcheck has quite a short connection timeout (it's optimized for localhost linkchecking). I'll investigate, but can't promise any fast turnaround.

[drym3r]

No problem, thanks!

[b3acon]

actually i have the same issue with "connection failed"

https://www.domain.tld/product-name/10025.4
- (88:56) <link> => https://www.domain.tld/theme/46c09ab3fb34724d339d029a49ca8888/css/all.css (connection failed)
- (88:56) <link> => https://www.domain.tld/theme/46c09ab3fb34724d339d029a49ca8888/css/all.css (connection failed)

[filiph]

@b3acon Is that the actual domain (www.domain.tld)? Just checking.

[jescalan]

Also having this issue - even just a way to manually increase the timeout would be super helpful. I could try to contribute this if you're up for it @filiph ?

[filiph]

That would be greatly appreciated, @jescalan!

[jescalan]

Ok so I forked it and tried locally increasing the timeout just to see if it would solve the problem. Unfortunately it did not - there were many "connection failed"'s, even with the timeout being 30s+. However, when I actually visit any of the pages, they load quickly and without issue. I think there may be something else going on here, but I do not have the dart expertise to diagnose it.

If it's helpful at all, the many "connection failed"s can be reproduced quite reliably by running the link checker against https://www.hashicorp.com

[filiph]

Thanks. I had a look just now and my theory is that this is the server refusing to connect what it sees like a DDoS attack (or, at the very least, a netiquette-ignoring spider).

Why I think so:

• Connection failures happen seemingly at random, for random pages, in a frequency of about 1:20 (1 connection error for every 20 passed checks).
• Trying the same site with blc (another link checker) does not show these failures. BLC also takes considerably longer time to crawl.

One way to test this hypothesis is to significantly increase the delay between retrievals. That is to say, significantly increase the constant here: https://github.com/filiph/linkcheck/blob/master/lib/src/server_info.dart#L8-L9 and see if that helps. I'd do it myself but I'm currently not at a computer with linkcheck repo checked out.

If this does help, then we could make the delay configurable.

That said, I always saw linkcheck more as a localhost tool, or a tool that one uses for one's own sites. If people use it to check others' sites, we should just hard code the duration between requests to be something like 1 second.

[jescalan]

@filiph this is great, really good call here. i will test the delay and verify if what you're saying here is accurate!

definitely agreed with localhost, the issue for us right now is that we have a large number of redirects that are harder to replicate when testing via localhost, and also we have this set up as github integration to run off a deploy preview URL so that we don't need to run a second build of the site inside CI. But given this issue and the expected slowdown, it may be time to think about revising this strategy!

[jescalan]

Confirmed that it is some sort of DDOS-protection measure and currently is only happening when hosting with Netlify specifically. I boosted the delay to over 1500ms and still experienced flakiness, so I'm not entirely sure what the story is here!

[mootari]

With up to eight threads (for external checks), increasing the timeout alone won't suffice.

If we could to set both the max number of threads as well as the timeouts, this should be enough to work around most rate-limit- or performance related issues.

[nkuehn]

Hi, I ended up here because of outstanding performance and completeness (including achor / fragment ID checking which seems to be a hard to implement rarity). We're also testing the remotely deployed preview URLs instead of the local build of the static sites because of various aspects like production server redirects, composing the overall site with independently deployed sub-sites, etc. and the concurrency seems to be a bit too high in fact. The sites are hosted as plain static content on Vercel.com.

We're currently using the nice linkinator (also from a "Googler" but javascript and no anchor checking) and had to tune the number of retries, retry jitter etc. too because of hitting DDoS protection. "retryErrors" is the feature that enabled us to push the limits of the external hosts at all, otherwise we'd have to reduce the overall speed a lot.

My dream implementation would dynamically reduce the concurrency to certain hosts when the number of needed exponential backoff retries grows. More than three tries have turned out to not be worth it for us and the jitter / delay should not be too long to not have the whole job stall into endless durations.

As a simpler fix, passing a max concurrency of non-localhost requests to the CLI could suffice. Maintaining maximum concurrency pools per external hosts sounds like it could be too complex given the potentially small impact.

[tooomm]

I'm running into the same issue... trying to run automated monthly health checks in CI on many images that we link to. Some customization options would be handy here.

[mikehearn]

It's not always DDoS protection related. I get this even on a server I know isn't DDoS protected.