fastai / fastlinkcheck

Check local static links and online links fast and in parallel
https://fastlinkcheck.fast.ai/
Apache License 2.0
13 stars 7 forks source link

broken links are not broken #79

Open slowkow opened 3 years ago

slowkow commented 3 years ago

Thanks for a cool project.

I tried it, but it doesn't seem to be working for me.

I ran link_check content got this (trimmed to just 2 results):


ERROR: The Following Broken Links or Paths were found:

- 'https://github.com/slowkow/circles' was found in the following pages:

- 'pubmed.ncbi.nlm.nih.gov/31110316' was found in the following pages

As I go ahead and manually test all of the links, it seems none of them are broken.

Am I missing something?

zonca commented 3 years ago

Same issue here, some github links fail. I tested directly the urlcheck function:

image

In text format:

urlcheck('https://www.bing.com/')

urlcheck('https://www.github.com/')

urlcheck('https://github.com/')

urlopen('https://github.com/').status

urlcheck('http://github.com/')

urlcheck('https://google.com/')

urlcheck('https://www.google.com')
fabge commented 3 years ago

Same here!

Ankur-singh commented 3 years ago

I am getting the same error. I did some digging and found that its because of timeout argument in urlopen function. Removing the timeout argument worked for me. But now it's taking much longer to check all the broken links.

Also, sometimes, it simply freezes. Under the hood, parallel method from fastcore is used to check all the URLs. Hence, the code will wait for all the threads to complete. I am guessing, without timeout argument, if any single thread is still under execution, the complete code is stuck.