raviqqe / muffet

Fast website link checker in Go
MIT License
2.46k stars 95 forks source link

Error: "missing port in address" when encountering local/relative paths #360

Closed gabek closed 4 months ago

gabek commented 5 months ago

I've been passively on the lookout for a link checker I can run against our documentation site for some time, and just ran into muffet. It's so fast! :)

I know this message isn't coming from muffet directly, and is coming from the Go stdlib. But I'm curious what I can do about this error being thrown for relative paths.

Thanks!

    missing port in address /troubleshoot/dropped-frames/
    missing port in address /troubleshoot/hardware-usage/
    missing port in address /troubleshoot/latency/
    missing port in address /troubleshoot/low-quality-video/
raviqqe commented 5 months ago

Can you provide the HTML files or the website URLs you are checking?

gabek commented 5 months ago

Sure! https://owncast.online/docs

For convenience, this is the invocation I'm using

muffet --buffer-size=8192 --max-connections=10 --color=always --rate-limit=2 --max-connections-per-host 10 --exclude="opencollective.com|github.com|example.com|mydomain.com" https://owncast.online/docs
raviqqe commented 5 months ago

The sitemap of the website doesn't seem to conform to the standard. Can you check that? https://www.sitemaps.org/protocol.html#locdef

gabek commented 5 months ago

Unfortunately, this is a Hugo generated site, so I'm not generating it myself. Is there a way to tell muffet to ignore the sitemap?

raviqqe commented 5 months ago

Maybe, --exclude option works for this case?

gabek commented 5 months ago

That seems to work, thanks so much!