search

raviqqe / muffet

Fast website link checker in Go
MIT License
2.49k stars 96 forks source link

Provide an argument to filter out certain HTTP error codes #291

Open spkane opened 1 year ago

spkane commented 1 year ago

Make it possible to ignore a set of HTTP error codes, like -g 403,999.

spkane commented 1 year ago

there might be another approach to handling/fixing this, but this is primarily being mentioned do to URLs like:

https://www.linkedin.com/in/spkane/ (which always returns an HTTP 999 to muffet, despite being a valid URL) and https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html (which always returns an HTTP 403 to muffet, despite being a valid URL)

spkane commented 1 year ago

Based on #292, being able to set some code to ignore and some to report/warn but not error on at a global level would be useful, in addition to eventually also being able to set this on a per-patter basis.

raviqqe commented 1 year ago

I also prefer a proposal in #292 to this because users can't know false positives when websites' behavior changes and error status codes are ignored globally. One concern of ignored status codes per pattern is their complexity...

spkane commented 1 year ago

Agreed.

292 basically built on this idea and came to me as I was trying to implement the scanner GitHub action into one of my website deployment processes.