Closed wchen-r7 closed 9 years ago
How about a companion tool / method to wipe out the broken references? If after removing broken refs, you'll also want to alert when a module is left with no references. Just seems silly to ship modules with known broken refs.
How about a companion tool / method to wipe out the broken references?
Automatically removing is one option for sure. But it's actually kind of hard to say what you're supposed to do with a bad link unless you inspect it. Sometimes it's bad because there's a typo, in that case removing the link would be kind of an overkill... just fix the typo, you know. Sometimes maybe the website is being DDOS'd so the link is only temporarily bad. Sometimes it's could be redirected (our Rex API doesn't follow redirects). Because of these unpredictable conditions I decided not to go there. This is actually kind of hard.
I wonder if we could use a site like isitdownorjustme or whatever it’s called to at least rule out the redirects, and maybe some types of DDoS
On Nov 5, 2014, at 7:10 PM, sinn3r notifications@github.com wrote:
How about a companion tool / method to wipe out the broken references?
Automatically removing is one option for sure. But it's actually kind of hard to say what you're supposed to do with a bad link unless you inspect it. Sometimes it's bad because there's a typo, in that case removing the link would be kind of an overkill... just fix the typo, you know. Sometimes maybe the website is being DDOS'd so the link is only temporarily bad. Sometimes it's could be redirected (our Rex API doesn't follow redirects). Because of these unpredictable conditions I decided not to go there.
— Reply to this email directly or view it on GitHub.
Yeah sometimes that's probably what you need to double check. Could be sort of an expensive one though (an extra web request to verify for every bad link). If I run the modified module_reference.rb tool for all module types (post, payload, aux, exploit, etc).... I think it takes about 3 to 4 hours.
hmmm yeah i guess it is hard if you're interested in preserving the link for inspection. But, at the end of whatever validation pass you take, there should still be a @tabassassin style sort of mass fix. If a link is so unreliable as to be DDoS'ed more than a couple times, it probably shouldn't be a reference.
Yeah I'll have to think about this. I might have to create another issue and call it "automate broken reference repair" or something like that.
Okeydokey.
I used the modified tool in #4138 to check all the references, and I found 159 broken links. I manually verified them one by one with a browser, so I'm confident this list is accurate.
They're broken due to all kinds of reasons: some domains no longer exist, resources/links are removed (or moved to somewhere I don't know), typos, etc, etc.