search

cr0hn / golismero-legacy

THIS IS A LEGACY VERSION PRESERVED FOR BACKUP, DO NOT USE
http://golismero-project.com
15 stars 4 forks source link

Spider doesn't implement the blacklist properly #244

Closed MarioVilas closed 10 years ago

MarioVilas commented 10 years ago

The spider blacklist is based on words, not actual URL paths. Also, the blacklist is only applied to URLs found by spidering, not to the URL that's about to be spidered - so if another plugin discovers a forbidden URL, it gets spidered too.

Maybe we should even consider making the blacklist global, so it applies to all plugins, not just the spider. Thoughts?

cr0hn commented 10 years ago

Make the black list global can be a good option. So the core must check each URL, instead of the spider.

This change can be hard to implement in the spider plugin. Maybe, for this beta release, we can leave the code so and consider to change it in the next release.

MarioVilas commented 10 years ago

Yes, the global blacklist is a long term solution. But we should fix this for the beta, it should be just one line of code, then move the ticket to the other milestone. :)

cr0hn commented 10 years ago

Oks. For the moment, the check is hardcoded in spider only.

MarioVilas commented 10 years ago

I'll close this ticket and create a new one.