Investigate how to better handle indexations of pages with crazy amounts of links

medialab / hyphe

Websites crawler with built-in exploration and control web interface

http://hyphe.medialab.sciences-po.fr/demo/

GNU Affero General Public License v3.0

329 stars 59 forks source link

Closed boogheta closed 1 year ago

boogheta commented 2 years ago

For instance g1.globo.com

boogheta commented 1 year ago

the new option to disable recording of internal links should do the trick!