Infinitely crawling on web address redirecting to itself

soprasteria / cybersecurity-dfm

Data Feed Manager (news watch orchestrator to predict topic with deepdetect and store cleaned text in elasticsearch)

GNU General Public License v3.0

40 stars 14 forks source link

Closed acabrol closed 6 years ago

acabrol commented 6 years ago

When dfm crawl a web address which is redirecting to the same web address the crawling will never end.

acabrol commented 6 years ago

Unminify short urls has been disabled.