search

nietaki / crawlie

A simple Elixir library for writing decently-performing crawlers with minimum effort.
MIT License
89 stars 11 forks source link

Elliminating duplicate urls #3

Closed nietaki closed 7 years ago

nietaki commented 7 years ago

Make UrlManager reject urls it already processed.

Bear in mind, if a url leaves the UrlManager, it doesn't mean it was processed - it can still be retried.