add check of already_downloaded by a hashset - Githubissues

hellock / icrawler

A multi-thread crawler framework with many builtin image crawlers provided.

http://icrawler.readthedocs.io/en/latest/

MIT License

857 stars 174 forks source link

add check of already_downloaded by a hashset #88

Closed ZhiyuanChen closed 3 years ago