Add Yandex and Wikimedia crawler to the static crawler

Huy-Ngo / skin-crawler

A web crawler for crawling dermoscopic images of skin cancer.

MIT License

4 stars 2 forks source link

Add Yandex and Wikimedia crawler to the static crawler #29

Closed Huy-Ngo closed 3 years ago

Huy-Ngo commented 3 years ago

It seems even for 1 search, the Wikimedia crawler is time taking. I suppose Yandex one would suffer the same problem. Maybe we should crawl them beforehand.

This also let us write code consistently with the other two crawlers.

Huy-Ngo commented 3 years ago

@DDHLong @dinhanhx your opinion on this? If you agree I'll resolve this in #27 right away.

dinhanhx commented 3 years ago

I agree with you crawling first because the requirements for this project don't say anything about real-time crawling.