Stop crawling websites at some point

XeniaRieger / Modern-Search-Engines

0 stars 0 forks source link

Closed XeniaRieger closed 4 months ago

XeniaRieger commented 4 months ago

If wikipedia etc. was cralwed x times stop

LarsMaronde99 commented 4 months ago

Look into self.__crawl_state dict. Add a new key and count how often the site has been crawled