Make crawler respect robots.txt constraints

XeniaRieger / Modern-Search-Engines

0 stars 0 forks source link

Closed LarsMaronde99 closed 2 months ago

LarsMaronde99 commented 2 months ago

make crawler respect the delay, request limit constraints, and allowed/disallowed paths constraints