search

MayankPandey01 / BrokenLinkHijacker

A Fast Broken Link Hijacker Tool written in Python
MIT License
97 stars 16 forks source link

Error while crawling at large size domains #6

Closed waelahmed-dev closed 1 year ago

waelahmed-dev commented 1 year ago

Sometimes, I get this error and I guess it happens when the target domain contains many links in it

ERROR While Crawling
maximum recursion depth exceeded while calling a Python object
waelahmed-dev commented 1 year ago

After adding this line: sys.setrecursionlimit(10000)

I got this error:

ERROR While Crawling
Exceeded 30 redirects.
MayankPandey01 commented 1 year ago

@waelahmed-dev you can change the value of sys.setrecursionlimit() on your Local System according to your use case.