osean-man
commented
3 years ago I have encountered this same issue.
Open vorarush opened 4 years ago
osean-man
commented
3 years ago I have encountered this same issue.
For sites with many links it gives "RecursionError: maximum recursion depth exceeded" error. I think it uses tail recursion and that is why it stucks at default recursion limit of python. The logic needs to be improvised to not face this limit issue. Can you help?
Traceback (most recent call last): File "email_crawler.py", line 146, in
crawl.crawl()
File "email_crawler.py", line 40, in crawl
self.crawl()
File "email_crawler.py", line 40, in crawl
self.crawl()
File "email_crawler.py", line 40, in crawl
self.crawl()
[Previous line repeated 958 more times]
File "email_crawler.py", line 36, in crawl
self.parse_url(url)
File "email_crawler.py", line 62, in parse_url
response = requests.get(current_url, headers=self.headers)
File "\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, kwargs)
File "\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, kwargs)
File "\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, send_kwargs)
File "\lib\site-packages\requests\sessions.py", line 665, in send
history = [resp for resp in gen]
File "\lib\site-packages\requests\sessions.py", line 665, in
history = [resp for resp in gen]
File "\lib\site-packages\requests\sessions.py", line 245, in resolve_redirects
adapter_kwargs
File "\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, kwargs)
File "\lib\site-packages\requests\adapters.py", line 533, in send
return self.build_response(request, resp)
File "\lib\site-packages\requests\adapters.py", line 265, in build_response
response = Response()
File "\lib\site-packages\requests\models.py", line 608, in init
self.headers = CaseInsensitiveDict()
File "\lib\site-packages\requests\structures.py", line 46, in init
self.update(data, kwargs)
File "\lib_collections_abc.py", line 839, in update
if isinstance(other, Mapping):
File "\lib\abc.py", line 182, in instancecheck
if subclass in cls._abc_cache:
RecursionError: maximum recursion depth exceeded