search

rbignon / doctoshotgun

Script to automatically book a vaccine slot on Doctolib in the next seven days.
GNU General Public License v3.0
549 stars 142 forks source link

Random RecursionError after a while #306

Open seranpion opened 2 years ago

seranpion commented 2 years ago

Hello.

I have now twice ran into this critical exception while running the script:

An unexpected exception of type RecursionError occurred. Arguments: ('maximum recursion depth exceeded while calling a Python object',)

Nothing else to pinpoint the cause.

This happens after the script has run a long time. I am running Python for windows version 3.9.150.1013 on windows 10.

Other python programs do not trigger this, so I suspect that the script itself causes it. I will be running the script with the debug verbosity now, in case it happens again.

rbignon commented 2 years ago

Hello,

With only this log it is difficult to understand what occurs. Please update doctoshotgun to get the latest commit which will print the backtrace if you run it with -d.

Carlotronics commented 2 years ago

Hello,

I just encountered the same problem. Here are the last logs and stacktrace:

$ ./doctoshotgun.py --debug --pfizer --only-third --start-date 13/12/2021 --end-date 22/12/2021 fr 'city_with_accent,city,city_with_spaces_and_accents,city,city' <email> '<password>'
[...]
2021-12-13 15:58:52,759:INFO:browser::browsers.py:369:save_response Response saved to 9d351aa014634061a5eddd0f51095444
2021-12-13 15:58:52,760:DEBUG:browser::browsers.py:880:internal_callback Handle https://www.doctolib.fr/search_results/4904596.json?limit=4&ref_visit_motive_ids%5B%5D=8192&speciality_id=5494&search_result_format=json with CenterResultPage
2021-12-13 15:58:52,760:DEBUG:root::doctoshotgun.py:868:main Skipping city 'Paris' Pharmacie Mirabeau Lellouche
An unexpected exception of type RecursionError occurred. Arguments:
('maximum recursion depth exceeded while calling a Python object',)
Traceback (most recent call last):
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 866, in main
    for center in docto.find_centers(cities, motives):
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 357, in find_centers
    for center in self.find_centers(where, motives, next_page):
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 357, in find_centers
    for center in self.find_centers(where, motives, next_page):
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 357, in find_centers
    for center in self.find_centers(where, motives, next_page):
  [Previous line repeated 969 more times]
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 323, in find_centers
    self.centers.go(where=city, params={
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/woob/browser/url.py", line 98, in go
    r = self.browser.location(self.build(**kwargs), params=params, data=data, json=json, method=method, headers=headers or {})
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/woob/browser/browsers.py", line 911, in location
    response = self.open(*args, **kwargs)
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/woob/browser/browsers.py", line 898, in open
    return super(PagesBrowser, self).open(callback=internal_callback, *args, **kwargs)
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/woob/browser/browsers.py", line 790, in open
    return super(DomainBrowser, self).open(req, *args, **kwargs)
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/woob/browser/browsers.py", line 531, in open
    response = self.session.send(preq,
  File "/home/cd/phowork/doctoshotgun/./doctoshotgun.py", line 76, in send
    resp = super().send(*args, **kwargs)
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/cd/phowork/doctoshotgun/venv/lib/python3.9/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.9/http/client.py", line 1377, in getresponse
    response.begin()
  File "/usr/lib/python3.9/http/client.py", line 339, in begin
    self.headers = self.msg = parse_headers(self.fp)
  File "/usr/lib/python3.9/http/client.py", line 238, in parse_headers
    return email.parser.Parser(_class=_class).parsestr(hstring)
  File "/usr/lib/python3.9/email/parser.py", line 67, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/lib/python3.9/email/parser.py", line 56, in parse
    feedparser.feed(data)
  File "/usr/lib/python3.9/email/feedparser.py", line 176, in feed
    self._call_parse()
  File "/usr/lib/python3.9/email/feedparser.py", line 180, in _call_parse
    self._parse()
  File "/usr/lib/python3.9/email/feedparser.py", line 295, in _parsegen
    if self._cur.get_content_maintype() == 'message':
  File "/usr/lib/python3.9/email/message.py", line 594, in get_content_maintype
    ctype = self.get_content_type()
  File "/usr/lib/python3.9/email/message.py", line 578, in get_content_type
    value = self.get('content-type', missing)
  File "/usr/lib/python3.9/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/lib/python3.9/email/_policybase.py", line 316, in header_fetch_parse
    return self._sanitize_header(name, value)
  File "/usr/lib/python3.9/email/_policybase.py", line 287, in _sanitize_header
    if _has_surrogates(value):
  File "/usr/lib/python3.9/email/utils.py", line 57, in _has_surrogates
    s.encode()
RecursionError: maximum recursion depth exceeded while calling a Python object

2021-12-13 15:58:53,351:DEBUG:browser::browsers.py:1095:dump_state Stored cookies into storage

I started the script at ~14:06 CET so it ran near 2 hours before crashing. The current commit is 8c22878 (on which master currently points).

I am running Python 3.9.9 and installed following packages:

Babel==2.9.1
certifi==2021.10.8
charset-normalizer==2.0.9
cloudscraper==1.2.58
colorama==0.4.4
cssselect==1.1.0
html2text==2020.1.16
idna==3.3
lxml==4.6.5
Pillow==8.4.0
pyparsing==3.0.6
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
requests==2.26.0
requests-toolbelt==0.9.1
six==1.16.0
termcolor==1.1.0
Unidecode==1.3.2
urllib3==1.26.7
woob==3.0