RecursionError when pulling more than 475 pages of Orders

[tomaszsmolarek]

Issue summary

When one pulls Orders data from API with auto-iteraton (no_iter_next=True), library iterates over pages and when it hits page 476 an error is thrown. Also, looks like there are memory problems because of this recursion as our servers just die and we have to reboot them, although I have no clear evidence of this. It's unknown if this impacts other data type (Customers, Products...) - although it's very likely it does as the code that crashes is generic collections code.

edited: Seems like default recursion limit in py3 is 1000. The code does 2 * 475 = 950 iterations. Looks like 50 is for other operations the code is doing. The code crashes at the same place, always.

Expected behavior

No error is thrown, code continues.

Actual behavior

Error is thrown:

  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/shopify/collection.py", line 112, in __iter__
    for item in self._current_iter:
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/shopify/collection.py", line 112, in __iter__
    for item in self._current_iter:
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/shopify/collection.py", line 112, in __iter__
    for item in self._current_iter:
  [Previous line repeated 949 more times]
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/shopify/collection.py", line 110, in __iter__
    self._current_iter = self.next_page()
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/shopify/collection.py", line 89, in next_page
    return self.__fetch_page(self.next_page_url, no_cache)
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/shopify/collection.py", line 92, in __fetch_page
    next = self.metadata["resource_class"].find(from_=url)
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/shopify/base.py", line 211, in find
    collection = super(ShopifyResource, cls).find(id_=id_, from_=from_, **kwargs)
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/pyactiveresource/activeresource.py", line 386, in find
    return cls._find_every(from_=from_, **kwargs)
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/pyactiveresource/activeresource.py", line 525, in _find_every
    response = cls.connection.get(path, cls.headers)
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/pyactiveresource/connection.py", line 329, in get
    return self._open('GET', path, headers=headers)
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/shopify/base.py", line 25, in _open
    self.response = super(ShopifyConnection, self)._open(*args, **kwargs)
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/pyactiveresource/connection.py", line 286, in _open
    http_response = self._handle_error(self._urlopen(request))
  File "/usr/local/.virtualenvs/agents36/lib/python3.6/site-packages/pyactiveresource/connection.py", line 316, in _urlopen
    return urllib.request.urlopen(request, timeout=self.timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/local/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/local/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/usr/local/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/local/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 694, in do_handshake
    match_hostname(self.getpeercert(), self.server_hostname)
  File "/usr/local/lib/python3.6/ssl.py", line 298, in match_hostname
    host_ip = ipaddress.ip_address(hostname)
  File "/usr/local/lib/python3.6/ipaddress.py", line 44, in ip_address
    return IPv4Address(address)
  File "/usr/local/lib/python3.6/ipaddress.py", line 1284, in __init__
    self._ip = self._ip_int_from_string(addr_str)
  File "/usr/local/lib/python3.6/ipaddress.py", line 1117, in _ip_int_from_string
    if len(octets) != 4:
RecursionError: maximum recursion depth exceeded in comparison
Command exited with return code 1

Steps to reproduce the problem

1. Make sure you have enough test data in shop - you can set page size to a fairly small number (e.g. 50) to have to use less data
2. Call API with python library with auto-iteration ON
3. Error raises

Specifications

• shopify_python_api version: 6.0.1 and 8.1.0 (possible more versions)
• Shopify API version used (e.g. '2020-07'): 2020-01 and 2020-07 (possible more versions)

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[babinos87]

If it is not fixed, I advise the bot to NOT close this ticket.

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[arkadiusz-lisiecki]

Hi! I have the same issue as @tomaszsmolarek :/.

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[tomek-at-rais]

We're also facing the same problem. :(

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[babinos87]

not resolved, still relevant, not stale

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[babinos87]

not resolved, still relevant, not stale

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[tomek-at-rais]

Not fixed, please don't close.

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[babinos87]

still happens, not fixed

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[babinos87]

not solved

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[babinos87]

Isn't shopify, a company with millions of users and revenue, interested in fixing this?

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[babinos87]

not fixed, still a problem

[alex5207]

not fixed, still a problem

Could you provide a code example that produces the error?

[github-actions[bot]]

This issue is stale because it has been open for 60 days with no activity. It will be closed if no further action occurs in 14 days.

[babinos87]

Has it been resolved? If not, why close it?