inventree / inventree-python

Python library for communication with inventree via API
https://docs.inventree.org/en/latest/api/python/python/
MIT License
27 stars 35 forks source link

Repeated API Connection crash when uploading large number of parameters #240

Closed realstudent01 closed 2 months ago

realstudent01 commented 3 months ago

I am trying to upload a large number of parameters from a script with the line param = Part.Parameter.create(api,{'part': part, 'template': template, 'data': value}). It works for a while and then fails with the following error. Adding a .05 second wait afterwards seems to allow it to work for longer, but it still crashes.

Error at api.request - POST @ http://server/api/part/parameter/

Traceback (most recent call last):

  File "FilePath\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 789, in urlopen

    response = self._make_request(

               ^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request

    response = conn.getresponse()

               ^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\urllib3\connection.py", line 464, in getresponse

    httplib_response = super().getresponse()

                       ^^^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\http\client.py", line 1428, in getresponse

    response.begin()

  File "FilePath\Python\Python312\Lib\http\client.py", line 331, in begin

    version, status, reason = self._read_status()

                              ^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\http\client.py", line 300, in _read_status

    raise RemoteDisconnected("Remote end closed connection without"

http.client.RemoteDisconnected: Remote end closed connection without response

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "FilePath\Python\Python312\Lib\site-packages\requests\adapters.py", line 667, in send

    resp = conn.urlopen(

           ^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 843, in urlopen

    retries = retries.increment(

              ^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\urllib3\util\retry.py", line 474, in increment

    raise reraise(type(error), error, _stacktrace)

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\urllib3\util\util.py", line 38, in reraise

    raise value.with_traceback(tb)

  File "FilePath\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 789, in urlopen

    response = self._make_request(

               ^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request

    response = conn.getresponse()

               ^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\urllib3\connection.py", line 464, in getresponse

    httplib_response = super().getresponse()

                       ^^^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\http\client.py", line 1428, in getresponse

    response.begin()

  File "FilePath\Python\Python312\Lib\http\client.py", line 331, in begin

    version, status, reason = self._read_status()

                              ^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\http\client.py", line 300, in _read_status

    raise RemoteDisconnected("Remote end closed connection without"

urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File "main.py", line 823, in <module>

    param = P.Parameter.create(api, { 'part': value.part, 'template': value.template, 'data': value.data})

            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\inventree\base.py", line 191, in create

    response = api.post(cls.URL, data, **kwargs)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\inventree\api.py", line 426, in post

    response = self.request(

               ^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\inventree\api.py", line 346, in request

    raise e

  File "FilePath\Python\Python312\Lib\site-packages\inventree\api.py", line 338, in request

    response = methods[method](api_url, **payload)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\requests\api.py", line 115, in post

    return request("post", url, data=data, json=json, **kwargs)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\requests\api.py", line 59, in request

    return session.request(method=method, url=url, **kwargs)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\requests\sessions.py", line 589, in request

    resp = self.send(prep, **send_kwargs)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\requests\sessions.py", line 703, in send

    r = adapter.send(request, **kwargs)

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "FilePath\Python\Python312\Lib\site-packages\requests\adapters.py", line 682, in send

    raise ConnectionError(err, request=request)

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

 

Error at api.request - POST @ http://server/api/part/parameter/ Traceback (most recent call last): File "FilePath\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 789, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\urllib3\connection.py", line 464, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\http\client.py", line 1428, in getresponse response.begin() File "FilePath\Python\Python312\Lib\http\client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\http\client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response During handling of the above exception, another exception occurred: Traceback (most recent call last): File "FilePath\Python\Python312\Lib\site-packages\requests\adapters.py", line 667, in send resp = conn.urlopen( ^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 843, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\urllib3\util\retry.py", line 474, in increment raise reraise(type(error), error, _stacktrace) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\urllib3\util\util.py", line 38, in reraise raise value.with_traceback(tb) File "FilePath\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 789, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\urllib3\connectionpool.py", line 536, in _make_request response = conn.getresponse() ^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\urllib3\connection.py", line 464, in getresponse httplib_response = super().getresponse() ^^^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\http\client.py", line 1428, in getresponse response.begin() File "FilePath\Python\Python312\Lib\http\client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\http\client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "main.py", line 823, in param = P.Parameter.create(api, { 'part': value.part, 'template': value.template, 'data': value.data}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\inventree\base.py", line 191, in create response = [api.post](http://api.post/)(cls.URL, data, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\inventree\api.py", line 426, in post response = self.request( ^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\inventree\api.py", line 346, in request raise e File "FilePath\Python\Python312\Lib\site-packages\inventree\api.py", line 338, in request response = methods[method](api_url, **payload) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\requests\api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\requests\api.py", line 59, in request return session.request(method=method, url=url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\requests\sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\requests\sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "FilePath\Python\Python312\Lib\site-packages\requests\adapters.py", line 682, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
matmair commented 2 months ago

Sounds like you are getting rate-limited by either the server or your reverse proxy. Are there errors thrown on the server? You should see them in your notifications if you are logged in asa superuser. If there are none it is probably a rate limit

realstudent01 commented 2 months ago

I've found a way around the reverse proxy issue, but I am now getting a server timeout. requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=8000): Read timed out. (read timeout=10). Is there a way to increase the timeout through the API?

realstudent01 commented 2 months ago

Nevermind

SchrodingersGat commented 2 months ago

I've found a way around the reverse proxy issue

What was the solution here @realstudent01 - in case it helps other users with a similar issue

realstudent01 commented 2 months ago

Unfortunately, it wasn't much of a solution... I simply switched to the inventree server I have running that has less firewalls in the way. My plan was to work on that one and then just import that database to the one that is more protected with mySQL manager.