vesper8
commented
2 months ago It seems to be related to the /upload/image somehow.
I'm passing the same image, converted to base64, for each iteration of my loop. This isn't a particular big image, only about 300kb.
It works fine the first couple of times but then I start seeing this error over and over again:
comfyui-worker | DEBUG | test-09828fb9-5b04-46cb-bae2-205ca2680559 | run_job return: {'error': '{"error_type": "<class \'requests.exceptions.ConnectionError\'>", "error_message": "HTTPConnectionPool(host=\'127.0.0.1\', port=8188): Max retries exceeded with url: /upload/image (Caused by NewConnectionError(\'<urllib3.connection.HTTPConnection object at 0x7fd5cead31f0>: Failed to establish a new connection: [Errno 111] Connection refused\'))", "error_traceback": "Traceback (most recent call last):\\n File \\"/usr/local/lib/python3.10/dist-packages/urllib3/connection.py\\", line 174, in _new_conn\\n conn = connection.create_connection(\\n File \\"/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py\\", line 95, in create_connection\\n raise err\\n File \\"/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py\\", line 85, in create_connection\\n sock.connect(sa)\\nConnectionRefusedError: [Errno 111] Connection refused\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n File \\"/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py\\", line 715, in urlopen\\n httplib_response = self._make_request(\\n File \\"/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py\\", line 416, in _make_request\\n conn.request(method, url, **httplib_request_kw)\\n File \\"/usr/local/lib/python3.10/dist-packages/urllib3/connection.py\\", line 244, in request\\n super(HTTPConnection, self).request(method, url, body=body, headers=headers)\\n File \\"/usr/lib/python3.10/http/client.py\\", line 1283, in request\\n self._send_request(method, url, body, headers, encode_chunked)\\n File \\"/usr/lib/python3.10/http/client.py\\", line 1329, in _send_request\\n self.endheaders(body, encode_chunked=encode_chunked)\\n File \\"/usr/lib/python3.10/http/client.py\\", line 1278, in endheaders\\n self._send_output(message_body, encode_chunked=encode_chunked)\\n File \\"/usr/lib/python3.10/http/client.py\\", line 1038, in _send_output\\n self.send(msg)\\n File \\"/usr/lib/No idea why it works at first and then doesn't..
TimPietrusky
Describe the bug
I'm testing the API locally before deploying to Runpod. I'm testing on a 4070 Super. When I make a single call to /runsync it will complete without fail every time.. and do a really nice job of it. But if I loop let's say 10 requests, it will always eventually stall. There is no more output in the terminal, and the fans keep on spinning.. it just gets stuck.. one would guess there might be some kind of memory leak. Or maybe it tries to load the same models again and again and the memory runs out. It's rather hard to debug I guess. This doesn't seem to happen if I'm generating small images but when they are larger images that take longer to generate, it happens without fail.
I should add that I'm using the same checkpoint, and doing the same operations in my loop, so it's not like I'm requesting it to load a different model repeatedly.
Is there a way to force a memory clean in between generations.. or maybe run with a higher level of verbosity?