Open dmugtasimov opened 3 years ago
After running dockerized unittests some celery tasks remain unprocessed that lead to errors in subsequent run of celery:
[2021-02-27 12:46:31,767: INFO/MainProcess] Received task: thenewboston_bank.tasks.blocks.send_signed_block[5c1d7a70-3715-4fb8-988a-9360d3efaa2c] [2021-02-27 12:46:31,768: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f64f4374280> (args:('thenewboston_bank.tasks.blocks.send_signed_block', '5c1d7a70-3715-4fb8-988a-9360d3efaa2c', {'lang': 'py', 'task': 'thenewboston_bank.tasks.blocks.send_signed_block', 'id': '5c1d7a70-3715-4fb8-988a-9360d3efaa2c', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '5c1d7a70-3715-4fb8-988a-9360d3efaa2c', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'block': {'account_number': '03bee52becfde71aa0c4373d3a80eba488edd15714abd9eb3e6239ab99cae4aa', 'message': {'balance_key': '03bee52becfde71aa0c4373d3a80eba488edd15714abd9eb3e6239ab99cae4aa', 'txs': [...]}, 'signature': '829e77e0bce6c6135ad118cfc9d9c39a5739201164c1fa9ad4905922cf55daa4e38ee1ad15aa15ded03f3eb57ecebaf286606029fd208ca56a36df40476fb00a'}, 'ip_address': '192.168.1.74', 'port': 8000, 'protocol': 'http', 'url_path': '/bank_blocks'}", 'origin': 'gen17@bb70df0745f1', 'reply_to': 'c00a13f9-c3e6-3c26-81ae-915bd29b16de', 'correlation_id':... kwargs:{}) [2021-02-27 12:46:31,770: INFO/MainProcess] Received task: thenewboston_bank.tasks.sync.set_primary_validator[fd539fa1-7699-4cec-b133-3b4c281b13af] [2021-02-27 12:46:31,771: DEBUG/MainProcess] Task accepted: thenewboston_bank.tasks.blocks.send_signed_block[5c1d7a70-3715-4fb8-988a-9360d3efaa2c] pid:15629 [2021-02-27 12:46:31,774: INFO/MainProcess] Received task: thenewboston_bank.tasks.sync.set_primary_validator[4d8e6bf5-6919-4882-9830-54d5cd478382] [2021-02-27 12:46:31,776: DEBUG/ForkPoolWorker-1] Starting new HTTP connection (1): 192.168.1.74:8000 [2021-02-27 12:46:34,825: INFO/MainProcess] Received task: thenewboston_bank.tasks.sync.set_primary_validator[c93ce199-8325-45bb-985f-bb63a3347572] [2021-02-27 12:46:34,823: ERROR/ForkPoolWorker-1] HTTPConnectionPool(host='192.168.1.74', port=8000): Max retries exceeded with url: /bank_blocks (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f64ec54d520>: Failed to establish a new connection: [Errno 113] No route to host')) Traceback (most recent call last): File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn conn = connection.create_connection( File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection raise err File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection sock.connect(sa) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen httplib_response = self._make_request( File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request conn.request(method, url, **httplib_request_kw) File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request super(HTTPConnection, self).request(method, url, body=body, headers=headers) File "/home/dmugtasimov/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked) File "/home/dmugtasimov/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/home/dmugtasimov/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/home/dmugtasimov/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 1010, in _send_output self.send(msg) File "/home/dmugtasimov/.pyenv/versions/3.9.2/lib/python3.9/http/client.py", line 950, in send self.connect() File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect conn = self._new_conn() File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f64ec54d520>: Failed to establish a new connection: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 439, in send resp = conn.urlopen( File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen retries = retries.increment( File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 573, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.1.74', port=8000): Max retries exceeded with url: /bank_blocks (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f64ec54d520>: Failed to establish a new connection: [Errno 113] No route to host')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/dmugtasimov/gitrep/thenewboston/Bank/thenewboston_bank/tasks/blocks.py", line 47, in send_signed_block post(url=url, body=signed_block) File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/thenewboston/utils/network.py", line 20, in post response = requests.post(url, json=body) File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/requests/api.py", line 119, in post return request('post', url, data=data, json=json, **kwargs) File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 542, in request resp = self.send(prep, **send_kwargs) File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 655, in send r = adapter.send(request, **kwargs) File "/home/dmugtasimov/.virtualenvs/thenewboston-bank-r7f_GShq-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.1.74', port=8000): Max retries exceeded with url: /bank_blocks (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f64ec54d520>: Failed to establish a new connection: [Errno 113] No route to host')) [2021-02-27 12:46:34,829: INFO/ForkPoolWorker-1] Task thenewboston_bank.tasks.blocks.send_signed_block[5c1d7a70-3715-4fb8-988a-9360d3efaa2c] succeeded in 3.0597239730000183s: None [2021-02-27 12:46:34,829: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f64f4374280> (args:('thenewboston_bank.tasks.sync.set_primary_validator', 'fd539fa1-7699-4cec-b133-3b4c281b13af', {'lang': 'py', 'task': 'thenewboston_bank.tasks.sync.set_primary_validator', 'id': 'fd539fa1-7699-4cec-b133-3b4c281b13af', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'fd539fa1-7699-4cec-b133-3b4c281b13af', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': '{}', 'origin': 'gen8@bb70df0745f1', 'reply_to': '2b1788a2-bad6-34ad-b975-8c60de7293cd', 'correlation_id': 'fd539fa1-7699-4cec-b133-3b4c281b13af', 'hostname': 'celery@dmugtasimov-Latitude-E5550', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'args': [], 'kwargs': {}}, b'[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{}) [2021-02-27 12:46:34,830: DEBUG/MainProcess] Task accepted: thenewboston_bank.tasks.sync.set_primary_validator[fd539fa1-7699-4cec-b133-3b4c281b13af] pid:15629 [2021-02-27 12:46:34,846: INFO/ForkPoolWorker-1] Task thenewboston_bank.tasks.sync.set_primary_validator[fd539fa1-7699-4cec-b133-3b4c281b13af] succeeded in 0.01564732399992863s: None [2021-02-27 12:46:34,847: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f64f4374280> (args:('thenewboston_bank.tasks.sync.set_primary_validator', '4d8e6bf5-6919-4882-9830-54d5cd478382', {'lang': 'py', 'task': 'thenewboston_bank.tasks.sync.set_primary_validator', 'id': '4d8e6bf5-6919-4882-9830-54d5cd478382', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '4d8e6bf5-6919-4882-9830-54d5cd478382', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': '{}', 'origin': 'gen14@bb70df0745f1', 'reply_to': 'e365b7e7-b086-3648-9c4d-33dca44b5c65', 'correlation_id': '4d8e6bf5-6919-4882-9830-54d5cd478382', 'hostname': 'celery@dmugtasimov-Latitude-E5550', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'args': [], 'kwargs': {}}, b'[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{}) [2021-02-27 12:46:34,848: DEBUG/MainProcess] Task accepted: thenewboston_bank.tasks.sync.set_primary_validator[4d8e6bf5-6919-4882-9830-54d5cd478382] pid:15629 [2021-02-27 12:46:34,863: INFO/ForkPoolWorker-1] Task thenewboston_bank.tasks.sync.set_primary_validator[4d8e6bf5-6919-4882-9830-54d5cd478382] succeeded in 0.015120853999633255s: None [2021-02-27 12:46:34,864: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f64f4374280> (args:('thenewboston_bank.tasks.sync.set_primary_validator', 'c93ce199-8325-45bb-985f-bb63a3347572', {'lang': 'py', 'task': 'thenewboston_bank.tasks.sync.set_primary_validator', 'id': 'c93ce199-8325-45bb-985f-bb63a3347572', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '5c1d7a70-3715-4fb8-988a-9360d3efaa2c', 'parent_id': '5c1d7a70-3715-4fb8-988a-9360d3efaa2c', 'argsrepr': '()', 'kwargsrepr': '{}', 'origin': 'gen15629@dmugtasimov-Latitude-E5550', 'reply_to': '3907e8bb-ff87-3187-8c86-da9c4f709f54', 'correlation_id': 'c93ce199-8325-45bb-985f-bb63a3347572', 'hostname': 'celery@dmugtasimov-Latitude-E5550', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'args': [], 'kwargs': {}}, b'[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{}) [2021-02-27 12:46:34,866: DEBUG/MainProcess] Task accepted: thenewboston_bank.tasks.sync.set_primary_validator[c93ce199-8325-45bb-985f-bb63a3347572] pid:15629 [2021-02-27 12:46:34,880: INFO/ForkPoolWorker-1] Task thenewboston_bank.tasks.sync.set_primary_validator[c93ce199-8325-45bb-985f-bb63a3347572] succeeded in 0.014641084999766463s: None
To reproduce:
make test-dockerized
make run-celery
After running dockerized unittests some celery tasks remain unprocessed that lead to errors in subsequent run of celery:
To reproduce:
make test-dockerizedmake run-celery