search

Podcastindex-org / podping-hivewriter

The hive writer component of podping.
MIT License
15 stars 5 forks source link

51 non standard rpc error response from hive rpc node #52

Closed brianoflondon closed 2 years ago

brianoflondon commented 2 years ago

@agates I monitor my logs and I think something I fixed before... needed more fixing.

agates commented 2 years ago

This PR doesn't make any code changes, what am I missing?

agates commented 2 years ago

Oh. You already pushed the code change to develop. Please don't do that.

brianoflondon commented 2 years ago

Oh. You already pushed the code change to develop. Please don't do that.

I kinda messed up and tried to reverse that commit and then put my changes in a branch.

brianoflondon commented 2 years ago

Success... this is the only error I've had for days and the code dealt with it properly:

2022-08-13T13:02:55+0000 | INFO | Lighthive Fastest: https://rpc.ecency.com
2022-08-13T13:02:55+0000 | INFO | Status - Uptime: 7:02:49 | IRIs Received: 46 | IRIs Deduped: 46 | IRIs Sent: 46 | last_node: https://rpc.ecency.com
2022-08-13T13:04:59+0000 | INFO | IRI batch_id 2f810e23-3cd3-43ee-8d20-0254a320f525 - Size of IRIs: 36
2022-08-13T13:04:59+0000 | INFO | Received 1 IRIs
2022-08-13T13:05:03+0000 | INFO | Backing off _send_request(...) for 0.9s (requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='rpc.ecency.com', port=443): Read timed out. (read timeout=3))
2022-08-13T13:05:40+0000 | INFO | Backing off _send_request(...) for 0.1s (requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='rpc.ecency.com', port=443): Read timed out. (read timeout=3))
2022-08-13T13:05:41+0000 | ERROR | send_notification error: transaction expiration exception:
2022-08-13T13:05:41+0000 | ERROR | Failed to send 1 IRIs
Traceback (most recent call last):
  File "/home/podping/app/src/podping_hivewriter/cli/podping.py", line 245, in server
    loop = asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 482, in failure_retry
    await self.send_notification_iris(
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 456, in send_notification_iris
    await self.send_notification(payload.dict(), hive_operation_id)
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 416, in send_notification
    raise ex
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 401, in send_notification
    await self._async_hive_broadcast(op=op, dry_run=self.dry_run)
  File "/home/podping/app/src/podping_hivewriter/async_wrapper.py", line 32, in wrapper
    return await async_fn(*args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/asgiref/sync.py", line 435, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/asgiref/sync.py", line 474, in thread_handler
    return func(*args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 184, in broadcast
    return self.transaction_builder.broadcast(
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/broadcast/transaction_builder.py", line 202, in broadcast
    resp = self.client('condenser_api').broadcast_transaction(
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 53, in callable
    return self.request(attr, *args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 152, in request
    self.validate_response(response)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 164, in validate_response
    raise RPCNodeException(
lighthive.exceptions.RPCNodeException: transaction expiration exception:
2022-08-13T13:05:41+0000 | ERROR | transaction expiration exception:
Traceback (most recent call last):
  File "/home/podping/app/src/podping_hivewriter/cli/podping.py", line 245, in server
    loop = asyncio.get_running_loop()
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 482, in failure_retry
    await self.send_notification_iris(
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 456, in send_notification_iris
    await self.send_notification(payload.dict(), hive_operation_id)
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 416, in send_notification
    raise ex
  File "/home/podping/app/src/podping_hivewriter/podping_hivewriter.py", line 401, in send_notification
    await self._async_hive_broadcast(op=op, dry_run=self.dry_run)
  File "/home/podping/app/src/podping_hivewriter/async_wrapper.py", line 32, in wrapper
    return await async_fn(*args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/asgiref/sync.py", line 435, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/asgiref/sync.py", line 474, in thread_handler
    return func(*args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 184, in broadcast
    return self.transaction_builder.broadcast(
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/broadcast/transaction_builder.py", line 202, in broadcast
    resp = self.client('condenser_api').broadcast_transaction(
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 53, in callable
    return self.request(attr, *args, **kwargs)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 152, in request
    self.validate_response(response)
  File "/home/podping/app/.venv/lib/python3.10/site-packages/lighthive/client.py", line 164, in validate_response
    raise RPCNodeException(
lighthive.exceptions.RPCNodeException: transaction expiration exception:
2022-08-13T13:05:41+0000 | WARNING | Waiting 3s before retry
2022-08-13T13:05:44+0000 | INFO | FAILURE COUNT: 1 - RETRYING 1 IRIs
2022-08-13T13:05:55+0000 | INFO | Lighthive Node: https://rpc.ausbit.dev
2022-08-13T13:05:55+0000 | INFO | JSON size: 96
2022-08-13T13:05:55+0000 | INFO | FAILURE CLEARED after 1 retries, 3s
daveajones commented 2 years ago

If the exception is being caught and handled gracefully, is there any way to clean that up and make it less of a huge log barf?

agates commented 2 years ago

If the exception is being caught and handled gracefully, is there any way to clean that up and make it less of a huge log barf?

I think we can stop logging known exception tracebacks now that we know what's working.