Closed brianoflondon closed 2 years ago
I don't think the case which is tested by test_#48 is covering this retry problem. It also causes the writer to dump out and potentially loose one or more of the IRIs which have been sent to write out.
I can see this with my 3pseak system because I have a backup server which watches to see if any given podping actually makes it to the chain. When one or more fail to be written, the backup server acts. That's what happend with this crash I believe.
I don't think the case which is tested by test_#48 is covering this retry problem. It also causes the writer to dump out and potentially loose one or more of the IRIs which have been sent to write out.
I can see this with my 3pseak system because I have a backup server which watches to see if any given podping actually makes it to the chain. When one or more fail to be written, the backup server acts. That's what happend with this crash I believe.
I don't think the case which is tested by test_#48 is covering this retry problem. It also causes the writer to dump out and potentially loose one or more of the IRIs which have been sent to write out.
I can see this with my 3pseak system because I have a backup server which watches to see if any given podping actually makes it to the chain. When one or more fail to be written, the backup server acts. That's what happend with this crash I believe.
I didn’t realize an IRI was being lost here when this happens. Good catch Brian.
I've built a version of this including my fix and I'm running it on my two 3speak systems so I'll watch it for a few days and see if this crops up gain.
Can you please test the version in this branch? I made a slight change.
https://github.com/Podcastindex-org/podping-hivewriter/tree/hotfix/1.2.5
If I can't catch the memory leak tonight, I'll publish this version as-is.
Also for what it's worth, I knew this was an issue when updating the hive-watcher for @daveajones, but didn't realize we were handling it differently.
I'm now running a rebuilt docker from the /hotfix/1.2.5-alpha.1 branch which I made. Git gave me all sorts of Detached HEAD complaints with your hotfix/1.2.5 tag.
The only errors I consistently see are these:
podping-3spk | 2022-08-07T02:08:10+0000 | INFO | Lighthive Fastest: https://hive-api.arcange.eu
podping-3spk | 2022-08-07T02:08:10+0000 | INFO | Status - Uptime: 15:49:29 | IRIs Received: 123 | IRIs Deduped: 123 | IRIs Sent: 123 | last_node: https://hive-api.arcange.eu
podping-3spk | Exception ignored in: weakref callback <function WeakKeyDictionary.__init__.<locals>.remove at 0x000000000523a520>
podping-3spk | Traceback (most recent call last):
podping-3spk | File "/opt/pypy/lib/pypy3.8/weakref.py", line 377, in remove
podping-3spk | del self.data[k]
podping-3spk | KeyError: <weakref at 0x0000000001bc1cc0; dead>
podping-3spk | Exception ignored in: weakref callback <function WeakKeyDictionary.__init__.<locals>.remove at 0x000000000523a520>
podping-3spk | Traceback (most recent call last):
podping-3spk | File "/opt/pypy/lib/pypy3.8/weakref.py", line 377, in remove
podping-3spk | del self.data[k]
podping-3spk | KeyError: <weakref at 0x0000000002272c00; dead>
podping-3spk | Exception ignored in: weakref callback <function WeakKeyDictionary.__init__.<locals>.remove at 0x000000000523a520>
podping-3spk | Traceback (most recent call last):
podping-3spk | File "/opt/pypy/lib/pypy3.8/weakref.py", line 377, in remove
podping-3spk | del self.data[k]
podping-3spk | KeyError: <weakref at 0x0000000002272c40; dead>
podping-3spk | 2022-08-07T02:11:11+0000 | INFO | Lighthive Fastest: https://api.deathwing.me
Git gave me all sorts of Detached HEAD complaints with your hotfix/1.2.5 tag.
All that means is you didn't checkout the branch correctly.
Git gave me all sorts of Detached HEAD complaints with your hotfix/1.2.5 tag.
All that means is you didn't checkout the branch correctly.
I have a PhD in not doing things properly with git.
This is what I get when I try to checkout your branch
This error... which I think stems from a non standard error response from one of the Hive RPC nodes (possibly rpc.ausbit.dev) I think can be fixed.