Open Kailai-Wang opened 1 year ago
Hmm, tbh we don't have a long-running instance currently. I can't give a constructive answer here. Does the worker recover from that?
Not really - it still produces blocks, but only once in a while.
Or to put it in another way, is there any hope that we get the aforementioned issue (https://github.com/integritee-network/worker/issues/1275) implemented soon? 🙌
Hmm, I have one guess. Maybe it is because we don't prune the parentchain headers in the light-client-db, which are all loaded in memory. This could slowly deteriorate the performance over time.
I am currently implementing https://github.com/integritee-network/worker/issues/1370, which introduces pruning of the parentchain headers. If we are lucky, this solves your issue.
And I have to admit that we are currently not planning to implement #1275 in the near future.
That's a good guess - we have over 385,000
sidechain blocks and the light_client.db
is now 20MB. Pruning it would definitely help!
might be related in case you want to remove it too: https://github.com/integritee-network/worker/issues/1281 - we only removed the check locally, however it doesn't prevent it from growing as the parentchain extrinsic is being added constantly
I believe the majority of the data in light_client.bin
are the hashes of extrinsics sent to parentchain. They accumulate without being pruned.
Seems, we are on the right track! :D Then the issue should be solved by next week or the week after on our end. 👍
Any update on this? :)
I did have to tend to some unforeseen things regarding our slot swap, which have occupied me until yesterday, and next week I am off. But the week after, I can definitely finish. ETA of remaining working hours ~8-10h.
I might not have fully resolved this issue in #1399; however, there is a follow-up issue in #1404, which I tend to tackle as soon as I have some feedback on my suggestion, which should resolve this issue..
We have observed that sometimes the sidechain produces blocks with a low chance. It means most of the aura slots end up with "Discarding proposal" due to "too long block production time".
However, neither the parachain nor the sidechain has a lot of traffic - they are quite idle. We have only one tee-worker node, so nothing related inter-node communication.
I don't know how it could happen - I could only impute it to the miscalculation of aura slot, which means the worker grabs the aura slot way too late.
Maybe it's related to https://github.com/integritee-network/worker/issues/1275? Or is there anyway to further dissect the problem to find out the root cause?