error: ChannelReceiveError, sender is disconnected: receiving on a closed channel

[sameh-farouk]

I tried to push 250 messages to Redis, I expected that this would take a bit longer to process but would work, or even in the worst case some requests would time out if the expiration set too low. but still, 250 messages are not too much for the default 1000 workers to handle.

What happens is that about a quarter of the total messages failed early/immediately, a response was pushed to the return queue with err: receive error reply from remote RMB:twin with id 41 not found`.

this happens even if the twin exists in the cache.

this is also reproducible.

the same test succeeded with RMB-go even with 1000 messages.

in the RMB debug log:

2022-06-01T19:13:38.632Z DEBUG [rmb_rs::http_workers::work_runner] can not retrieve twin from substrate for this id '41' because of ChannelReceiveError, sender is disconnected: receiving on a closed channel
Encountered an error: uncategorized error
Enable a logger to see more information.

how to reproduce: you can try any script to push 250+ (when you raise the number of messages the failure rate increase) messages to Redis, I have a tool used before in testing RMB-v.

1- I started the RMB daemon with default args

rmb-rs --key-type sr25519 -s "wss://tfchain.dev.grid.tf" -d -m "<MNEMONICS>"

2- Started a process to handle the messages

python3 msg_handler.py &

3- Started a process to send multiple messages to my twin

python3 rmb_tester.py --dest 41 --timeout 120 --count 250

[sameh-farouk]

this now partially fixed after this commit 98e38c0e69fe81ce09c5fe0505521c1298fc054c. but still need some optimization to prevent the failure of delivery that happens due to the issue in fetching the twin id from the substrate when there are a lot of queries.

[sameh-farouk]

my suggestion

• n workers will try at the same time to fetch one, or multiple twins.
  • some workers succeed to fetch the twin (cache updated) and some will receive an error because of the mentioned limitation.
• now workers that received an error instead of returning the error would implement a simple pause (random small amount of time to prevent all paused workers from talking/sending again at the same time) and then will try again to fetch the twin, should get it from cache this time (and in case of multiple twins will it will fetch from the substrate and likely succeeded this time)