search

deso-protocol / run

Run your own DeSo node
https://docs.deso.org
190 stars 94 forks source link

Backend service crashes after "Peer.outHandler: Problem sending message to peer" #67

Closed MirekR closed 3 years ago

MirekR commented 3 years ago

Hi, we've been experience strange behavior of the node - it seems to be loosing connection to one of it's peers resulting in crash of the backend container - see the logs attached. Not sure if that's related but also from time to time we can see huge consumption of the memory by the backend (up 99.5%) causing unresponsiveness of the system.

Dockers are run in the Google Cloud VM, 6CPU and 32BG failed.log

maebeam commented 3 years ago

Are you running out of disk space? I'm seeing a lot of weird errors in that log

MirekR commented 3 years ago

At the moment, disk usage is at 88%, meaning 8.5GB is still free.

maebeam commented 3 years ago

I'd recommend increasing your disk size and also making sure ulimit -n is set to a number higher than default. Please let me know if that doesn't help and I'm happy to reopen this issue.

MirekR commented 3 years ago

Hi, it looks like server is not falling over after increasing disk quota but unfortunately doesn't behave correctly either. It felt out of sync with the logs attached. It almost feels these two issues are connected as you can see mention of lost connections. Do you think it's setup or there is fundamental issue with the backend server and how it handles disconnecting servers / how it selecting peers?

no-sync.log

maebeam commented 3 years ago

Can you provide the complete log output? Something else failed earlier up in the logs

MirekR commented 3 years ago

Unfortunately I purged the server to get it working again. I'll get full logs when it happens next time.