Crash dump delivery [WIP]

biinilya commented 3 years ago

Introduction

Due to the distributed nature of the spacemeshos, we expect a strong issue with node failures on a remote hardware of our endusers. Regarding this we want to improve the process of crash feedback.

User scenario

Node crash while running or bootstrapping. The user is willing to receive support from the core team, but the \re is no good tool for that.

Problem

The crash destroys the whole node process providing some goroutine stacktrace, sometimes damaged.
This crash info is not normalized, it can easily exhaust the user terminal buffer and thus make it impossible to send.
The crash info does not have hardware and config context attached, thus providing an almost non-reproducible dump.

Solution

Hook a system signal, implement a handler that will generate a crash dump including all the contextual information and dump it to a proprietary binary file.
Include a cli tool to upload all the recent crash dumps.
Implement a server part which will automatically sort out the issues to known/unknown and probably create a GitHub issue.

biinilya commented 3 years ago

spacemeshos/go-spacemesh#2277

antonlerner commented 3 years ago

Also, zip log files into crash dump file

biinilya commented 3 years ago

Also, zip log files into crash dump file

I am thinking to integrate this into the binary as well as other contextual information such as goroutines.

spacemeshos / SMIPS