search

dragonflydb / dragonfly

A modern replacement for Redis and Memcached
https://www.dragonflydb.io/
Other
25.5k stars 931 forks source link

investigate minicore dumper integration with dragonfly #3902

Open romange opened 3 hours ago

romange commented 3 hours ago

Corefiles are important so we could iterate and fix datarace/segfault bugs

Unfortunately with in-memory stores, linux-created corefiles can reach dozens, hundreds of gigabytes.

I would like to explore two alternatives:

  1. https://github.com/Percona-Lab/coredumper/
  2. https://github.com/google/breakpad/

These are the challenges I see with coredumpers:

  1. ARM64 support is a must
  2. Fibers support is a must. Our fibers allocate their stack on heap and it's not immediately clear if any of the coredumpers will work out of the box.

The goal is to investigate what can be the simplest and the least intrusive way to integrate mini-coredumper into Dragonfly.

romange commented 2 hours ago

Also, there is this one: https://github.com/diamon/minicoredumper

Its library is licensed under https://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html but it seems that LGPL allows us linking the library and keep the BUSD license we have in place.

romange commented 2 hours ago

Another caveat: we saw problems with using a piped /proc/sys/kernel/core_pattern when a proactor thread crashes. In other words when a kill signal is sent to a ProactorXX thread and not a "main" thread. We have not investigated it further.

romange commented 2 hours ago

The last one looks promising: https://tracingsummit.org/ts/2016/files/Ogness_minicoredumper.pdf