Centralized error collection

[dswd]

We should have a central component that collects and handles all error reports. The components should push their errors to this component and only store them until they are forwarded.

What is needed for that:

• [x] A separate service dedicated to collecting errors
• [X] A standardized error representation containing:
  • Error code, error message, timestamp, ToMaTo component
  • Additional data
  • Stack trace, environment data
• [X] A way to aggregate similar errors into groups
• [x] A library for reporting errors and submitting them to the collector

Important notes / Things to care about:

• Fast error dumping: Errors should be dumped quickly with no RPC calls and only minimal IO involved, so that the caller does not have to wait
  • [x] On error: quickly store error information in RAM and on disk
  • [x] In a separate thread: send errors to collector service and remove them from storage
  • [x] What about environment information that requires external calls?
• Repeated/periodic errors: Errors that happen often should not spam the error collection.
  • [x] Similar errors are aggregated
  • [ ] The hash based grouping can be improved by removing pre-configured patterns from the data
• Errors inside error handling: Errors in error handling should be treated gracefully
  • [ ] Both errors should be dumped if possible
  • [ ] Avoid recursive error dumping
• Errors inside tight loop: Errors that occur in a tight loop, i.e. appear massively should be ignored
  • [ ] Do not report errors when to-be-submitted-queue is full (report this incident as an error once)
• Massive failure: When a service is failing due to out-of-memory, full disk, or io errors lots of different calls will fail. This should be detected and handled gracefully.
  • [ ] Detect common basic problems and report them instead of the errors
  • [ ] Allow admins to remove all errors that only exist on one specific source

[dswd]

Ok, this got more important right now. The recent power outage has shown that the current way to deal with error dumps is not viable:

• Many hosts have over 14k dumps for repeating errors like unreachable database or state mismatches
• The backend core has over 360k dumps (1.4GB). Deleting all of that took several minutes.
• The backend core kept on eating memory until it crashed with 10GB of RAM taken. I guess it was trying to keep all dumps in memory.

This issue should be the very next thing we take care about.

[a-gerhard]

I have actually implemented a limit for error dumps, it's just not yet deployed on master... https://github.com/GLab/ToMaTo/pull/1429

[dswd]

I took the opportunity and updated the master right after the power outage. So maybe it won't happen again.