add abiltity to configure error output

[ekneg54]

This changes the error handling implementation of logprep. This PR aims at two main goals.

1. no error should permit further processing

This is handled by not raising FatalOutputError or FatalInputError but instead raise a CriticalInputError or a CriticalOutputError. No error is handled in the pipeline process anymore all error causing events are written to error output. To achieve this, I had to change the batch_finished_callback mechanic. As now every event gotten from input via get_next is committed to kafka utilizing the batch_finished_callback mechanic in the pipeline.py. no connections between intput and output connectors anymore.

3. no event should get lost

to make it simple. This PR has to ensure, that every event goes into output, error output or gets logged to console as last resort.

Every error raising event will be serialized together with its raising error to an error event and is put into a multiprocessing.Queue (ThrottlingQueue). In the main Thread these events were handled in a configured error output connector which indeed can be any output connector implemented in logprep.

To achieve theses goals I had to reimplement the opensearch output connector to simplify things a lot.

Please have a look on my changes and lets discuss. Feel free to give feedback and to ask your questions. It is a very big PR. Sorry for that but the cut was a fundamental one.

[ekneg54]

additional work:

• [x] add error output to health checks
• [x] add metric for error output queue
• [x] add option in helm chart to configure error_output
• [x] inspect behavior on high error count situations -> it looks that the error_output is dying and not restartet
• [x] what happens if the error_queue is full? if the error queue is full, the put in pipeline is blocking further processing, but before this happens, the processing is throttled by the throttling queue
• [x] fix failed events metric
• [x] is putting an event to the error queue an error? (versus failed event ) yes, because the counts can be different because of batched outputs causing many failed events with one critcal_output_error
• [x] kubernetes kills pod before error queue is empty grace period was increased to 300s in helm charts. this carries the shutdown behavior in connection with smooth error_backlog_size config options of about 1000. this works in our setup.
• [x] kafka input is only processing half the topics with 1 process (perhaps a client is not closed but dereferenced during setup)
• [ ] inspect performance regression
• [x] inspect memeory leak

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 91.03448% with 26 lines in your changes missing coverage. Please review.

Project coverage is 94.09%. Comparing base (eca9240) to head (1e150eb). Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
logprep/connector/opensearch/output.py	72.50%	11 Missing :warning:
logprep/framework/pipeline_manager.py	91.20%	8 Missing :warning:
logprep/framework/pipeline.py	90.90%	7 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #668 +/- ## ========================================== - Coverage 94.27% 94.09% -0.18% ========================================== Files 146 146 Lines 9777 9914 +137 ========================================== + Hits 9217 9329 +112 - Misses 560 585 +25 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.