search

SigNoz / signoz

SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
https://signoz.io
Other
18.89k stars 1.23k forks source link

Option in web front end to reprocess log data using updated receiver/processor/etc configurations #4719

Open bwhitehead0 opened 7 months ago

bwhitehead0 commented 7 months ago

Is your feature request related to a problem?

When you preprocess logs, for example, to update a receiver to add multiline log support to the logs, there is no easy way to reprocess the previously ingested log data to reflect the updated preprocessing.

Describe the solution you'd like

An option within the web front end to re-apply preprocessing to specific defined logs (probably contingent on the ability to specify defined log groups within the front end in order to not bog down processing systems?).

Describe alternatives you've considered

According to @nityanandagohain in slack support chat, presently the only way to do this is to re-ingest the data from scratch, or execute SQL updates manually.

Additional context

When updating log ingestion, preprocessing, etc, it's possible to not know ahead of time that there might be scenarios requiring new preprocessing, for example if it's discovered that sensitive data is written to logs occasionally, etc. This requires modifying log processing, and there should be a way to apply the processing to previously ingested logs.

Thank you for your feature request – we love each and every one!

welcome[bot] commented 7 months ago

Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.