search

fstab / grok_exporter

Export Prometheus metrics from arbitrary unstructured log data.
Apache License 2.0
898 stars 152 forks source link

When this GA version of 1.0.0 will be released #149

Open cawamata opened 3 years ago

cawamata commented 3 years ago

From tag info, 1.0.0.RC5 was released on 22th on last September. Do you have any fatal issues to release 1.0.0 version. If exist, can you list up ? if I can help, I will do.

fstab commented 3 years ago

Hi @cawamata, thanks for offering your help. I agree, it's time to get 1.0.0 done. There is no missing feature in my opinion, the only thing holding me back is that I wanted to improve self monitoring. There were some rare issues on memory or cpu consumption. This is not necessarily a bug, because you can easily write a couple of regular expressions that use a huge amount of cpu, and delayed processing will increase memory usage as the buffer fills. However, it's currently hard to find our whether it is caused by a regular expression gone wild or whether it is a bug. Therefore I would like to improve the build-in metrics so that users can easily see where cpu time is spent, etc.

I just pushed my current state to the built-in-metrics-improvement branch. Metrics are in selfMonitoring.go. I want to set up a long running load test, scripts for setting this up are in hack/loadtest/. The next steps are to improve the hack/loadtest/load-generator.go to create significant load, to improve hack/loadtest/config.yaml to have reasonable matches for the load test, and to provide metrics such that it's easy to see where the bottlenecks are. Moreover, some additional metrics would be great, for example how often each log file was rotated, etc.

If you have time to help it will be highly appreciated :)