search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
61 stars 3.5k forks source link

Instrument core and plugins with Elastic APM #13460

Open jsvd opened 2 years ago

jsvd commented 2 years ago

Currently Logstash doesn't have a good way to understand the latency introduced while processing events.

The Elastic APM agent could be leveraged to measure how events progress through the pipelines. The creation of an event would start a trace that would end after the pipeline worker finished processing the batch that contains said event. Each step, namely passing through plugins and going in/out of the queue, would constitute spans.

the implementation can be done in multiple ways but it could target the base classes of plugins. The Elastic Observability solution provides APM agents in both Ruby and Java so either could be used if the target is the Java core classes or Ruby plugins.

joshdover commented 2 years ago

@elastic/logstash has anyone ever tried seeing if we get anything useful with the auto-attach jar instrumentation? You can deploy standalone via a jar: https://www.elastic.co/guide/en/apm/agent/java/current/setup-attach-cli.html

You can download it and run it like this:

wget https://search.maven.org/remotecontent?filepath=co/elastic/apm/apm-agent-attach-cli/1.34.1/apm-agent-attach-cli-1.34.1.jar 
java -jar apm-agent-attach-cli-1.34.1.jar \
    --exclude-user root \
    --include-main MyApplication my-application.jar \
    --include-vmarg elastic.apm.attach=true \
    --continuous \
    --config service_name=my-cool-service \
    --config server_url=http://localhost:8200