search

hypertrace / javaagent

Hypertrace OpenTelemetry Java agent with payload/body and headers data capture.
Apache License 2.0
33 stars 15 forks source link

Startup takes around 30s #212

Open pavolloffay opened 3 years ago

pavolloffay commented 3 years ago

Startup in api-anomaly-detector service in Traceable takes around 30s at least based on the agent logs

Picked up JAVA_TOOL_OPTIONS: -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0 -javaagent:/mnt/hypertrace/hypertrace-agent-all.jar
[main] INFO org.hypertrace.agent.core.config.HypertraceConfig - Config loaded: {"serviceName":"api-anomaly-detector","reporting":{"endpoint":"http://collector:9411/api/v2/spans","opa":{"endpoint":"http://aaa:8181/","pollPeriodSeconds":30}},"dataCapture":{"httpHeaders":{"request":true,"response":true},"httpBody":{"request":true,"response":true},"rpcMetadata":{"request":true,"response":true},"rpcBody":{"request":true,"response":true},"bodyMaxSizeBytes":131072},"propagationFormats":["TRACECONTEXT"]}
[opentelemetry.auto.trace 2021-01-09 06:41:22:455 +0000] [main] INFO org.hypertrace.agent.core.config.HypertraceConfig - Config loaded: {"serviceName":"api-anomaly-detector","reporting":{"endpoint":"http://collector.aaa:9411/api/v2/spans","opa":{"endpoint":"http://aaa:8181/","pollPeriodSeconds":30}},"dataCapture":{"httpHeaders":{"request":true,"response":true},"httpBody":{"request":true,"response":true},"rpcMetadata":{"request":true,"response":true},"rpcBody":{"request":true,"response":true},"bodyMaxSizeBytes":131072},"propagationFormats":["TRACECONTEXT"]}
[opentelemetry.auto.trace 2021-01-09 06:41:41:350 +0000] [main] INFO io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller - Installed span exporter: io.opentelemetry.exporter.zipkin.ZipkinSpanExporter
[opentelemetry.auto.trace 2021-01-09 06:41:41:562 +0000] [main] INFO io.opentelemetry.javaagent.tooling.PropagatorsInitializer - Added tracecontext propagator
Hypertrace agent started, version: 0.7.5
2021-01-09 06:41:51.959 [main] INFO  o.h.c.s.PlatformServiceLauncher - Trying to start PlatformService ...
pavolloffay commented 3 years ago

Here are log from the query service 44s:

Picked up JAVA_TOOL_OPTIONS: -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0 -javaagent:/mnt/hypertrace/hypertrace-agent-all.jar
[main] INFO org.hypertrace.agent.core.config.HypertraceConfig - Config loaded: {"serviceName":"query-service","reporting":{"endpoint":"http://collector.traceableai:9411/api/v2/spans","opa":{"endpoint":"http://opa.traceableai:8181/","pollPeriodSeconds":30}},"dataCapture":{"httpHeaders":{"request":true,"response":true},"httpBody":{"request":true,"response":true},"rpcMetadata":{"request":true,"response":true},"rpcBody":{"request":true,"response":true},"bodyMaxSizeBytes":131072},"propagationFormats":["TRACECONTEXT"]}
[opentelemetry.auto.trace 2021-01-08 17:38:23:811 +0000] [main] INFO org.hypertrace.agent.core.config.HypertraceConfig - Config loaded: {"serviceName":"query-service","reporting":{"endpoint":"http://collector.traceableai:9411/api/v2/spans","opa":{"endpoint":"http://opa.traceableai:8181/","pollPeriodSeconds":30}},"dataCapture":{"httpHeaders":{"request":true,"response":true},"httpBody":{"request":true,"response":true},"rpcMetadata":{"request":true,"response":true},"rpcBody":{"request":true,"response":true},"bodyMaxSizeBytes":131072},"propagationFormats":["TRACECONTEXT"]}
[opentelemetry.auto.trace 2021-01-08 17:39:07:507 +0000] [main] INFO io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller - Installed span exporter: io.opentelemetry.exporter.zipkin.ZipkinSpanExporter
[opentelemetry.auto.trace 2021-01-08 17:39:07:908 +0000] [main] INFO io.opentelemetry.javaagent.tooling.PropagatorsInitializer - Added tracecontext propagator
Hypertrace agent started, version: 0.7.5
pavolloffay commented 3 years ago

First, we should see if there is any regression between the upstream build and HT and then try to figure out how to improve the startup time.

ravisingal commented 3 years ago

one of the possible reason could be cpu throttling. in case of api-anomaly-detector, it has cpu request=0.25 and limit=0.5. How much cpu does it need?

pavolloffay commented 3 years ago

How much cpu does it need?

There is no minimum required CPU, however as you pointed out restricted resources can affect startup time.