Prometheus Monitoring for Java Web Applications without Modifying their Source Code.
The promagent-maven-plugin
is a tool for creating custom Java agents for Prometheus monitoring.
The Java agents instrument Java Web Applications with Prometheus metrics without modifying the applications' source code.
The agents use the Byte Buddy bytecode manipulation library to insert Prometheus metrics during application startup.
The Promagent code repository contains two projects:
promagent-framework
: Provides, among other things, the promagent-api
and the promagent-maven-plugin
helping you to create your own agents.promagent-example
: An example agent providing metrics for Spring Boot and Java EE applications:
The example agent was tested with Tomcat for the Spring Boot example and with the Wildfly application server for the Java EE example.
Clone Promagent from GitHub:
git clone https://github.com/fstab/promagent.git
cd promagent
The promagent-api
and promagent-maven-plugin
are not on Maven Central yet. Run the following commands to make them available locally (in ~/.m2/repository/
):
cd promagent-framework
mvn clean install
cd ..
Compile the example agent. This should create the file ./promagent-example/target/promagent.jar
:
cd promagent-example
mvn clean verify
cd ..
The following runs with Java 8 and was not tested with Java 9 yet.
Download and compile a Spring Boot Getting Started application.
git clone https://github.com/spring-guides/gs-accessing-data-rest.git
cd gs-accessing-data-rest/complete
mvn clean package
cd ../..
Run the Spring Boot application with the Promagent attached.
java \
-javaagent:promagent/promagent-example/target/promagent.jar=port=9300 \
-jar gs-accessing-data-rest/complete/target/gs-accessing-data-rest-0.1.0.jar
Go to http://localhost:8080 to view the Spring Boot application, go to http://localhost:9300/metrics to view the Prometheus metrics.
_This demo runs with Java 8. For a Java 9 version, see JAVA_9_DEMO.md._
Download and compile a Wildfly Quickstart application.
git clone https://github.com/wildfly/quickstart.git
cd quickstart/kitchensink
mvn clean package
cd ../..
Download and extract the Wildfly application server.
curl -O http://download.jboss.org/wildfly/10.1.0.Final/wildfly-10.1.0.Final.tar.gz
tar xfz wildfly-10.1.0.Final.tar.gz
Run the Wildfly application server with the Promagent attached.
cd wildfly-10.1.0.Final
LOGMANAGER_JAR=$(find $(pwd) -name 'jboss-logmanager-*.jar')
export JAVA_OPTS="
-Xbootclasspath/p:${LOGMANAGER_JAR}
-Djboss.modules.system.pkgs=org.jboss.logmanager,io.promagent.agent
-Djava.util.logging.manager=org.jboss.logmanager.LogManager
-javaagent:../promagent/promagent-example/target/promagent.jar=port=9300
"
./bin/standalone.sh
In a new Shell window, deploy the quickstart application.
cd wildfly-10.1.0.Final
./bin/jboss-cli.sh --connect --command="deploy ../quickstart/kitchensink/target/kitchensink.war"
Go to http://localhost:8080/kitchensink to view the quickstart application, go to http://localhost:9300/metrics to view the Prometheus metrics.
A Promagent is implemented as a set of Hooks. A Hook is a Java class meeting the following requirements:
@Hook
.MetricsStore
.@Before
or @After
. Those methods must take exactly the same parameters as the method you want to intercept (there is one exception to that rule: @Afer
methods may have two additional parameters annotated with @Returned
and @Thrown
, see Hook Annotations below).The best way to get started is to have a look at the ServletHook
and JdbcHook
in the promagent-example
.
A simple Hook counting the number of Servlet requests looks as follows:
@Hook(instruments = "javax.servlet.Servlet")
public class ServletHook {
private final Counter servletRequestsTotal;
public ServletHook(MetricsStore metricsStore) {
servletRequestsTotal = metricsStore.createOrGet(new MetricDef<>(
"servlet_requests_total",
(name, registry) -> Counter.build()
.name(name)
.help("Total number of Servlet requests.")
.register(registry)
));
}
@After(method = "service")
public void after(ServletRequest request, ServletResponse response) {
httpRequestsTotal.inc();
}
}
To build a Promagent project with Maven, you need two entries in the pom.xml
. First, the promagent-api
must be included as a dependency:
<dependency>
<groupId>io.promagent</groupId>
<artifactId>promagent-api</artifactId>
<version>1.0-SNAPSHOT</version>
<scope>provided</scope> <!-- provided at runtime by the internal agent implementation -->
</dependency>
Second, the promagent-maven-plugin
that creates an agent JAR:
<build>
<finalName>promagent</finalName>
<plugins>
<plugin>
<groupId>io.promagent</groupId>
<artifactId>promagent-maven-plugin</artifactId>
<version>1.0-SNAPSHOT</version>
<executions>
<execution>
<id>promagent</id>
<phase>package</phase>
<goals>
<goal>build</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
With these two things included, mvn clean package
should produce a working Java agent in target/promagent.jar
.
A Hook's lifecycle depends on whether the instrumented method call is a nested call or not. A call is nested when the method is called by another method that was instrumented with the same hook. This happens for example if a Servlet's service() method calls another Servlet's service() method.
By default, nested calls are ignored and the Hook is only invoked for the outer call. In the Servlet example, this is the intended behavior, because it guarantees that each HTTP request is counted only once, even even if a Servlet internally calls another Servlet to handle the request.
If the Hook is defined with @Hook(skipNestedCalls = false)
the Hook will be invoked for all nested calls, not only for the outer call.
For each outer call, a new Hook instance is created. If the Hook implements both a @Before
and an @After
method, the same instance is used for @Before
and @After
. That way, you can set a start time as a member variable in the @Before
method, and use it in the @After
method to calculate the duration of the call.
For nested calls, the Hook instance from the outer call is re-used. That way, you can put data into member variables in order to pass that data down the call stack.
Most applications use static variables to maintain Prometheus metrics, as described in the Prometheus Client Library for Java documentation:
# Doesn't work with Promagent
static final Counter counter = Counter.build()
.name("requests_total")
.help("Total requests.")
.register();
Unfortunately, static variables are maintained per deployment in an application server. When an application is re-deployed, a new instance of the same Counter
is created, which causes conflicts in the Prometheus registry (as the Prometheus registry is maintained by Promagent, it survives re-deployments).
Moreover, it is impossible to instrument a mix of internal modules (like an internal Servlet in the JAX-RS implementation) and deployments (like Servlets in a WAR file) with static variables.
To prevent this, Promagent requires Hooks to use the MetricsStore
to maintain metrics:
# This is the correct way with Promagent
Counter counter = metricsStore.createOrGet(new MetricDef<>(
"requests_total",
(name, registry) -> Counter.build()
.name(name)
.help("Total requests.")
.register(registry)
));
The Promagent library will take care that the Counter
is created only once, and that the Counter
instance is re-used across multiple deployments and internal modules in an application server.
@Hook
: Hook classes are annotated with @Hook(instruments = {...}, skipNestedCalls = true)
. The instruments
parameter takes a list of Strings specifying the names of the classes or interfaces to be instrumented, like {"javax.servlet.Servlet", "javax.servlet.Filter"}
. The Hook instruments not only the classes or interfaces themselves, but all sub-classes or implementations of these classes or interfaces. The skipNestedCalls
parameter is described in A Hook's Life Cycle above.@Before
: Hook methods annotated with @Before(method = {...})
are invoked when an instrumented method is entered. The method
parameter takes a list of Strings specifying the names of the intercepted methods, like {"service", "doFilter"}
. The number and types of arguments are derived from the method itself, i.e. the Hook method annotated with @Before
must take the exact same parameters as the methods it wants to instrument.@After
: Hook methods annotated with @After(method = {...})
are invoked when an instrumented method is left. @After
methods are always called, even if the instrumented method terminates with an Exception. The semantics is the same as with the @Before
annotation. Methods annotated with @After
may have two additional parameters, one parameter annotated with @Returned
and one parameter annotated with @Thrown
. These parameters are ignored when determining the signature of the instrumented method.@Returned
: It might be useful to learn the return value of an instrumented method. In order to do so, methods annotated with @After
may have an additional parameter annotated with @Returned
, where the type corresponds to the return type of the intercepted method. If the instrumented method returns regularly, the return value is provided. If the method returns exceptionally, null
(or the default type for primitive types, like 0
for int
) is provided. @Returned
parameters are only allowed in @After
methods, not in @Before
methods.@Thrown
: The @Thrown
annotation is like @Returned
, but to learn an Exception thrown from an instrumented method. The type should be Throwable
to avoid class cast errors on unexpected RuntimeExceptions or Errors. If the instrumented method does not throw an exception, the parameter annotated with @Thrown
will be null
.The Prometheus server internally stores one time series for each observed set of label values. The time series database in the Prometheus server can easily handle thousands of different time series, but millions of different time series could be a problem. Therefore, it is important to keep the number of different label values relatively small. Unique user IDs, timestamps, or session keys should not be used as label values.
The promagent-example
strips HTTP URLs and SQL queries to make sure that there are not too many different label values:
/item/{id}
as labels, not in actual paths like /item/123
.INSERT INTO ITEMS (ID, NAME, PRICE) VALUES (...)
as a label, not in its values like INSERT INTO ITEMS (ID, NAME, PRICE) VALUES (23, 'abc', 17.5)
.Of course, replacing path parameters and SQL values is application specific. The promagent-example
implements a very simple replacement in ServletHook.stripPathParameters()
and JdbcHook.stripValues()
, but you probably need to customize these methods for your application.
The promagent-example
project contains an alternative Maven configuration in pom-with-docker-tests.xml
.
This configuration uses the docker-maven-plugin
to create Docker images and run integration tests against Docker containers.
The Wildfly tests can be run as follows:
cd promagent-example
mvn -f pom-with-docker-tests.xml clean verify -Pwildfly
cd ..
The Spring Boot tests can be run as follows:
cd promagent-example
mvn -f pom-with-docker-tests.xml clean verify -Pspring
cd ..
The first run takes a while, because the Docker images need to be built. Once the images are available on the local systems, runs are significantly faster.
Promagent supports three different ways of exposing metrics to the Prometheus server:
port
is used, as for example -javaagent:agent.jar=host=localhost,port=9300
.
The host
argument is optional, it defaults to the wildcard IP address.
If port
is omitted the built-in server is not started.This is a demo project. The main goal is to learn the internals of bytecode manipulation and class loading in Java application servers. I am planning to work on the following:
JmxCollector
in promagent-example
.The promagent-api
and promagent-maven-plugin
are not yet available on Maven Central, but they will be uploaded when the API becomes a bit more stable.
If you want to write your own agent and are looking for examples of methods you might want to instrument, look at related projects, like inspectIT (hooks are configured here) or stagemonitor.
This project is supported as part of the R&D activities at ConSol Software GmbH. See the ConSol Labs Blog for more info.