johrstrom commented 6 years ago

I use this plugin for very long lived tests (they're functional tests that run forever actually). But that's perhaps not the general use case. This enhancement is to support the PushGateway as well as a server.

I'm thinking this implementation looks like this:

A separate Listener class so users choose either server or push gateway (or both?)
They share a lot of the implementation like configuration and all the internals of how labels are extracted from samples.
This other class probably extends AbstractBackendListenerClient

johrstrom commented 6 years ago

@dirkjot let's please continue the discussion here. After I thought about this with the smart part of my brain, I figured it'd be better separated into 2 classes. Thoughts?

dirkjot commented 6 years ago

Not having looked at the internals too much yet, I was thinking of just one listener class with an option to push at end. Exposing stats during the jmeter run (as the plugin does now) is not going to hurt anybody who does not need it, so we do not need to ability to switch that off. So that would leave a checkbox to switch "Push at end" on or off, and a text area with the URL of the push gateway.

var23rav commented 6 years ago

An alternative solution can be, export the metrics from result(.jtl) file. Can we try somthing as simple as JmeterResultCsv2PrometheusMetricsExporter.py

arunavijay commented 5 years ago

Hi, First of all, thanks for writing up this plugin! I wanted to see if there is any update on the PushGateway support. I intend to report my performance test results to Prometheus through PushGateway as my tests are short lived. So, having the push gateway support would be very useful. Thanks!

johrstrom commented 5 years ago

Unfortunately, I've been actively re-working the UI for the listeners to support more diverse metric creation & use. That's taken up all of my time and it's been a long time coming, so I have to deliver all of it before I can move on to this.

Would something like what @var23rav has suggested work for you? Or more like what @dirkjot describes. I think both in the end would be optimal, the first would take less code, but the latter may be more reasonable for users (if it can be implemented in a sane way without keeping tons of metrics forever).

arunavijay commented 5 years ago

Hi @johrstrom, Thanks for the quick reply. I feel something that's integrated with the plugin would be easier for users(second option).

aurolle commented 5 years ago

Hi is there any update? It would be a good solution also for distributed testing

Germey commented 4 years ago

any progress?

GiovanniPaoloGibilisco commented 4 years ago

The current solution is already fine for distributed testing as there is no need to report to JMeter master any sample. Prometheus can scrape from all load injectors Am I missing something?

Il giorno mer 11 mar 2020 alle ore 17:46 崔庆才丨静觅 notifications@github.com ha scritto:

any progress?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/johrstrom/jmeter-prometheus-plugin/issues/23#issuecomment-597743054, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNQ7533PQ75N5NAKGPFHQDRG655JANCNFSM4EBQZAMQ .

vbarra1982 commented 4 years ago

In my opinion it would be great to have the push gateway capability, imagine when you must run jmeter on jenkins (which is a typical case to track the performance continuously). I think JMeter should push the metrics - because it is a short lived process. Additionally, it's not very straightforward to say prometheus which "hostname" it must scrap data from, as jenkins runs across several nodes.

steinbrueckri commented 4 years ago

I also thought it would be cooler if it was directly in the plugin. Until then you could maybe do it with the following workaround.

curl -s http://localhost:9100/metrics | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance

itsurkan commented 3 years ago

I also thought it would be cooler if it was directly in the plugin. Until then you could maybe do it with the following workaround.
curl -s http://localhost:9100/metrics | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance

Hello, where to run this script?

steinbrueckri commented 3 years ago

I also thought it would be cooler if it was directly in the plugin. Until then you could maybe do it with the following workaround.
curl -s http://localhost:9100/metrics | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance
Hello, where to run this script?

In our case, we run JMeter tests in a container and the main process is a script. This script runs the command in 5 seconds intervals. Sadly this not public so I cannot reference this. If this doesn't help you ping me again after the holidays and I will post a snippet. 😅

Cheers 🥂 Richard

UPDATE 04.01.21: @itsurkan

Docker

FROM weaveworks/flagger-loadtester:0.18.0

ARG JMETER_VERSION="5.3"

ENV JMETER_HOME /opt/apache-jmeter-${JMETER_VERSION}
ENV JMETER_BIN ${JMETER_HOME}/bin
ENV MIRROR_HOST http://mirror.synyx.de/apache/jmeter
ENV JMETER_DOWNLOAD_URL ${MIRROR_HOST}/binaries/apache-jmeter-${JMETER_VERSION}.tgz
ENV JMETER_PLUGINS_DOWNLOAD_URL https://repo1.maven.org/maven2
ENV JMETER_PLUGINS_FOLDER ${JMETER_HOME}/lib/ext/

USER root

# hadolint ignore=DL3018
RUN apk add --no-cache --update ca-certificates \
    && update-ca-certificates \
    && apk add --no-cache --update openjdk8-jre tzdata curl unzip bash \
    && mkdir -p /tmp/jmeter  \
    && echo "${JMETER_DOWNLOAD_URL}" \
    && curl -L --silent ${JMETER_DOWNLOAD_URL} >  /tmp/jmeter/apache-jmeter-${JMETER_VERSION}.tgz  \
    && mkdir -p /opt  \
    && tar -xzf /tmp/jmeter/apache-jmeter-${JMETER_VERSION}.tgz -C /opt  \
    && rm -rf /tmp/jmeter

# FIXME: please find a smarter solution
RUN curl -L --silent ${JMETER_PLUGINS_DOWNLOAD_URL}/kg/apc/jmeter-plugins-autostop/0.1/jmeter-plugins-autostop-0.1.jar -o ${JMETER_PLUGINS_FOLDER}/jmeter-plugins-autostop-0.1.jar
RUN curl -L --silent ${JMETER_PLUGINS_DOWNLOAD_URL}/kg/apc/jmeter-plugins-dummy/0.2/jmeter-plugins-dummy-0.2.jar -o ${JMETER_PLUGINS_FOLDER}/jmeter-plugins-dummy-0.2.jar
RUN curl -L --silent ${JMETER_PLUGINS_DOWNLOAD_URL}/kg/apc/jmeter-plugins-cmn-jmeter/0.5/jmeter-plugins-cmn-jmeter-0.5.jar -o ${JMETER_PLUGINS_FOLDER}/jmeter-plugins-cmn-jmeter-0.5.jar
RUN curl -L --silent ${JMETER_PLUGINS_DOWNLOAD_URL}/com/github/johrstrom/jmeter-prometheus-plugin/0.6.0/jmeter-prometheus-plugin-0.6.0.jar -o ${JMETER_PLUGINS_FOLDER}/jmeter-prometheus-plugin-0.6.0.jar

ENV PATH $PATH:$JMETER_BIN

COPY src/resources/jmeter/ /jmeter
COPY src/resources/launch.sh /launch.sh
RUN chmod +x /launch.sh

# default user from flagger-loadtester base image
USER app

Launcher

#!/bin/bash
set -e

# externally variables 
JVM_ARGS=${JVM_ARGS:-"-Dfile.encoding=UTF-8 -Dconsole.encoding=UTF-8"}
TARGET_HOST=${TARGET_HOST:-""}
JMETER_TEST_FILE=${JMETER_TEST_FILE:-"/jmeter/loadtest.jmx"}
JMETER_PROPERTIES_FILE=${JMETER_PROPERTIES_FILE:-"/jmeter/jmeter.properties"}
JMETER_ARGS=${JMETER_ARGS:-""}
JMETER_METRIC_URL="http://localhost:9270/metrics"
PUSHGATEWAY_HOST=${PUSHGATEWAY_HOST:-"http://prometheus-pushgateway.monitoring:9091"}
PUSHGATEWAY_JOB_NAME=${PUSHGATEWAY_JOB_NAME:-"loadtest"}
PUSHGATEWAY_SEND_METRICS_SECONDS=${PUSHGATEWAY_SEND_METRICS_SECONDS:-"10"}

# internally variables
export _JMETER_ARGS="${JMETER_ARGS} -Jtarget_host=${TARGET_HOST} --nongui --systemPropertyFile ${JMETER_PROPERTIES_FILE} --testfile ${JMETER_TEST_FILE}"

# Get version from the target
if [ -z "${TARGET_HOST}" ]; then
    echo "INFO: TARGET_HOST is not set"
else
    VERSION=${curl -s ${TARGET_HOST}/metrics | grep version | cut -d '"' -f 4}
    PUSHGATEWAY_JOB_NAME="$PUSHGATEWAY_JOB_NAME-$VERSION"
fi
echo "START: Running Jmeter on `date`"

echo "JVM_ARGS: $JVM_ARGS"
echo "JMETER_ARGS: $JMETER_ARGS"
echo "_JMETER_ARGS: $_JMETER_ARGS"

## run the jmeter in background
jmeter $_JMETER_ARGS &
JMETER_PID=$!

## run while the jmeter is running
while kill -0 $JMETER_PID 2> /dev/null; do
    # check if pushgateway is enabled
    if [[ ! "$PUSHGATEWAY_HOST" -eq "false" ]]; then
        echo "send metrics from JMETER_METRIC_URL $JMETER_METRIC_URL to PUSHGATEWAY $PUSHGATEWAY_HOST/metrics/job/$PUSHGATEWAY_JOB_NAME/instance/$HOSTNAME"
        curl -s $JMETER_METRIC_URL | curl --data-binary @- $PUSHGATEWAY_HOST/metrics/job/$PUSHGATEWAY_JOB_NAME/instance/$HOSTNAME
        sleep $PUSHGATEWAY_SEND_METRICS_SECONDS
    else
        echo "INFO: pushgateway use is disabled."
    fi
done

echo "END: Running Jmeter on `date`"

dhanyanayak24 commented 2 years ago

I also thought it would be cooler if it was directly in the plugin. Until then you could maybe do it with the following workaround.

Hi Richard,

I went through the above code to run your jmeter shell script from the launcher and how metrics are pushed to pushgateway.

I tried the curl command where we are pushing the metrics from 9270 (jmeter) to pushgateway. But is there any way where we can avoid using the jmeter prometheus plugin(i mean without setting up prometheus listener n all in the jmeter script) and then directly pass jmeter result to pushgateway.

I tried disabling the prometheus listener and then using the same curl command..It will work as the jmeter metrics willnot be triggered to 9270. So is there any way to use pushgateway without the plugin n we get the jmeter results.

Thanks Dhanya

curl -s http://localhost:9100/metrics | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job/instance/some_instance

Hello, where to run this script?

In our case, we run JMeter tests in a container and the main process is a script. This script runs the command in 5 seconds intervals. Sadly this not public so I cannot reference this. If this doesn't help you ping me again after the holidays and I will post a snippet. 😅

Cheers 🥂 Richard

UPDATE 04.01.21: @itsurkan

Docker

FROM weaveworks/flagger-loadtester:0.18.0

ARG JMETER_VERSION="5.3"

ENV JMETER_HOME /opt/apache-jmeter-${JMETER_VERSION}
ENV JMETER_BIN ${JMETER_HOME}/bin
ENV MIRROR_HOST http://mirror.synyx.de/apache/jmeter
ENV JMETER_DOWNLOAD_URL ${MIRROR_HOST}/binaries/apache-jmeter-${JMETER_VERSION}.tgz
ENV JMETER_PLUGINS_DOWNLOAD_URL https://repo1.maven.org/maven2
ENV JMETER_PLUGINS_FOLDER ${JMETER_HOME}/lib/ext/

USER root

# hadolint ignore=DL3018
RUN apk add --no-cache --update ca-certificates \
  && update-ca-certificates \
  && apk add --no-cache --update openjdk8-jre tzdata curl unzip bash \
  && mkdir -p /tmp/jmeter  \
  && echo "${JMETER_DOWNLOAD_URL}" \
  && curl -L --silent ${JMETER_DOWNLOAD_URL} >  /tmp/jmeter/apache-jmeter-${JMETER_VERSION}.tgz  \
  && mkdir -p /opt  \
  && tar -xzf /tmp/jmeter/apache-jmeter-${JMETER_VERSION}.tgz -C /opt  \
  && rm -rf /tmp/jmeter

# FIXME: please find a smarter solution
RUN curl -L --silent ${JMETER_PLUGINS_DOWNLOAD_URL}/kg/apc/jmeter-plugins-autostop/0.1/jmeter-plugins-autostop-0.1.jar -o ${JMETER_PLUGINS_FOLDER}/jmeter-plugins-autostop-0.1.jar
RUN curl -L --silent ${JMETER_PLUGINS_DOWNLOAD_URL}/kg/apc/jmeter-plugins-dummy/0.2/jmeter-plugins-dummy-0.2.jar -o ${JMETER_PLUGINS_FOLDER}/jmeter-plugins-dummy-0.2.jar
RUN curl -L --silent ${JMETER_PLUGINS_DOWNLOAD_URL}/kg/apc/jmeter-plugins-cmn-jmeter/0.5/jmeter-plugins-cmn-jmeter-0.5.jar -o ${JMETER_PLUGINS_FOLDER}/jmeter-plugins-cmn-jmeter-0.5.jar
RUN curl -L --silent ${JMETER_PLUGINS_DOWNLOAD_URL}/com/github/johrstrom/jmeter-prometheus-plugin/0.6.0/jmeter-prometheus-plugin-0.6.0.jar -o ${JMETER_PLUGINS_FOLDER}/jmeter-prometheus-plugin-0.6.0.jar

ENV PATH $PATH:$JMETER_BIN

COPY src/resources/jmeter/ /jmeter
COPY src/resources/launch.sh /launch.sh
RUN chmod +x /launch.sh

# default user from flagger-loadtester base image
USER app

Launcher

#!/bin/bash
set -e

# externally variables 
JVM_ARGS=${JVM_ARGS:-"-Dfile.encoding=UTF-8 -Dconsole.encoding=UTF-8"}
TARGET_HOST=${TARGET_HOST:-""}
JMETER_TEST_FILE=${JMETER_TEST_FILE:-"/jmeter/loadtest.jmx"}
JMETER_PROPERTIES_FILE=${JMETER_PROPERTIES_FILE:-"/jmeter/jmeter.properties"}
JMETER_ARGS=${JMETER_ARGS:-""}
JMETER_METRIC_URL="http://localhost:9270/metrics"
PUSHGATEWAY_HOST=${PUSHGATEWAY_HOST:-"http://prometheus-pushgateway.monitoring:9091"}
PUSHGATEWAY_JOB_NAME=${PUSHGATEWAY_JOB_NAME:-"loadtest"}
PUSHGATEWAY_SEND_METRICS_SECONDS=${PUSHGATEWAY_SEND_METRICS_SECONDS:-"10"}

# internally variables
export _JMETER_ARGS="${JMETER_ARGS} -Jtarget_host=${TARGET_HOST} --nongui --systemPropertyFile ${JMETER_PROPERTIES_FILE} --testfile ${JMETER_TEST_FILE}"

# Get version from the target
if [ -z "${TARGET_HOST}" ]; then
    echo "INFO: TARGET_HOST is not set"
else
    VERSION=${curl -s ${TARGET_HOST}/metrics | grep version | cut -d '"' -f 4}
    PUSHGATEWAY_JOB_NAME="$PUSHGATEWAY_JOB_NAME-$VERSION"
fi
echo "START: Running Jmeter on `date`"

echo "JVM_ARGS: $JVM_ARGS"
echo "JMETER_ARGS: $JMETER_ARGS"
echo "_JMETER_ARGS: $_JMETER_ARGS"

## run the jmeter in background
jmeter $_JMETER_ARGS &
JMETER_PID=$!

## run while the jmeter is running
while kill -0 $JMETER_PID 2> /dev/null; do
    # check if pushgateway is enabled
    if [[ ! "$PUSHGATEWAY_HOST" -eq "false" ]]; then
        echo "send metrics from JMETER_METRIC_URL $JMETER_METRIC_URL to PUSHGATEWAY $PUSHGATEWAY_HOST/metrics/job/$PUSHGATEWAY_JOB_NAME/instance/$HOSTNAME"
        curl -s $JMETER_METRIC_URL | curl --data-binary @- $PUSHGATEWAY_HOST/metrics/job/$PUSHGATEWAY_JOB_NAME/instance/$HOSTNAME
        sleep $PUSHGATEWAY_SEND_METRICS_SECONDS
    else
        echo "INFO: pushgateway use is disabled."
    fi
done

echo "END: Running Jmeter on `date`"

johrstrom commented 2 years ago

@steinbrueckri thanks for the workaround! If you submitted a pull request with the same - I'd be happy to accept it.

I tried the curl command where we are pushing the metrics from 9270 (jmeter) to pushgateway. But is there any way where we can avoid using the jmeter prometheus plugin(i mean without setting up prometheus listener n all in the jmeter script) and then directly pass jmeter result to pushgateway.

@dhanyanayak24 ~~- you'd need a listener of some kind. You could pipe simple data writer to awk + some kind of data writer then curl, but the formatting and labeling is a little complex.~~ (Striking through because you'd actually need some program, likely one of the prometheus libraries, to interpret and aggregate a csv

Not to my knowledge - though there could be a fork.

steinbrueckri commented 2 years ago

@steinbrueckri thanks for the workaround! If you submitted a pull request with the same - I'd be happy to accept it.

@johrstrom As Update in the documentation (Readme) or as "Example Implementation" in ./examples?

Cheer Richard

johrstrom commented 2 years ago

maybe a push-gateway directory like so? I don't think I can distribute it with jar, but at least other folks can use it.

push-gateway/Dockerfile
push-gateway/launch.sh

johrstrom / jmeter-prometheus-plugin

Support Push Gateway #23

Docker

Launcher

Docker

Launcher