Could this help in AWS Batch and Prometheus?

[codeeMadness]

Hi Bwalti,

I'm working for a Java project which deployed as a AWS Batch Job (its underlayer run on Fargate Task). Currently, we would like to send metrics to Managed Prometheus from AWS BatchJob. Would your sample setup helps on this study case? What about Prometheus remote write? Sorry for those questions, I'm still a new learner on this field. Kindly advice.

Regards,

[BWalti]

This repository only demonstrates how to create and expose some basic metrics via Prometheus exporter, so that they are fetchable if you have a routed network connection to this application. (so Prometheus could potentially scrape the metrics)

BUT, AWS Batch jobs are short lived (edit: relatively.. usually not "always up"), therefore it doesn't make sense to configure Prometheus to reach out to this job, so the batch job would need to actively push metrics to AWS Managed Prometheus (either directly, or - in my case preferably - using the Open Telemetry collector sidecar)

I was actually working in an environment, where we were using AWS cloud services. I wasn't used to AWS before, so I imagined, as AWS Batch Jobs is built on top of Fargate, we could apply the same pattern to all solutions (running an Open Telemetry collector sidecar alongside the "application" container to collect metrics and push them to where we needed them)

BUT last time I checked, AWS Batch jobs did NOT SUPPORT sidecars, that may have changed in the meantime though: https://aws.amazon.com/about-aws/whats-new/2024/02/aws-batch-multi-container-jobs/

so the idea was to always run an Open Telemetry collector (or the AWS modified version of it) and configure it appropriately.

Hope that helps?

[codeeMadness]

Hi BWalti, Thanks for replying.

What do you think about sending metrics via Prometheus remote-write? (without OTEL) Is this possible? If yes. how we can store the Counter? I'm thinking of storing Counter in database. Kindly advice :)

Regards,

[BWalti]

I'm no Java expert, but last time I checked I didn't find any Java Prometheus remote-write library which seemed "good quality". And as far as I understood, this all goes down to "Prometheus prefers the pull principle". Nevertheless sometimes a Push is required, e.g. in cases where the metric producing service is stopping quick after producing metrics (AWS Batch Job).

So as there might be no good Java Prometheus remote write library around, but you still need to push the metrics, what can you do?

• Add an intermediary service which fetches the metrics from the endpoint and pushes them to Prometheus. (But what, if the intermediary services doesn't get to do the pull before the Batch Job dies?)
• Use a "push gateway" (but that might continue exposing the "same metric" until a new Batch Job runs and may "override" last value)
• use a mix of both:
  • push metrics from your service to the intermediary, which directly pushes them to Prometheus using Prometheus remote write - in this case, the first push from your app producing metrics to the intermediary can be done in a different way (e.g. OTLP exporter)
  • use the OpenTelemetry collector to receive OTLP pushed metrics and push them using Prometheus remote write to AWS Managed Prometheus (this collector can potentially be run as a sidecar to your normal app container, therefore has the same lifetime)
• other solutions..

So I went with the OTLP push to the sidecar and using configuration for the sidecar to ultimately push the metrics to Prometheus.

[codeeMadness]

Hi BWalti,

Thanks so much for your suggestion. I thought we can treat Prometheus Remote Write as an API endpoint, right? Because we do have a Remote Write endpoint and also ARN for "write" permission. So my idea is let the batch produce some metrics and send those metrics to Prometheus Remote Write (just like sending data via POST API) And we only need some place to store the counter.

Sorry if my idea is silly. (bc I'm completely new on this era).

Regards,