search

tomkerkhove / promitor

Bringing Azure Monitor metrics where you need them.
https://promitor.io
MIT License
248 stars 91 forks source link

feat: Provide support for batch scraping #2459

Open hkfgo opened 3 months ago

hkfgo commented 3 months ago

Hey Tom, hope all is well!

I've attached this diagram to illustrate my proposed change.

Screenshot 2024-03-22 at 10 23 21 PM

How to Batch?

Scrape jobs are currently defined per <resource, metric>. I'm taking the simplest approach to just batch resources looking for the same metric.

Max batch size will be a configurable parameter, under runtime configuration.

Changes to Internal Metrics

Internal Promitor metrics will need to be modified under batch scraping mode. promitor_scrape_success and promitor_scrape_error metrics should be appended with an additional tag batched, to indicate the mode of operation(0/1) A histogram metric promitor_batch_size should be added to track actual batch sizes promitor_ratelimit_arm_throttled/promitor_ratelimit_arm metrics will no longer be published, since batch scraping does not go through ARM.

Other considerations

Batch scraping will not be free anymore but comes with other benefits(like no longer being subject to ARM throttling). This should be documented for all users

Implements #2284

github-actions[bot] commented 3 months ago

Thank you for your contribution! 🙏 We will review it as soon as possible.

tomkerkhove commented 3 months ago

Thanks! Is it safe to say that the top of the diagram is current and bottom shows these changes?

hkfgo commented 1 month ago

Hey I'm revisiting this PR now since SDK migration is almost across the finish line. Do you know if the batch scraping API is available in sovereign clouds and what URLs we should use? We are particularly interested in the US Gov cloud. I can't find any information online(Azure doesn't really mentioned the existence of sovereign clouds to begin with)

tomkerkhove commented 1 month ago

Do you know if the batch scraping API is available in sovereign clouds and what URLs we should use?

I do not know, sorry! What is the URL for public cloud?

tomkerkhove commented 1 month ago
hkfgo commented 1 month ago

Awesome! How did you find this out?

tomkerkhove commented 1 month ago

I reached out to the team :D

tomkerkhove commented 1 month ago

@hkfgo Do you know if you'll have time to finalize this PR next week? I'd like to cut a release next week.

hkfgo commented 1 month ago

@hkfgo Do you know if you'll have time to finalize this PR next week? I'd like to cut a release next week.

We should plan for batch scraping to be on the release after this coming one. There are still a decent amount of work left and I'm not able to work on this full time.

I do have time to get SDK migration done though. In that case we should have something major-ish to release :)

tomkerkhove commented 1 month ago

That's OK - Thank you!