Feature Request: Do not deprecate custom queries.

[sig-piskule]

Proposal

Use case. Why is this important?

I am upgrading to Kubernetes 1.25 / 1.26. In it, PodSecurityPolicies are deprecated. This forces me to upgrade the my Postgres Exporter Helm Chart to version 5.0.0.

Version 4.X of the helm chart allows me to specify custom queries. However, this docker image seems to be deprecating custom metrics.

My custom metrics for example, are application specific. They are not Postgres specific. Ideally I do not want to convert my queries to Go-Lang code and recompile the Postgres Exporter just for my application. Because these queries are application specific, I would never write a custom collector nor submit a pull request here.

In addition, it seems that the recommendation is to now use this tool: https://github.com/burningalchemist/sql_exporter

However, now that means I must maintain and update Postgres Exporter, as well as SQL Exporter. This is twice the amount of work required for regular maintenance and patching. This particularly becomes burdensome when Helm Charts, Docker Images, Application Versions, and Kubernetes all need to be in-sync. This becomes very complicated at enterprise grade sizes when hundreds of Postgres servers and applications are maintained.

Is there a reason why this feature is being removed? The alternatives being provided increase the technical burden.

Note For other visitors from Google, postgres_exporter.yml does not allow you to input custom queries. Custom queries are totally deprecated, and the Postgres Exporter Helm Chart no longer supports custom queries.

Issue Link: https://github.com/prometheus-community/helm-charts/issues/3792

[SuperQ]

My custom metrics for example, are application specific

This is why we're intentionally deprecating the queries configuration file. This exporter is not meant to do both database server monitoring and business application monitoring. These are two different use cases and we do not want to support both here.

[ebini]

My custom metrics for example, are application specific

This is why we're intentionally deprecating the queries configuration file. This exporter is not meant to do both database server monitoring and business application monitoring. These are two different use cases and we do not want to support both here.

I am pretty sure a lot of users use the postgres exporter to get (also or only) "application metrics" via the database. So i hope you re-enable this feature. If you don't want to support this anymore, please add a note at the readme that this is (meanwhile) the only use case of this application.

[gkozyryatskyy]

My custom metrics for example, are application specific

This is why we're intentionally deprecating the queries configuration file. This exporter is not meant to do both database server monitoring and business application monitoring. These are two different use cases and we do not want to support both here.

I am pretty sure a lot of users use the postgres exporter to get (also or only) "application metrics" via the database. So i hope you re-enable this feature. If you don't want to support this anymore, please add a note at the readme that this is (meanwhile) the only use case of this application.

+1 we also looking for exactly this feature and deprecation forces us to find another solution

[Aaron-ML]

My custom metrics for example, are application specific

This is why we're intentionally deprecating the queries configuration file. This exporter is not meant to do both database server monitoring and business application monitoring. These are two different use cases and we do not want to support both here.

The exporter might not have been designed with that in mind, but it's used heavily for both and it works well.

It's a shame to see such customizability removed. We use this exporter heavily for both without issue, now we need to pivot entirely due to this.

Disabling custom queries also impacts server monitoring as well, there's so many cases where you need to write functions for things like Citus clusters or extensions just to pull monitoring data, now we cannot do that with this exporter.

I get that supporting users custom queries is a nightmare, however was hoping to have more of a warning on using custom queries being unsupported instead but still allow it to be used. Bummer

[paulwinex]

Thank you for your help, now render and logs correctly. But I have a new problem. In addition to the main render file, my scene generates a JSON file that is saved there next to the PNG files. I see that on your side it is successfully created and visible, but the download process ignores it and I only receive PNG files. How can I download everything I have into the output folder?

пт, 8 дек. 2023 г. в 22:33, Aaron Lucas @.***>:

My custom metrics for example, are application specific

This is why we're intentionally deprecating the queries configuration file. This exporter is not meant to do both database server monitoring and business application monitoring. These are two different use cases and we do not want to support both here.

The exporter might not have been designed with that in mind, but it's used heavily for both and it works well.

It's a shame to see such customizability removed. We use this exporter heavily for both without issue, now we need to pivot entirely due to this.

Disabling custom queries also impacts server monitoring as well, there's so many cases where you need to write functions for things like Citus clusters or extensions just to pull monitoring data, now we cannot do that with this exporter.

I get that supporting users custom queries is a nightmare, however was hoping to have more of a warning on using custom queries being unsupported instead but still allow it to be used. Bummer

— Reply to this email directly, view it on GitHub https://github.com/prometheus-community/postgres_exporter/issues/909#issuecomment-1847710940, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABX34ZF4KL7U7UEG54VFP7TYINTPRAVCNFSM6AAAAAA4VJUH5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBXG4YTAOJUGA . You are receiving this because you are subscribed to this thread.Message ID: @.*** com>

[SuperQ]

Disabling custom queries also impacts server monitoring as well, there's so many cases where you need to write functions for things like Citus clusters or extensions just to pull monitoring data, now we cannot do that with this exporter.

Thanks for this insight. It shows exactly the problem. This is exactly one of reasons the why we're turning this off. Users extend the exporter without ever sharing those enhancements. No issues filed, no PRs opened to add new queries. The community suffers due to lack of collaboration.

We've had more work done to the built-in collectors in the last year than all previous years combined.

@Aaron-ML If you have queries that are useful please share them by opening new issues. We would be happy to add new collectors for Citus clusters.

[SuperQ]

I get that supporting users custom queries is a nightmare, however was hoping to have more of a warning on using custom queries being unsupported instead but still allow it to be used.

That is what deprecation means. It is a warning that sometime in the future it may be removed. The queries.yaml is still supported. It has not been removed. We understand that people are still using it. We have not yet announced what version it will be removed in.