References in custom-vars are not resolved

[fl0wx]

is there a reason why the references in custom vars are not getting resolved? customvar: vars.query = "max by (k8s_cluster, pod) (rate(container_cpu_cfs_throttled_periods_total{container=\"$host.vars.kubernetes_appname$\"}[5m]))"

customvars grafana debug output:

Custom Variables | &var-query=max%20by%20%28k8s_cluster%2C%20pod%29%20%28rate%28container_cpu_cfs_throttled_periods_total%7Bcontainer%3D%22%24host.vars.kubernetes_appname%24%22%7D%5B5m%5D%29%29 -- | --

the following is not working: vars.query = "max by (k8s_cluster, pod) (rate(container_cpu_cfs_throttled_periods_total{container=\"{{host.vars.kubernetes_appname}}\"}[5m]))"

I need this to build dynamic grafana prometheus dashboards which can be embedded as grafana graphs in icinga.

Your Environment

• Icinga Web 2 Grafana version (System - About): 1.4.2
• Icinga Web 2 version and modules (System - About): 2.7.3
• Version used (icinga2 --version): 2.11.3
• Operating System and version:
• Enabled features (icinga2 feature list):
• Config validation (icinga2 daemon -C):

[Mikesch-mp]

You should ask Icinga why the variable is not resolved. The module takes the variable like it is in Icinga. Check different approaches how icinga will render the variable and confirm it with icinga object list --type host|service --name whateverand check the vars section of the output.

[fl0wx]

@Mikesch-mp thanks for the reply, looks like this is an icinga issue yes. Output of the object list command:

icinga2 object list --type service --name generic_k8s_heapused | grep "query =" |head -n 1
    * query = "(base_memory_usedHeap_bytes{app="$host.vars.kubernetes_appname$", kubernetes_namespace="$host.vars.kubernetes_namespace$", prometheus_cluster=''} / base_memory_maxHeap_bytes{app="$host.vars.kubernetes_appname$", kubernetes_namespace="$host.vars.kubernetes_namespace$", prometheus_cluster=''}) * 100"

[nilmerg]

I'm posting this here instead of in the forums, as I think Carsten should know this as well:

After thinking about this more, I'm pretty sure this isn't something Icinga Web 2 could solve. The custom variable in question is accessed by the grafana module directly if I'm not mistaken. It isn't passed in by some hook or so. So it's entirely the grafana module's duty to resolve macros in there. Even if Icinga Web 2 would resolve them in the UI, this still wouldn't affect the grafana module as it bypasses it by accessing the variables directly. As noted in the just now mentioned issue, Icinga Web 2 is unable to resolve Icinga 2 macros however, so the grafana module would need to implement support for this natively on its own.

[fl0wx]

@Mikesch-mp : would it be possible that the variables can be resolved in the future? I thought they will be resolved but as you can see in the previous comment, they will not (in icinga2web)

[Mikesch-mp]

it should be resolved by icinga and not by icingaweb. You are using the director? if yes kick it and use the power of icinga conf and icinga dsl. The Grafana module will never do the job icnga has to do or icingaweb has to do, it only uses the given hooks.

[fl0wx]

@Mikesch-mp : I also opened an issue in the icinga2 repo and linked them. if you think that director is the issue, how would you solve this specific issue with icinga conf and icinga dsl? Any hint or snippet would be useful. Thanks and greetings fl0w

[Mikesch-mp]

You maybe can try something like {{ return host.vars.whatever }}

But iam only 2 days away from my 3 month holidays. no time left to test

[fl0wx]

@Mikesch-mp thanks for the reply. I cannot drop director here, so i need a solution which works with director. Sadly i cannot find anything useful in the director documentation.

[fl0wx]

@Mikesch-mp Any ideas ? {{ return host.vars.whatever }} in the variable field doesnt work. Like i said, i cannot drop director even if i want. so i really need a solution to get this to work. Maybe you could also help me how to implement this on my own if you wont do that. The problem is that @nilmerg says you should implement this and you said icinga itself should implement this. This doesnt lead to any helpful solution :(

[fl0wx]

I'm posting this here instead of in the forums, as I think Carsten should know this as well:

After thinking about this more, I'm pretty sure this isn't something Icinga Web 2 could solve. The custom variable in question is accessed by the grafana module directly if I'm not mistaken. It isn't passed in by some hook or so. So it's entirely the grafana module's duty to resolve macros in there. Even if Icinga Web 2 would resolve them in the UI, this still wouldn't affect the grafana module as it bypasses it by accessing the variables directly. As noted in the just now mentioned issue, Icinga Web 2 is unable to resolve Icinga 2 macros however, so the grafana module would need to implement support for this natively on its own.

@nilmerg hey as far as i see and if im getting it right, grafana module is using fetchCustomVars to get all custom vars from icingaweb: https://github.com/Mikesch-mp/icingaweb2-module-grafana/blob/master/library/Grafana/ProvidedHook/Grapher.php#L334 -> https://github.com/Icinga/icingaweb2/blob/3203cd8feabac22e89db52f588e6872c68a4abc9/modules/monitoring/library/Monitoring/Object/MonitoredObject.php#L403

so i assume, grafana just gets key value pairs from icingaweb and uses the value for each variable. But the value returned from icingaweb2 isnt resolved.

Is that correct? Cant you provide more help here please?

[nilmerg]

As mentioned here we don't plan to add support for this as part of the monitoring module as the syntax of how Icinga 2 resolves custom variable macros differs from how the monitoring module does. Icinga DB Web however could support it and has the capability already.

However, this does not help the grafana module. (or any module, actually) A module using the hooks fetches the custom variables directly, "raw", so to say. So if it wants to support resolving macros there, a module implementing the Icinga DB Web hooks could make use of the Macros trait. So, once the grafana module merges/adds support for Icinga DB, this should be easy to implement.

[fl0wx]

@nilmerg the macro part of icingadb sounds great. @Mikesch-mp are you considering this is your icingdb rework? And ist there any estimated time for the release?