oVirt Grafana dashboards all blank

[WildStar2022]

Affected version

oVirt Engine: (oVirt Version 4.5.5-1.el8) (includes nightly update of oVirt-engine)

Describe the bug

SELinux is preventing /usr/sbin/grafana-server from name_connect access on the tcp_socket port 5432, resulting in all oVirt Grafana dashboards having no data.

To reproduce

n/a

Expected behavior

n/a

Screenshots

n/a

Additional context

Added this local policy to fix the issue:

ausearch -c 'grafana-server' --raw | audit2allow -M my-grafanaserver semodule -X 300 -i my-grafanaserver.pp

[Ecsi1337]

/ovirt-test.local/ovirt-engine-grafana/datasources/edit/DS_OVIRT_DWH handler=/api/ds/query logger=tsdb.postgres t=2024-02-05T14:42:09.434075498+01:00 level=error msg="query error" err="dial tcp [::1]:5432: connect: permission denied"

" ausearch -c 'grafana-server' --raw | audit2allow -M my-grafanaserver semodule -X 300 -i my-grafanaserver.pp "

this solution didn't work for me

[WildStar2022]

If your issue is indeed also with SELinux, have a look for a similar fix in your engine's /var/log/messages

Example:

messages-20240128:Jan 27 15:23:16 ovirt-engine setroubleshoot[5962]: SELinux is preventing /usr/sbin/grafana-server from name_connect access on the tcp_socket port 5432.#012#012* Plugin catchall (100. confidence) suggests **#012#012If you believe that grafana-server should be allowed name_connect access on the port 5432 tcp_socket by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'grafana-server' --raw | audit2allow -M my-grafanaserver#012# semodule -X 300 -i my-grafanaserver.pp#012

Hope that helps.

[Ecsi1337]

The entries referring to the error have successfully disappeared from /var/log/messages, but even though I set the log settings of the Grafana server and the Postgresql server to debug mode, I still cannot connect to the database. "db query error: failed to connect to server - please inspect Grafana server log for details" error. I can still log in successfully from the console with the command "psql -U ovirt_engine_history_grafana -d ovirt_engine_history -W -h localhost".

Do you have any other ideas?

[mwperina]

Hi, That smells like recent selinux issue when Grafana cannot connect to Postgresql data source. As a quick workaround you can use custom selinux policy similar to https://github.com/oVirt/ovirt-system-tests/pull/324

We are waiting on proper fix on Grafana side, which we can reuse in oVirt DWH.

Martin

On Thu, 8 Feb 2024, 09:14 Ecsi1337, @.***> wrote:

The entries referring to the error have successfully disappeared from /var/log/messages, but even though I set the log settings of the Grafana server and the Postgresql server to debug mode, I still cannot connect to the database. "db query error: failed to connect to server - please inspect Grafana server log for details" error. I can still log in successfully from the console with the command "psql -U ovirt_engine_history_grafana -d ovirt_engine_history -W -h localhost".

Do you have any other ideas?

— Reply to this email directly, view it on GitHub https://github.com/oVirt/ovirt-engine/issues/915#issuecomment-1933560921, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTC3MI5G6FWI6OQ3V2YBIDYSSCMXAVCNFSM6AAAAABCNOJSL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZTGU3DAOJSGE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[0ffer]

Hi everyone. For grafana >= 9.2.10-15 version there is already included patch with selinux flag for grafana and postgresql.

And if you have grafana with correct version, you can enable it by hands with command semanage boolean --modify --on grafana_can_tcp_connect_postgresql_port

I created pr for enable this bool automatically during engine-setup: https://github.com/oVirt/ovirt-dwh/pull/67