Looking for clarifications

[BGmot]

Hello, this is not a bug or a feature request, I just can't wrap my head around how zabbix-web container connects to zabbix-server. zabbix-web application uses /usr/share/zabbix/conf/zabbix.conf.php is used for connection to Zabbix server settings, I am looking at my deployment and see that:

$ kubectl exec -n mblab mblab-zabbix-web-786855b659-l9dfg -- grep ZBX_SERVER /usr/share/zabbix/conf/zabbix.conf.php 
if (getenv('ZBX_SERVER_HOST')) {
    $ZBX_SERVER      = getenv('ZBX_SERVER_HOST');
    $ZBX_SERVER_PORT = getenv('ZBX_SERVER_PORT');
$ZBX_SERVER_NAME = getenv('ZBX_SERVER_NAME');

but we don't provide any ZBX_SERVER_HOST or ZBX_SERVER_PORT environment variables. Following confirms that:

$ kubectl exec -n mblab mblab-zabbix-web-786855b659-l9dfg -- env | grep ZBX_SERVER
$

Could you please explain where Zabbix WebUI takes these settings from? Thank you.

[aeciopires]

Hi @BGmot!

Great question!

Short answer: this is a mystery for me. :-P :-)

Long answer: I reviewed all helm chart files, I searched on all files of /usr/share/zabbix and /etc/zabbix directories in the zabbix-web pod; I searched in Google and StackOverflow, I readed the documentation of Zabbix, I readed the documentation of the Zabbix Docker Images and I looked in the repository https://github.com/zabbix/zabbix-docker. But I didn't understand how this service works in Kubernetes without these environment variables.

I saw the bellow environment variables created in the zabbix-web pod. I think they are used by application, but how that works is a mystery for me.

Hi @sa-ChristianAnton! Can you help us?

ZABBIX_ZABBIX_WEBSERVICE_SERVICE_HOST=10.103.95.238
POSTGRES_PASSWORD=CHANGE_HERE
ZABBIX_ZABBIX_WEBSERVICE_PORT_10053_TCP_PROTO=tcp
ZABBIX_ZABBIX_SERVER_PORT_10051_TCP=tcp://10.103.230.210:10051
ZBX_SOURCES=https://git.zabbix.com/scm/zbx/zabbix.git
ZABBIX_ZABBIX_AGENT_PORT_10050_TCP_PROTO=tcp
ZABBIX_ZABBIX_WEB_PORT_80_TCP_PROTO=tcp
ZABBIX_ZABBIX_WEBSERVICE_SERVICE_PORT_WEBSERVICE=10053
ZABBIX_ZABBIX_AGENT_SERVICE_HOST=10.110.37.11
DB_SERVER_HOST=zabbix-postgresql
ZABBIX_POSTGRESQL_PORT_5432_TCP_PORT=5432
ZABBIX_ZABBIX_SERVER_PORT_10051_TCP_PROTO=tcp
ZABBIX_ZABBIX_SERVER_PORT_10051_TCP_ADDR=10.103.230.210
ZABBIX_ZABBIX_WEB_SERVICE_HOST=10.105.120.42
ZBX_VERSION=6.2.6
ZABBIX_ZABBIX_WEB_PORT_80_TCP_ADDR=10.105.120.42
ZABBIX_ZABBIX_AGENT_PORT=tcp://10.110.37.11:10050
ZABBIX_POSTGRESQL_SERVICE_PORT_DB=5432
ZABBIX_ZABBIX_WEBSERVICE_PORT_10053_TCP_PORT=10053
ZABBIX_POSTGRESQL_PORT_5432_TCP_ADDR=10.107.32.53
ZABBIX_ZABBIX_WEBSERVICE_PORT_10053_TCP=tcp://10.103.95.238:10053
ZABBIX_POSTGRESQL_SERVICE_PORT=5432
ZABBIX_POSTGRESQL_PORT_5432_TCP_PROTO=tcp
ZABBIX_ZABBIX_SERVER_PORT_10052_TCP_ADDR=10.103.230.210
ZABBIX_POSTGRESQL_PORT_5432_TCP=tcp://10.107.32.53:5432
ZABBIX_ZABBIX_WEB_PORT=tcp://10.105.120.42:80
ZABBIX_POSTGRESQL_SERVICE_HOST=10.107.32.53
ZABBIX_ZABBIX_WEB_SERVICE_PORT=80
DB_SERVER_PORT=5432
ZABBIX_ZABBIX_SERVER_SERVICE_PORT_ZABBIX_JMX=10052
ZABBIX_ZABBIX_WEBSERVICE_PORT=tcp://10.103.95.238:10053
ZABBIX_ZABBIX_SERVER_PORT_10051_TCP_PORT=10051
ZABBIX_ZABBIX_AGENT_PORT_10050_TCP=tcp://10.110.37.11:10050
ZABBIX_POSTGRESQL_PORT=tcp://10.107.32.53:5432
ZABBIX_ZABBIX_SERVER_SERVICE_PORT=10051
ZABBIX_ZABBIX_WEB_SERVICE_PORT_ZABBIX_WEB=80
ZABBIX_ZABBIX_SERVER_PORT_10052_TCP_PROTO=tcp
ZABBIX_ZABBIX_AGENT_SERVICE_PORT=10050
ZABBIX_ZABBIX_AGENT_SERVICE_PORT_ZABBIX_AGENT=10050
ZABBIX_ZABBIX_SERVER_SERVICE_HOST=10.103.230.210
ZABBIX_ZABBIX_AGENT_PORT_10050_TCP_ADDR=10.110.37.11
ZABBIX_ZABBIX_SERVER_SERVICE_PORT_ZABBIX_SERVER=10051
ZABBIX_ZABBIX_SERVER_PORT_10052_TCP_PORT=10052
ZABBIX_ZABBIX_SERVER_PORT_10052_TCP=tcp://10.103.230.210:10052
ZABBIX_ZABBIX_WEB_PORT_80_TCP=tcp://10.105.120.42:80
ZABBIX_ZABBIX_AGENT_PORT_10050_TCP_PORT=10050
ZABBIX_ZABBIX_SERVER_PORT=tcp://10.103.230.210:10051
ZABBIX_ZABBIX_WEBSERVICE_SERVICE_PORT=10053
POSTGRES_DB=zabbix
ZABBIX_ZABBIX_WEB_PORT_80_TCP_PORT=80
ZABBIX_ZABBIX_WEBSERVICE_PORT_10053_TCP_ADDR=10.103.95.238

[sa-ChristianAnton]

Oh, that’s Simple! Since Zabbix 6.0 and with the new feature of Zabbix Server High Availability, if one or both of the options HANodeName and HANodeAddress are set in true Zabbix SERVER, then the web interface gathers the address of the active instance of the Zabbix server to connect to from the database. This way it can always connect the active one.

[BGmot]

You are genius @sa-ChristianAnton ! How could I forget about HA?! I am sorry @aeciopires for wasting your time. I spent quite some time as well trying to figure out this before asking here.

One last question if you don't mind gentlemen: how do you export container logs? If I were to use Zabbix in k8s in production I'd love to have all the logs somewhere.

[aeciopires]

Thank you very much @sa-ChristianAnton! I did not know that.

@BGmot in my case, I use EKS (Kubernetes managed by AWS) and GKE (Kubernetes managed by GCP) and I see the logs of containers in CloudWatch (AWS) and Log Explorer (GCP) or use kubectl logs or Lens (Kubernetes IDE).

After a few days, the logs are removed. We do not store logs for long weeks.

But I know of people who store the logs in Elastic Stack or Graylog, exporting the logs using a logstash as a sidecar

[fibbs]

You are genius @sa-ChristianAnton ! How could I forget about HA?! I am sorry @aeciopires for wasting your time. I spent quite some time as well trying to figure out this before asking here.

Thank you. I am happy to help.

One last question if you don't mind gentlemen: how do you export container logs? If I were to use Zabbix in k8s in production I'd love to have all the logs somewhere.

Well, we (as in the developers of the Helm Chart) don't do anything special with the logs at all. The reason is the concept: in Kubernetes world, containers log directly to stdout and the platform collects these logs, stores them on disk of the nodes for a certain (relatively short) while and deletes all the old clutter (rotation). If you want more, there are several possibilities: the cloud hyperscalers usually offer some kind of log platform, or if you are on your own, you can use Grafana Loki which is a great companion to Grafana for metrics. Grafana Loki is quite easy to ramp-up and collects and historizes logs of containers based on custom ressources objects of type PodLogs, which instruct the Loki stack to collect logs from containers meeting the criteria written in the CRD objects.

[BGmot]

Thanks for quick replies! I'd be comfortable to send logs to Elastic or even to old good syslogd. But Zabbix configuration option LogType allows only file, system and console. So I somehow need to digest containers' stdout. But you are right this is not related to this helm, I need to learn more about kubernetes. Thank you.

[aeciopires]

Hi @BGmot!

For future questions or doubts, we can start use the discussion feature of Github: https://github.com/zabbix-community/helm-zabbix/discussions/landing.

I believe this can help others people.

[BGmot]

Great idea! Thank you.