Wrong subfields, from previous logs

[SemperPeritus]

For some reason, I am getting incorrect subfields for queries from Grafana. These same fields are absent (as they should be) when making an identical queries in VMUI.

In Grafana, fields are present that appeared in the previous log and remain until they receive a new actual (real) value in the following log. The data displayed in Logs and DataFrame JSON (from Query) are identical.

I'm not sure if the issue is with the plugin or VictoriaLogs because I don't know exactly how Grafana retrieves the data.

Grafana:

{
  "_om": "172.18.0.3 - - [19/Aug/2024:04:32:29 +0000] \"POST /jsrpc.php?output=json-rpc HTTP/1.1\" 200 74 \"https://zabbix.host/zabbix.php?action=dashboard.view&dashboardid=1\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36\" \"[IP]\"",
  "_parser_error": "",
  "_parser_format": "combined",
  "_parser": "nginx",
  "_stream_id": "000000000000000035198090cc8cb3dd03c99de6717fec2e",
  "container_created_at": "2024-06-27T15:18:08.051113458Z",
  "container_id": "be7f6654dfa7d46b1462559cb5eec3d79552ea6779b4d3068837e56a9497c03a",
  "container_name": "zabbix-server-zabbix-web-nginx-pgsql-1",
  "environment": "production",
  "host": "cloud-fsn1",
  "image": "zabbix/zabbix-web-nginx-pgsql:ubuntu-7.0-latest",
  "log.agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
  "log.cid": "6179",
  "log.client": "172.18.0.3",
  "log.compression": "[IP]",
  "log.host": "prometheus.host",
  "log.message": "user \"admin\": password mismatch",
  "log.pid": "633",
  "log.referer": "https://zabbix.host/zabbix.php?action=dashboard.view&dashboardid=1",
  "log.request": "POST /jsrpc.php?output=json-rpc HTTP/1.1",
  "log.server": "prometheus.host",
  "log.severity": "error",
  "log.size": "74",
  "log.status": "200",
  "log.tid": "633",
  "log.timestamp": "2024-08-19T04:32:29Z",
  "message": "",
  "source_type": "docker_logs",
  "stream": "stdout"
}

VMUI:

{
  "_msg": "172.18.0.3 - - [19/Aug/2024:04:32:29 +0000] \"POST /jsrpc.php?output=json-rpc HTTP/1.1\" 200 74 \"https://zabbix.host/zabbix.php?action=dashboard.view&dashboardid=1\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36\" \"[IP]\"",
  "_om": "172.18.0.3 - - [19/Aug/2024:04:32:29 +0000] \"POST /jsrpc.php?output=json-rpc HTTP/1.1\" 200 74 \"https://zabbix.host/zabbix.php?action=dashboard.view&dashboardid=1\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36\" \"[IP]\"",
  "_parser_format": "combined",
  "_parser": "nginx",
  "_stream_id": "000000000000000035198090cc8cb3dd03c99de6717fec2e",
  "_stream": "{container_name=\"zabbix-server-zabbix-web-nginx-pgsql-1\",host=\"cloud-fsn1\",source_type=\"docker_logs\"}",
  "_time": "2024-08-19T04:32:29.532715333Z",
  "container_created_at": "2024-06-27T15:18:08.051113458Z",
  "container_id": "be7f6654dfa7d46b1462559cb5eec3d79552ea6779b4d3068837e56a9497c03a",
  "container_name": "zabbix-server-zabbix-web-nginx-pgsql-1",
  "environment": "production",
  "host": "cloud-fsn1",
  "image": "zabbix/zabbix-web-nginx-pgsql:ubuntu-7.0-latest",
  "log.agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
  "log.client": "172.18.0.3",
  "log.compression": "[IP]",
  "log.referer": "https://zabbix.host/zabbix.php?action=dashboard.view&dashboardid=1",
  "log.request": "POST /jsrpc.php?output=json-rpc HTTP/1.1",
  "log.size": "74",
  "log.status": "200",
  "log.timestamp": "2024-08-19T04:32:29Z",
  "source_type": "docker_logs",
  "stream": "stdout"
}

Difference:

{
  "log.cid": "6179",
  "log.host": "prometheus.host",
  "log.message": "user \"admin\": password mismatch",
  "log.pid": "633",
  "log.server": "prometheus.host",
  "log.severity": "error",
  "log.tid": "633",
}

[serrrios]

+1

[sydevinfra]

I have the same issue. Grafana : v11.1.3 Datasource : v0.4.0 VictoriaLogs : v0.28.0

It happens when the query results inlude several streams with different fields In vmui the logs are grouped by stream and each stream have it's own set of fields In Grafana all the logs have all the fields, (if one is missing it grabs the value from the next log that have it, as described)

[hagen1778]

@dmitryk-dk could you please take a look?

[dmitryk-dk]

Hi @SemperPeritus ! The issue was fixed in the release. Please reopen issue if you find any problem