Loki query: when a query contains more filters, more logs are returned in some cases

[ythsu]

Describe the bug In my dataset, when a query contains more filters, more logs are returned in some cases. For example, the query {container="my-container", namespace="my-ns"} |= " 401 " returned 78 logs; while the query with one more filter, like {container="my-container", namespace="my-ns"} |= " 401 " |= "1.2.3.4", returned 211 logs in the same time range. (The labels and IP in above queries are mocked to prevent information leak) However, not every query in this pattern results in incorrect outcome. I can get expected results in other cases.

To Reproduce Steps to reproduce the behavior:

1. Query loki with expression X: {container="my-container", namespace="my-ns"} |= " 401 "
2. Query loki with expression Y, which includes an additional filter: {container="my-container", namespace="my-ns"} |= " 401 " |= "1.2.3.4"

Expected behavior Since expr X has fewer filters, I expected the number of results from expr X is larger than or equal to the number of results from expr Y.

Environment:

• Infrastructure: Kubernetes, loki v2.3.0
• Deployment tool: helm

Screenshots, Promtail config, or terminal output {container="my-container", namespace="my-ns"} |= " 401 "

{container="my-container", namespace="my-ns"} |= " 401 " |= "1.2.3.4"

More Information

• I have tried the query {container="my-container", namespace="my-ns"} |= "1.2.3.4", it returned 211 lines.
• The results are the same in loki v2.2.1 and v2.3.0.
• I can confirm that the backend storage, openstack swift, contains all logs.
• Both index and data are stored in openstack swift.
• There are 3 ingesters, 3 queriers, a memcached cluster for chunk, a memcached cluster for index, 3 distributers, 3 gateways, 2 query frontends and 1 compactor in my k8s env. The client side is promtail.
• I have restarted all querier pods and two memcached clusters, but the result is the same.
• Debug log from pod querier shows that, both queries have the same total processed lines, total processed bytes and total chunks ref. So I infer that they retrieved the same data chunks.

[DylanGuedes]

Out of curiosity: were you querying this from the frontend, right? If so, do you mind also querying it directly from a querier too? Trying to isolate that the issue is within the query-frontend and not in other parts.

[mohamedmansour]

The same thing for me in the frontend, I am basically doing {unit="prometheus.service"} in grafana, and when I change the time range, it doesn't respect the time range, this has happened recently. I cannot get the last 5 minutes of logs. When I inspect the log it says:

from:"1667167963719"
to:"1667168263719"

Is this related?