kloyan
commented
3 years ago I investigated the issue a little further and I can provide additional information. The To part of the time range (or the end query param) is actually taken into account. However, the From part (or the start query param) seems to be ignored. As you can see in the following query, all log messages are before end=1603805151000000000, but some of them are also before start=1603805090000000000 which is wrong.
{
"request": {
"url": "api/datasources/proxy/2/loki/api/v1/query_range?direction=BACKWARD&limit=1000&query=%7Bapp%3D%22nginx-ingress%22%2C%20component%3D%22controller%22%7D&start=1603805090000000000&end=1603805151000000000&step=1",
"hideFromInspector": false
},
"response": {
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"component": "controller",
"filename": "/var/log/pods/kube-system_addons-nginx-ingress-controller-6cd497b99d-w4vvx_fbd9e8f0-00c4-4a37-ac8b-4ab7c53c2292/nginx-ingress-controller/0.log",
"job": "kube-system/nginx-ingress",
"pod_template_hash": "6cd497b99d",
"release": "addons",
"stream": "stdout",
"app": "nginx-ingress",
"container": "nginx-ingress-controller",
"namespace": "kube-system",
"pod": "addons-nginx-ingress-controller-6cd497b99d-w4vvx"
},
"values": [
[
"1603805150976103863",
"<DUMMY-MSG>"
],
[
"1603805147751147222",
"<DUMMY-MSG>"
],
[
"1603805143635594359",
"<DUMMY-MSG>"
],
[
"1603805140386420384",
"<DUMMY-MSG>"
],
[
"1603805137226062567",
"<DUMMY-MSG>"
],
[
"1603805137145685791",
"<DUMMY-MSG>"
],
[
"1603805137129129713",
"<DUMMY-MSG>"
],
[
"1603805136664735621",
"<DUMMY-MSG>"
],
[
"1603805135861734257",
"<DUMMY-MSG>"
],
[
"1603805135034548449",
"<DUMMY-MSG>"
],
[
"1603804991247843157",
"<DUMMY-MSG>"
],
[
"1603804982235876264",
"<DUMMY-MSG>"
],
[
"1603804982208278159",
"<DUMMY-MSG>"
],
[
"1603804982188696773",
"<DUMMY-MSG>"
],
[
"1603804975262965586",
"<DUMMY-MSG>"
],
[
"1603804975226670989",
"<DUMMY-MSG>"
],
[
"1603804975159244651",
"<DUMMY-MSG>"
],
[
"1603804964699517500",
"<DUMMY-MSG>"
],
[
"1603804964662348128",
"<DUMMY-MSG>"
],
[
"1603804964657599079",
"<DUMMY-MSG>"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 16750480,
"linesProcessedPerSecond": 31113,
"totalBytesProcessed": 43608,
"totalLinesProcessed": 81,
"execTime": 0.002603388
},
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0
},
"ingester": {
"totalReached": 2,
"totalChunksMatched": 1,
"totalBatches": 0,
"totalLinesSent": 0,
"headChunkBytes": 43608,
"headChunkLines": 81,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0
}
}
}
}
}
slim-bean
AMoghrabi
periklis
sandeepsukhani
pcbl
Bug Description Hello! Since upgrading from 1.6.1 to 2.0.0, any time range queries that I perform from Grafana (or directly from the Loki API) return all logs up until the configured max lines for the data source (e.g. 1000). The specified time range is not taken into account at all. The problem occurs for both relative and absolute time ranges. I'm running a single instance of monolith Loki 2.0.0 with GCS as a chunk store and boltdb-shipper as an index store. This setup worked fine in 1.6.1. If instead of boltdb-shipper, I switch to the vanilla boltdb, everything works as expected even on 2.0.0. During my testing, I deployed a clean Loki (empty bucket and persistent volumes) every time I changed the chunk/index store types.
To Reproduce Steps to reproduce the behavior:
Expected behavior Time ranged queries to work properly. Only logs that fall between the specified times should be displayed.
Environment:
Promtail config is the same as in the official helm chart. This is my Loki config:
I executed a range query while using boltdb-shipper and while using vanilla boltdb. Here are logs from Loki and request/response from the Grafana query inspector:
Logs when using GCS + boltdb-shipper (problem occurs)
Loki logs:
Query Inspector (Notice how only the first log message falls in the specified range):
Logs when using vanilla boltdb (problem does not occur)
Loki logs:
Query Inspector (all messages are in the specified range):
Please, let me know if you need more information.
Best regards, Kaloyan