Open kilo-gi opened 4 months ago
This has been observed by others on the community forums:
https://community.grafana.com/t/line-format-reduces-log-count-to-only-a-few/119730
Hi.
Thanks @kilo-gi for providing specific inputs that triggers this behavior.
I believe I have identified the commit that introduced this behavior: https://github.com/grafana/loki/commit/793a689d1ffd78333fe651d81addfaa180ae8886
I crafted a set of 6 records, with the following behavior:
Generates 6 records on both commits, when queried with
LOKI_ADDR="http://localhost:65100" ./praccu-scratch/logcli-linux-amd64 query \
'{image="synthetic_15"} |= ``' -o jsonl --quiet
Generates 6 records on commit d24fe3e68be4be16b3654e6aedafaa71ac9d307c [0] when queried with
LOKI_ADDR="http://localhost:65100" ./praccu-scratch/logcli-linux-amd64 query \
'{image="synthetic_15"} |= `` | json | line_format "."' -o jsonl --quiet
Generates 2 records on commit 793a689d1ffd78333fe651d81addfaa180ae8886 [1] with
LOKI_ADDR="http://localhost:65100" ./praccu-scratch/logcli-linux-amd64 query \
'{image="synthetic_15"} |= `` | json | line_format "."' -o jsonl --quiet
The way I was able to trigger this was to:
Example data: synthetic.15.json
Script that generates the synthetic data:
import time
_TEMPLATE = """{"text": "2024-06-06 14:48:12.183 |INFO | 9500__2024_06_06_21_43_32_358250__P000_W0000/C0000__user_100.74.76.3_LINE_REPLACE - _lsh_central_server.score_and_build_index_step logged_runtime: 0.4835s", "record": {"elapsed": {"repr": "0:00:13.271696", "seconds": 13.271696}, "exception": null, "extra": {"command_key": "COMMAND_KEY", "machine": "user_100.74.76.3_5057", "container": "fcfa9e6dd7792bd76d9fc777dabf828e281f8969cfd357bd62b14688b5d7cb7f", "user": "buildbot", "image": "SYNTHETIC", "run": "buildbot_job", "command_label": "computronium", "hammer_id": null, "local_rank": null, "global_rank": null, "relaunch_name": null, "full_location": ""}, "file": {"name": "log_utils.py", "path": "/opt/projects/generally_intelligent/computronium/computronium/common/log_utils.py"}, "function": "log_runtime", "level": {"icon": "ℹ️", "name": "INFO", "no": 20}, "line": 449, "message": "_lsh_central_server.score_and_build_index_step logged_runtime: 0.4835s", "module": "log_utils", "name": "computronium.common.log_utils", "process": {"id": 9271, "name": "MainProcess"}, "thread": {"id": THREAD_ID, "name": "MainThread"}, "time": {"repr": "2024-06-06 14:48:12.183338-07:00", "timestamp": TIMESTAMP}}}"""
def fill_record(synthetic, timestamp, command_key, line_diff):
record = _TEMPLATE
record = record.replace("THREAD_ID", str(line_diff))
record = record.replace("SYNTHETIC", f"synthetic_{synthetic}")
record = record.replace("COMMAND_KEY", f"9500__2024_06_06_21_43_32_358250__P000_W0000/C0000__user_100.74.76.3_{command_key}")
record = record.replace("TIMESTAMP", str(timestamp))
#record = record.replace('LINE_REPLACE', str(line_diff))
return record
if __name__ == "__main__":
ts = time.time()
ts2 = ts + 1.0
ck = "ck0"
ck2 = "ck1"
synthetic = 15
with open(f"example_logs/synthetic.{synthetic}.json", "w") as out:
for i, (ts_out, ck_out) in enumerate([(ts, ck), (ts, ck2), (ts2, ck), (ts2, ck2), (ts2, ck), (ts2, ck)]):
print(fill_record(synthetic, ts_out, ck_out, i), file=out)
I think the following block of code is the most likely culprit: https://github.com/grafana/loki/commit/793a689d1ffd78333fe651d81addfaa180ae8886#diff-89016a3dad2dc806b1c9db8e6232914701a99a7908ba0ac37742b27d03e8f618L233
[0] https://hub.docker.com/layers/grafana/loki/main-d24fe3e-amd64/images/sha256-71016ac17acb8f5d8d706a94d5405e05728d038d65a857b1ca77aaa027a8e59d?context=explore [1] https://hub.docker.com/layers/grafana/loki/main-793a689-amd64/images/sha256-3a20c3e72dbd4734e19ee32910811512e2d027387f6fc6079f32e37fb4f86558?context=explore
To clarify why I generated the data I generated: in the data @kilo-gi shared, the 7 dropped records all have the basic behavior of identical timestamps and lines, which AFAIK is a proxy for identical timestamps and lines in this case.
Studying this, I am wondering if this isn't a bug, but rather a bugfix, and if head is WAI? Is (timestamp, line) considered sufficient to uniquely identify a record?
Describe the bug In v3.0.0, Loki will quietly drop some log lines in some cases when the query includes a line format.
To Reproduce Steps to reproduce the behavior:
saved checkpoint at batch 75
| json | line_format.
'saved checkpoint at batch 75
| json'Expected behavior Neither the version nor the line format pipeline directive should change the count of logs received.
Environment:
Deployment tool: Running the images directly in docker, starting them via ansible.
It's a single replica deployment.
Screenshots, Promtail config, or terminal output I've posted screenshots and dumps of logs in the community slack: https://grafana.slack.com/archives/C06RUQJHTHQ/p1713891805440339
Here's the loki config we're using for v3.0.0:
Querying v3.0.0 without the line format gives 1800 lines: https://files.slack.com/files-pri/T05675Y01-F06V8NGCVB9/download/screenshot_2024-04-23_at_12.03.56___pm.png?origin_team=T05675Y01 https://files.slack.com/files-pri/T05675Y01-F0706C4AS85/download/explore-v3.0.0-no-line-format.json?origin_team=T05675Y01
v3.0.0 with the line format gives 1793 lines: https://grafana.slack.com/files/U06V4UJKA7N/F0706CB2A8M/screenshot_2024-04-23_at_12.05.42___pm.png?origin_team=T05675Y01&origin_channel=C05675Y4F https://files.slack.com/files-pri/T05675Y01-F070Y3GHSDN/download/explore-v3.0.0-with-line-format.json?origin_team=T05675Y01
v2.7.5 without the line format is 1800 lines: https://grafana.slack.com/files/U06V4UJKA7N/F0709A4M34J/screenshot_2024-04-23_at_12.06.52___pm.png?origin_team=T05675Y01&origin_channel=C05675Y4F https://files.slack.com/files-pri/T05675Y01-F070Y3NP4U8/download/explore-v2.7.5-no-line-format.json?origin_team=T05675Y01
And v2.7.5 with the line format is 1800 lines: https://grafana.slack.com/files/U06V4UJKA7N/F0706CTN38V/screenshot_2024-04-23_at_12.08.55___pm.png?origin_team=T05675Y01&origin_channel=C05675Y4F https://files.slack.com/files-pri/T05675Y01-F070Y3ZBERW/download/explore-v2.7.5-with-line-format.json?origin_team=T05675Y01
To get good diffs you have to remove all the "service_name": lines fields and format the json. With that done: