joe-elliott
commented
1 year ago Thanks for filing the issue. Not having any error logs is quite frustrating as it would be the most likely source of good information here. Some things we can do:
- Check
tempo_ingester_blocks_flushed_totalto see if it is flushing blocks at the same rate. - Check
tempo_ingester_failed_flushes_totalto see if it we are metric'ing failed flushes but for some reason not logging info. - Check
tempodb_backend_request_duration_seconds_countto see if there's any network interruption between storage and the ingester. - Dump the contents of the
/var/tempo/folder (or wherever you are storing ingester data). Let's see if the wal files or the complete blocks are building up (or both).
Is the ingester unexpectedly restarting? or even gracefully restarting before this issue occurs? I have seen something similar once, but it occurred after an ingester OOMed and then had issues reading its wal on restart.
We have also seen issues in AKS where mounted drives will swap to read only. Perhaps try to manually delete a file from the drive when this issue is occurring? Tempo should log if it fails to remove a file, but I think it's still a good test.
LasseHels
liam-howe-maersk
zhuoyuan-liu
github-actions[bot]
Describe the bug Our ingesters'
PersistentVolumeClaims intermittently fill up, seemingly at random.To Reproduce We have not been able to consistently reproduce the issue.
Expected behavior We would expect that the used space on the
PersistentVolumeClaimremains fairly consistent, and does not spike.Environment:
Additional Context We run a total of nine ingesters; three in each zone across three zones.
For the past couple of months, the used space of an ingester's PVC will start gradually increasing. This gradual increase continues until it approaches 100%, at which point we have to manually intervene.
Here are a couple of screenshots of the PVC space usage when it happens:
2023-02-21T00:25:00Zand2023-02-21T04:00:00Z, the PVC of one of our ingesters in zoneAwent from ~3GB of usage to ~19GB:2023-02-21T23:50:00Zand2023-02-22T02:50:00Z, the PVC of one of our ingesters in zoneCwent from ~ 3GB of usage to ~18.4GB:2023-02-12T20:20:00Zand2023-02-13T15:28:00Z, the PVC of one of our ingesters in zoneCwent from ~2GB of usage to ~18.9GB:We have observed this issue across different Tempo versions, namely
v1.5.0andv2.0.0(which we recently upgraded to).Initially, we figured that it was an issue with the flushing mechanism of the ingester, but preliminary log analysis has been unable to confirm this theory.
Using our first example from above (the ingester from zone
A): Error logs (excludingTRACE_TOO_LARGE) from the period before and after the increase in space usage:As we can see, no
errorlogs were emitted in the period leading up to the disk being full. The firsterrorlog emitted is at2023-02-21T04:02:34, which is when there was no space left:"error completing wal block with local backend: error creating block: write /var/tempo/wal/blocks/single-tenant/f291dded-9dea-4dfa-985b-5984c5421f49/data.parquet: no space left on device"For the record, I am also attaching the graph of all logs (excluding
TRACE_TOO_LARGE):This graph shows two spikes: one at ~
2023-02-21T04:00:00Z, and a second one at ~2023-02-21T06:30:00Z. The first spike here are theno space left on deviceerror messages; the second spike are logs from the manual intervention.We would be much appreciative of any help with this issue, as we are somewhat stuck at the moment, and are unsure of how to proceed.
Please let me know of any additional information that may be of use in debugging the issue. I will make sure it is swiftly provided.
As a final note: I have created this issue as a bug, but I appreciate the fact that the problem can just as well be a misconfiguration on our side as it can be a bug in Tempo itself. Our team is very happy to use Tempo in general, and this is a nut that we cannot seem to crack on our own.
Thanks.