frairon
commented
1 year ago It is what it says it is, the disk is just full. There is usually not more to it.
That does not necessarily mean that leveldb is filling up the disk, could be another process running in the same container. It really depends on your setup and the data.
E.g. if you have indefinite amount of keys and never clean them up, your disk will fill up eventually.
Maybe check the storage directory /mad/tables/ to see how much the table actually uses.
If you have access to the kafka cluster via shell, you can check if the bytes kafka uses on disk for this topic roughly align with the size on the processor disk. It's not going to be the same, but shouldn't drift apart in orders of magnitude.
You could also erase the disk and restart the processor. Or create a view on the processor table. Then it will refetch all values from kafka. But I doubt it would make a difference. We never had any issues with levelup cleaning up old data.
Let me know if that helped.
Recently I have encountered scenario where the processor is running out of space in the device . What are the steps that I can use to diagnose why LevelDB is running out of space and what is taking the additional space .
Few of the logs lines : ` Setup failed. Cannot start processor for partition 3: error setting up partition table: error building storage: error opening leveldb: write /mad/tables/orchestrator-table.3/MANIFEST-192032: no space left on device
`