Closed VladRodionov closed 10 years ago
There is no way to mitigate this issue. We can not retry file write operation because we do not know how was the last (failed) write. The failed write might have succeeded partially, but data might be corrupted?
The recommendation: do not over-allocate cache partition.
Added check on cache partition, if usable space becomes less than ( 1 - storageHighWatermark) * totalPartitionSpace then we purge the oldest file from cache.
Configuration parameters are: storage.recycler.high.watermark (default, 0.98) storage.recycler.low.watermark (defualt, 0.97)
To increase usable partition space one can use tune2fs command
Found the explanation for the discrepancy between df -h /cache and du -s /cache
[ec2-user@ip-10-146-230-202 test]$ df -h /cache
Filesystem Size Used Avail Use% Mounted on
/dev/md0 148G 142G 3.5G 98% /cache
[ec2-user@ip-10-146-230-202 test]$ du -s -h /cache
du: cannot read directory ‘/cache/lost+found’: Permission denied
97G /cache
The most common cause of this effect is open files that have been deleted.
The kernel will only free the disk blocks of a deleted file if it is not in use at the time of its deletion. Otherwise that is deferred until the file is closed, or the system is rebooted.
Therefore, before deleting old cache file we must guarantee that there are no open file descriptors hanging around, then close it, then delete it. To check if we have deleted files hanging around:
sudo lsof 2>/dev/null | grep deleted
Finally, did it right. It took some time to figure out the reason.
Final 24h drive test on AWS showed no partition space leakage. Fixed.
We need special treatment for this error: DISK STORAGE =137686247146 ITEMS IN STORE=15840571 14/05/25 12:31:08 FATAL storage.FileExtStorage: java.io.IOException: No space left on device java.io.IOException: No space left on device at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:51) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205) at com.koda.integ.hbase.storage.FileExtStorage$FileFlusher.run(FileExtStorage.java:303) 14/05/25 12:31:08 FATAL storage.FileExtStorage: file-flusher thread died. 14/05/25 12:31:08 FATAL storage.FileExtStorage: java.io.IOException: No space left on device java.io.IOException: No space left on device at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:51) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205) at com.koda.integ.hbase.storage.FileExtStorage$FileFlusher.run(FileExtStorage.java:303) 14/05/25 12:31:08 FATAL storage.FileExtStorage: file-flusher thread died.
Couple notes: