rincebrain
commented
3 years ago How large is the pool (used/total)? How much RAM is the ARC configured to use, out of how much on the VM in total? How fast is the storage? OOMing that fast implies either almost no RAM or reasonably quick storage (or both), assuming it's the ARC filling that's exerting all that memory pressure. If you want to get the VM running again, the things I'd try would be:
- simplest would be importing the pool on another system and issuing a
zpool scrub -por-sas desired. - I think you could reach into the VM and create an /etc/modprobe.d/zfs.conf with something like
options zfs zfs_no_scrub_io=1to make scrub not actually scrub (though if just the metadata crawl is enough to exhaust things, that won't save you) - You could also try something hackish like sticking
while true; do zpool scrub -p [thatpool] && break; done;in /etc/rc.local, though be sure to remove it before rebooting, aszpool scrub -p [pool]on a pool without a scrub running is also an error. (I just put thewhile truebecause I'm not sure what guarantees there are about when rc.local gets run relative to the pool import; if you know better than I that it gets run strictly later, you can drop it.)
omarkilani
behlendorf
System information
Describe the problem you're observing
I was doing a stress test on a zfs pool running Postgres. I left it running overnight and came back to a locked up VM. Nothing on the console from the lock up that I could see, but I suspect zfs was behind the lock up.
When I rebooted the VM, I ran scrub on the pool. The machine ran out of memory in about 5 seconds and the OOM kicked in, and eventually the machine rebooted.
If I import the pool again the scrub kicks off again automatically and the machine runs out of memory again.
Will try 2.0.4 soon.
Describe how to reproduce the problem
Include any warning/errors/backtraces from the system logs