behlendorf
commented
6 years ago @twarberg thanks for reporting this issue. Based on the stacks you provided it appears that somehow one of the global kmalloc-* slabs (as shown by slabtop) has been damaged. This resulted in the BUG which both zfs and ext4 encountered as well as several other processes on the system.
Unfortunately anything running in kernel space could have potentially caused this issue and we don't have any clear evidence. All we know is that the l2arc was the first victim. After inspecting the l2arc code I don't see how it could have caused this, but since you have good suspicions let me suggest two things you could try to help us narrow this down.
1) If possible disable the l2arc entirely in your environment to rule it out as a cause, or
2) Continue to use the l2arc and set the spl_kmem_cache_slab_limit module option to 0. This will move the memory allocations from the kernel slab implementation to the spl slab implementation which should provide us some additional debug information if the issue repeats.
twarberg
adam-dej
System information
Describe the problem you're observing
We're running on Google Cloud and is trying to use an local NVMe SSD for L2arc but run into the crash after 7-12 days. We've seen it on 2 different systems a MySQL and a Postgres server. It happened on the Postgres server with both 0.7.1 and 0.7.2 with about a weeks interval (2x on 0.7.1 and once on 0.7.2). Unfortunately we do not have console logs for those which is why we haven't reported it until now. We decided to give 0.7.3 a try after upgrading the MySQL server and got a crash after 10 days. Both systems are busy 24/7 with the Postgres being write heavy and the MySQL read heavy. Postgres is at least 2x as disk active. The Postgres server has been running stable without L2arc for months.
Common settings
Describe how to reproduce the problem
Run a busy MySQL or Postgres on Google Cloud instance with local NVMe SSD L2arc for 7-12 days.
Include any warning/errors/backtraces from the system logs