Open bnaecker opened 10 months ago
If you want to preserve the contents, I think the basic pattern is:
debug
to debug-old
debug-old
to debug
, unlinking them as you godebug-old
directory once emptyNB: You cannot merely rename the files from debug-old
to debug
, unfortunately, because it is a cross-file system operation, and links cannot span file systems.
I don't think this has the same underlying cause as #4269. In that case, the dataset rpool/zone
was created with -o zoned=on
, which prevented it from being mounted. Looking at zpool history
on the dogfood rack, where we see this name conflict, I don't see that option being set on the dataset creation:
BRM42220051 # zfs get -Hpo value name /pool/ext/e6d2fe1d-c74d-40cd-8fae-bc7d06bdaac8/crypt/debug/
oxp_e6d2fe1d-c74d-40cd-8fae-bc7d06bdaac8/crypt
BRM42220051 # zpool history oxp_e6d2fe1d-c74d-40cd-8fae-bc7d06bdaac8 | grep 'zfs create .*/debug'
1986-12-28.16:21:37 zfs create -o mountpoint=/pool/ext/e6d2fe1d-c74d-40cd-8fae-bc7d06bdaac8/crypt/debug oxp_e6d2fe1d-c74d-40cd-8fae-bc7d06bdaac8/crypt/debug
BRM42220051 #
So this dataset is not zoned, which is good.
While working on some zone bundle improvements, I noticed that the debug datasets we create on the U.2s, which are used for cores and archived logs, are not actually mounted.
When the sled-agent starts up, we create a hierarchy of datasets on each U.2. These are structured like
<pool_name>/crypt
, with child datasets underneath, such aszone
ordebug
. The<pool_name>/crypt/zone
datasets have further children under them, which are the root filesystems for each zone we launch in the control plane. As an example, from the current dogfood rack, we have:The
debug
dataset is intended for cores, crash dumps, and also archived logs from the zones. Here is where it's supposed to be mounted:And there are indeed directories there:
However, if we look at the dataset that those directories actually belong to, we see this:
I.e., they are in the base
crypt
dataset, not the expected one atcrypt/debug
. And in fact, those directories do not show up in/etc/mnttab
:And just to double-check, they are in fact not mounted:
Also, we appear to still be archiving logs into those debug directories, in the
crypt
dataset:And we can see there are many files in the archive directory implied by that last Propolis zone name:
There are also core files in those locations:
To summarize:
So, it appears that those directories have been created at some prior point, which prevents ZFS from automounting the
.../crypt/debug
dataset over them. It's not totally clear to me what the right path forward here is. Deleting those directories is needed for them to be automounted, but we also don't want to necessarily blow away any of the existing debug data.