Open FatBoyXPC opened 1 month ago
This does not currently exist, but it would be very handy indeed!
Would be a good addition to have. Just now it's also possible to run disko with existing partitions, but we have not covered all edge cases yet. So there might be cases where it eats your data.
I feel like a proper implementation of this would also open the door to safer idempotent --mode format
. It's even kind-of required to make sure the implementations for serializing the current state and making only necessary changes don't drift apart.
Basically, my idea would be to try to serialize the current state of drives into a JSON first. From this, a Nix attrset can be generated trivially. That's what this feature is about.
As a follow-up, when running disko, it would do the same serialization and then diff that against the final attrset of the current configuration. Only the differences would then be taken into account when formatting.
Maybe this should even be a new mode.
As for how to implement this, I would say the _create
attribute for each type should be joined by a _serialize
. This doesn't work for the top level, but once we found the drives, the partitions on them and their types, we can dispatch to each of those to figure out the values of their options.
For the follow-up, a _modify
attribute would probably be the way to go.
Where would you store this json file?
In a temporary directory or a fifo even. It should be regenerated for both the disko config and the real state on every run.
Ok, so I played around a little bit, and it seems this isn't quite as easy as I initially imagined.
The main problem is that the current structure of disko heavily depends on evaluation the config
value.
Dispatching to the filesystem-specific modules is also done by recursing into the configuration attributes. This means that to generate a config, you have to
types/gpt.nix
)zfs.nix
, ext.nix
etc.)This seems to be the only way of achieving this I can come up with.
After a bit of hacking, the first stage is basically this jq script:
.blockdevices | {
disk: (reduce .[] as $disk ({};
.[($disk.children.[0].partlabel | split("-") | .[1]) // $disk.kname] = {
device: $disk.path,
type: "disk",
content: ({
type: $disk.pttype
})
}
))
}
Running lsblk --output KNAME,PATH,PARTLABEL,PTTYPE --tree --json | jq -f query-disks.jq
on my machine, I get this output:
{
"disk": {
"scratch_drive": {
"device": "/dev/sda",
"type": "disk",
"content": {
"type": "gpt"
}
},
"tank_2": {
"device": "/dev/sdb",
"type": "disk",
"content": {
"type": "gpt"
}
},
"tank_1": {
"device": "/dev/sdc",
"type": "disk",
"content": {
"type": "gpt"
}
},
"nix_store": {
"device": "/dev/sdd",
"type": "disk",
"content": {
"type": "gpt"
}
},
"boot_drive": {
"device": "/dev/sde",
"type": "disk",
"content": {
"type": "gpt"
}
}
}
}
This converts easily to a nix attrset with builtins.fromJSON
:
$ nix eval --expr "builtins.fromJSON ''$(lsblk --output KNAME,PATH,TYPE,LABEL,PARTLABEL,PTTYPE,FSTYPE,SIZE,MOUNTPOINTS --tree --json | jq -f query-disks.jq)''"
{ disk = { boot_drive = { content = { type = "gpt"; }; device = "/dev/sde"; type = "disk"; }; nix_store = { content = { type = "gpt"; }; device = "/dev/sdd"; type = "disk"; }; scratch_drive = { content = { type = "gpt"; }; device = "/dev/sda"; type = "disk"; }; tank_1 = { content = { type = "gpt"; }; device = "/dev/sdc"; type = "disk"; }; tank_2 = { content = { type = "gpt"; }; device = "/dev/sdb"; type = "disk"; }; }; }
So that's a start.
Some caveats I already discovered:
$disk.children.[0].partlabel | split("-") | .[1]
part above. This only works if no dashes are in the disk or partition names!/
and /var/lib/containers/storage/overlay
as the mountpoint for my root drivemountOptions
can be determined from /etc/fstab
. Not sure if there's a better way
x-initrd.mount
and defaults
options should be ignored, though. The former is created by NixOS to mount the drives before systemd runsby-id
path for each disk instead of the /dev
path (which would be /dev/disk/by-id/ata-Samsung_SSD_850_EVO_250GB_S21PNSAG237083K
for dev/sda
, for example), but to reliably get that information, you have to query udevadm, and there can by multiple such links, so the user would have to choose which one they prefer, or we implement some heuristicnix_store
with end = "-8G" and
swapwith
size = "100%", but
lsblkobviously reports
224.9Gand
8G` insteadextraArgs
, it's only relevant during creation and it seems to me that it's difficult to accurately determine these from an existing partitiontype = "EF00";
. For regular linux filesystems, this isn't necessary, for some others it might be
sgdisk -L
, but that doesn't map to GPT's GUIDsgdisk -l /dev/sde
8300
:7f05 ChromeOS hibernate 8200 Linux swap
8300 Linux filesystem 8301 Linux reserved
8302 Linux /home 8303 Linux x86 root (/)
8304 Linux x86-64 root (/) 8305 Linux ARM64 root (/)
8306 Linux /srv 8307 Linux ARM32 root (/)
8308 Linux dm-crypt 8309 Linux LUKS
830a Linux IA-64 root (/) 830b Linux x86 root verity
830c Linux x86-64 root verity 830d Linux ARM32 root verity
830e Linux ARM64 root verity 830f Linux IA-64 root verity
8310 Linux /var 8311 Linux /var/tmp
8312 Linux user's home 8313 Linux x86 /usr
8314 Linux x86-64 /usr 8315 Linux ARM32 /usr
8316 Linux ARM64 /usr 8317 Linux IA-64 /usr
8318 Linux x86 /usr verity 8319 Linux x86-64 /usr verity
831a Linux ARM32 /usr verity 831b Linux ARM64 /usr verity
831c Linux IA-64 /usr verity 8400 Intel Rapid Start
That's it. I probably won't have more time to work on this for the next weeks, unfortunately.
Would it be easier at some point to just dump all the json and have a language that is more high-level than bash to use this as an input and compare it with reality? The error handling in bash is not very good and the templating at some stage also becomes not so readable.
Yes, 100%. I feel like disko in general might be easier to maintain if instead of letting nix generate the final script, we "just" wrote a program that runs nix eval -f disko.nix --json
and then works from those values, but that's a whole separate discussion, and there might be good reasons against it.
In terms of which language to use, do you or @Lassulus have a preference? If we're just talking about a single script, python seems to be the way to go, but if we want to expand the functionality further and actually make this a solid tool that uses as little bash as possible, Rust or Go could be good options as well.
Is an option like this available? It would be pretty handy to be able to run a command that reads the current disk(s) specified and outputs to a
disk-config.nix
. One example of why this is handy is if I want to install nix into a laptop that's already dual booting windows and arch. I'd rather not blow away the windows partition as sometimes windows can be helpful in troubleshooting hardware.