leftwo
commented
2 years ago On sock, after an uninstall, there are still crucible regions. You can see these from zfs list:
alan@sock:crucible$ zfs list -o name | grep crucible
oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible
oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions
oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b/crucible/regions/439a26ff-e6dc-4794-a8c8-2ace6d175616
oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible
oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions
oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/12ac64e4-c489-4e4b-8272-5a40847bdb28
oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible
oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions
oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/e0f5e6a3-80f5-4892-9bfa-16fd09ea1e7c
rpool/data/crucible
rpool/zone/oxz_crucible_oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
rpool/zone/oxz_crucible_oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
rpool/zone/oxz_crucible_oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03(Anything below oxp_.../crucible/regions/)
After ./tools/create_virtual_hardware.sh is run, the crucible zpools are created:
alan@sock:omicron$ zpool list -o name,size,alloc,free
NAME SIZE ALLOC FREE
oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b 49.5G 9.79G 39.7G
oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 49.5G 8.39G 41.1G
oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 49.5G 8.39G 41.1G
rpool 952G 452G 500GHowever, if ./tools/destroy_virtual_hardware.sh is not run, any regions (disks) created inside these pools will remain, even across install/uninstall of omicron.
I believe this is the desired behavior (to allow upgrades), but if one is doing development and testing, be away that old regions may come alive even if there is no upstairs for them to connect to. This can result in crucible-downstairs processes being started, and any disk space these regions are using will not be available.
Crucible appears to suffer the same fate
Originally posted by @leftwo in https://github.com/oxidecomputer/omicron/issues/1119#issuecomment-1170554806