search

oxidecomputer / crucible

A storage service.
Mozilla Public License 2.0
157 stars 16 forks source link

Region extent db file corruption after sled was rebooted with kmdb enabled #996

Open askfongjojo opened 10 months ago

askfongjojo commented 10 months ago

We have several downstairs on rack3 that hit this kind of error after the crucible zone came back from a sled reboot:

[ Oct  7 22:13:57 Disabled. ]
[ Oct  7 22:13:57 Enabled. ]
[ Oct  7 22:13:57 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/downstairs.sh"). ]
{"msg":"current number of open files limit 65536 is already the maximum","v":0,"name":"crucible","level":30,"time":"2023-10-07T22:13:57.390738293Z","hostname":"oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36","pid":8992}
{"msg":"Opened existing region file \"/data/regions/0a7868bd-9778-49c5-bc05-45892a9f7ec6/region.json\"","v":0,"name":"crucible","level":30,"time":"2023-10-07T22:13:57.391071316Z","hostname":"oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36","pid":8992}
{"msg":"Database read version 1","v":0,"name":"crucible","level":30,"time":"2023-10-07T22:13:57.391101694Z","hostname":"oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36","pid":8992}
{"msg":"Database write version 1","v":0,"name":"crucible","level":30,"time":"2023-10-07T22:13:57.39111552Z","hostname":"oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36","pid":8992}
{"msg":"Error: Open of db file \"/data/regions/0a7868bd-9778-49c5-bc05-45892a9f7ec6/00/000/028.db\" for extent#40 returned: file is not a database","v":0,"name":"crucible","level":50,"time":"2023-10-07T22:13:57.455828553Z","hostname":"oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36","pid":8992}
Error: Open of db file "/data/regions/0a7868bd-9778-49c5-bc05-45892a9f7ec6/00/000/028.db" for extent#40 returned: file is not a database
[ Oct  7 22:13:57 Stopping because service exited with an error. ]

The sled in question (sled 10 on rack3) was rebooted with kmdb enabled to troubleshoot https://github.com/oxidecomputer/stlouis/issues/281. Three other regions also ran into the same kind of error when their crucible zones came back up after the sled reboot.

For now, I have disabled these downstairs to prevent the indefinite retries to bring up the SMF service:

root@oxz_crucible_c5d44f20-3fd7-4859-9386-e81239dc413e:~# svcadm disable downstairs-a101225c-5829-45d3-9129-792a6b13fd06

root@oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36:~# svcadm disable downstairs-0a7868bd-9778-49c5-bc05-45892a9f7ec6

root@oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36:~# svcadm disable downstairs-612d3da0-8332-4483-b2a8-04e4e013dccd

root@oxz_crucible_864fdb33-a36d-4bb0-96e6-a3b3b057ba8b:~# svcadm disable downstairs-14f17cc8-4857-49c4-861c-a1bb831a4dbb

The complete crucible zone logs are located in catacomb:/staff/augustus/rack3/instances/7f82a998-6f4f-4e53-bfde-325f339a850c ( pool/ext/*/crypt/debug/oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36/oxide-crucible-downstairs:downstairs-0a7868bd-9778-49c5-bc05-45892a9f7ec6.log.*).

The region tar file is being exported from the rack. I'll add the link if we decide to upload it to catacomb for future reference..

askfongjojo commented 10 months ago

Zpool status seems fine and the disks also have plenty of space:

BRM42220024 # zpool status oxp_cf2f1644-1870-4819-8595-e183d05020db
  pool: oxp_cf2f1644-1870-4819-8595-e183d05020db
 state: ONLINE
  scan: none requested
config:

        NAME                                      STATE     READ WRITE CKSUM
        oxp_cf2f1644-1870-4819-8595-e183d05020db  ONLINE       0     0     0
          c3t0014EE81000BC27Bd0s0                 ONLINE       0     0     0

errors: No known data errors
BRM42220024 # zfs list -r
NAME                                                                                                           USED  AVAIL     REFER  MOUNTPOINT
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2                                                                       461M   724G       96K  /oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/backing                                                               192K   724G       96K  /pool/int/oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/backing
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/backing/fmd                                                            96K   500M       96K  /var/fm/fmd
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/cluster                                                               144K   724G      144K  /pool/int/685e6727-9ddd-47f7-8e46-1b9cd6f796d2/cluster
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/config                                                                116K   724G      116K  /pool/int/685e6727-9ddd-47f7-8e46-1b9cd6f796d2/config
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/crash                                                                  96K   724G       96K  /pool/int/685e6727-9ddd-47f7-8e46-1b9cd6f796d2/crash
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/debug                                                                  96K   100G       96K  /pool/int/685e6727-9ddd-47f7-8e46-1b9cd6f796d2/debug
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/install                                                               455M   724G      455M  /pool/int/685e6727-9ddd-47f7-8e46-1b9cd6f796d2/install
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457                                                                       461M   724G       96K  /oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/backing                                                               240K   724G       96K  /pool/int/oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/backing
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/backing/fmd                                                           144K   500M      144K  /var/fm/fmd
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/cluster                                                               144K   724G      144K  /pool/int/7e37de81-8827-4fa3-863e-f2c6df3bb457/cluster
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/config                                                                116K   724G      116K  /pool/int/7e37de81-8827-4fa3-863e-f2c6df3bb457/config
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/crash                                                                  96K   724G       96K  /pool/int/7e37de81-8827-4fa3-863e-f2c6df3bb457/crash
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/debug                                                                  96K   100G       96K  /pool/int/7e37de81-8827-4fa3-863e-f2c6df3bb457/debug
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/install                                                               455M   724G      455M  /pool/int/7e37de81-8827-4fa3-863e-f2c6df3bb457/install
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/swap                                                                   88K   724G       88K  -
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58                                                                      57.2G  2.76T       24K  /oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crucible                                                             25.0G  2.76T       26K  /data
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crucible/regions                                                     25.0G  2.76T       24K  /data/regions
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crucible/regions/a101225c-5829-45d3-9129-792a6b13fd06                12.6G  47.4G     6.96G  /data/regions/a101225c-5829-45d3-9129-792a6b13fd06
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt                                                                32.1G  2.76T     31.9G  /pool/ext/20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/debug                                                            99K   100G       99K  /pool/ext/20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/debug
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/zone                                                            286M  2.76T      100K  /pool/ext/20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/zone
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/zone/oxz_crucible_b8f67f67-a2e4-4e40-a0ea-a4689572372f          286M  2.76T      286M  /pool/ext/20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/zone/oxz_crucible_b8f67f67-a2e4-4e40-a0ea-a4689572372f
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7                                                                      2.51G  2.81T       24K  /oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crucible                                                               48K  2.81T       24K  /data
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crucible/regions                                                       24K  2.81T       24K  /data/regions
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt                                                                2.50G  2.81T     2.22G  /pool/ext/2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/debug                                                            99K   100G       99K  /pool/ext/2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/debug
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/zone                                                            286M  2.81T      100K  /pool/ext/2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/zone
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/zone/oxz_crucible_e9d38f63-688a-47cd-93f6-1abe41f44cfa          286M  2.81T      286M  /pool/ext/2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/zone/oxz_crucible_e9d38f63-688a-47cd-93f6-1abe41f44cfa
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6                                                                      35.5G  2.78T       24K  /oxp_3a2616b2-81af-42d4-8c93-343db10b96f6
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crucible                                                             3.75G  2.78T       26K  /data
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crucible/regions                                                     3.75G  2.78T       24K  /data/regions
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crucible/regions/80cfef8a-4350-4b24-8f54-c083b6e8f085                1.76G  7.24G     1.76G  /data/regions/80cfef8a-4350-4b24-8f54-c083b6e8f085
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crypt                                                                31.8G  2.78T     31.8G  /pool/ext/3a2616b2-81af-42d4-8c93-343db10b96f6/crypt
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crypt/debug                                                            99K   100G       99K  /pool/ext/3a2616b2-81af-42d4-8c93-343db10b96f6/crypt/debug
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crypt/zone                                                             99K  2.78T       99K  /pool/ext/3a2616b2-81af-42d4-8c93-343db10b96f6/crypt/zone
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1                                                                      10.4G  2.81T       24K  /oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crucible                                                               48K  2.81T       24K  /data
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crucible/regions                                                       24K  2.81T       24K  /data/regions
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt                                                                10.4G  2.81T     9.84G  /pool/ext/42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/debug                                                            99K   100G       99K  /pool/ext/42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/debug
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone                                                            576M  2.81T      133K  /pool/ext/42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone/oxz_crucible_0c654458-3ca3-470b-96d9-76a05e120c9f          286M  2.81T      286M  /pool/ext/42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone/oxz_crucible_0c654458-3ca3-470b-96d9-76a05e120c9f
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone/oxz_crucible_c5d44f20-3fd7-4859-9386-e81239dc413e          290M  2.81T      290M  /pool/ext/42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone/oxz_crucible_c5d44f20-3fd7-4859-9386-e81239dc413e
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada                                                                      7.94G  2.81T       24K  /oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crucible                                                             7.65G  2.81T       26K  /data
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crucible/regions                                                     7.65G  2.81T       25K  /data/regions
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crucible/regions/736a34f2-056c-4123-9044-ff96dd042093                5.15G  6.85G     5.15G  /data/regions/736a34f2-056c-4123-9044-ff96dd042093
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crucible/regions/ea9beb14-12ca-4d64-ac29-64f374fd7f48                1.08G  4.92G     1.08G  /data/regions/ea9beb14-12ca-4d64-ac29-64f374fd7f48
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt                                                                 286M  2.81T      133K  /pool/ext/8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/debug                                                            99K   100G       99K  /pool/ext/8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/debug
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/zone                                                            286M  2.81T      100K  /pool/ext/8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/zone
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/zone/oxz_crucible_cfd81aac-3801-4493-831e-d9ac04001292          286M  2.81T      286M  /pool/ext/8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/zone/oxz_crucible_cfd81aac-3801-4493-831e-d9ac04001292
oxp_cf2f1644-1870-4819-8595-e183d05020db                                                                      26.2G  2.79T       24K  /oxp_cf2f1644-1870-4819-8595-e183d05020db
oxp_cf2f1644-1870-4819-8595-e183d05020db/crucible                                                             15.0G  2.79T       26K  /data
oxp_cf2f1644-1870-4819-8595-e183d05020db/crucible/regions                                                     15.0G  2.79T       26K  /data/regions
oxp_cf2f1644-1870-4819-8595-e183d05020db/crucible/regions/0a7868bd-9778-49c5-bc05-45892a9f7ec6                3.98G  14.0G     3.98G  /data/regions/0a7868bd-9778-49c5-bc05-45892a9f7ec6
oxp_cf2f1644-1870-4819-8595-e183d05020db/crucible/regions/612d3da0-8332-4483-b2a8-04e4e013dccd                4.04G  14.0G     4.04G  /data/regions/612d3da0-8332-4483-b2a8-04e4e013dccd
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt                                                                11.2G  2.79T     10.3G  /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt/debug                                                            99K   100G       99K  /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt/debug
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt/zone                                                            857M  2.79T      166K  /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt/zone
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_0a2c6672-6439-4edf-9a63-50f9a25df161          285M  2.79T      285M  /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_0a2c6672-6439-4edf-9a63-50f9a25df161
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_836e0c14-32fd-4d5f-919d-1b37b69bf987          286M  2.79T      286M  /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_836e0c14-32fd-4d5f-919d-1b37b69bf987
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_cedf39de-04e4-4ee9-b818-db29c3f6f7aa          286M  2.79T      286M  /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_cedf39de-04e4-4ee9-b818-db29c3f6f7aa
oxp_ef2578b8-c062-4636-a715-182ffa3ded04                                                                      45.3G  2.77T       24K  /oxp_ef2578b8-c062-4636-a715-182ffa3ded04
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crucible                                                             23.8G  2.77T     26.5K  /data
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crucible/regions                                                     23.8G  2.77T       27K  /data/regions
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crucible/regions/08f4d6d3-f230-4c81-9531-6ae14172e868                 248M  17.8G      248M  /data/regions/08f4d6d3-f230-4c81-9531-6ae14172e868
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crucible/regions/451f3554-33cf-479e-8a30-363fd0d6dfc4                1.76G  7.24G     1.76G  /data/regions/451f3554-33cf-479e-8a30-363fd0d6dfc4
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crucible/regions/85b6cbc0-beb8-4608-9816-b6899e9a84e5                7.62M  30.0G     7.62M  /data/regions/85b6cbc0-beb8-4608-9816-b6899e9a84e5
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt                                                                21.5G  2.77T     20.7G  /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt/debug                                                            99K   100G       99K  /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt/debug
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone                                                            884M  2.77T      166K  /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36          403M  2.77T      403M  /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_crucible_864fdb33-a36d-4bb0-96e6-a3b3b057ba8b          290M  2.77T      290M  /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_crucible_864fdb33-a36d-4bb0-96e6-a3b3b057ba8b
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_ntp_1b49557d-8184-4556-9826-8505b043a8b5               191M  2.77T      191M  /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_ntp_1b49557d-8184-4556-9826-8505b043a8b5
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2                                                                      15.7G  2.80T       24K  /oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crucible                                                               48K  2.80T       24K  /data
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crucible/regions                                                       24K  2.80T       24K  /data/regions
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt                                                                15.7G  2.80T     15.7G  /pool/ext/f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt/debug                                                            99K   100G       99K  /pool/ext/f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt/debug
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt/zone                                                             99K  2.80T       99K  /pool/ext/f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt/zone
oxp_f59674a4-437e-4846-9abf-a71679082c82                                                                      35.9G  2.78T       24K  /oxp_f59674a4-437e-4846-9abf-a71679082c82
oxp_f59674a4-437e-4846-9abf-a71679082c82/crucible                                                             5.00G  2.78T     25.5K  /data
oxp_f59674a4-437e-4846-9abf-a71679082c82/crucible/regions                                                     5.00G  2.78T       24K  /data/regions
oxp_f59674a4-437e-4846-9abf-a71679082c82/crucible/regions/14f17cc8-4857-49c4-861c-a1bb831a4dbb                4.81G  7.19G     4.81G  /data/regions/14f17cc8-4857-49c4-861c-a1bb831a4dbb
oxp_f59674a4-437e-4846-9abf-a71679082c82/crypt                                                                30.9G  2.78T     30.9G  /pool/ext/f59674a4-437e-4846-9abf-a71679082c82/crypt
oxp_f59674a4-437e-4846-9abf-a71679082c82/crypt/debug                                                          4.76M   100G     4.76M  /pool/ext/f59674a4-437e-4846-9abf-a71679082c82/crypt/debug
oxp_f59674a4-437e-4846-9abf-a71679082c82/crypt/zone                                                             99K  2.78T       99K  /pool/ext/f59674a4-437e-4846-9abf-a71679082c82/crypt/zone
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048                                                                      20.8G  2.80T       24K  /oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crucible                                                               48K  2.80T       24K  /data
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crucible/regions                                                       24K  2.80T       24K  /data/regions
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt                                                                20.8G  2.80T     20.5G  /pool/ext/f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/debug                                                            99K   100G       99K  /pool/ext/f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/debug
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/zone                                                            278M  2.80T      109K  /pool/ext/f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/zone
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/zone/oxz_propolis-server_00f28d42-62e8-4401-9095-00f2eacf5b55   278M  2.80T      278M  /pool/ext/f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/zone/oxz_propolis-server_00f28d42-62e8-4401-9095-00f2eacf5b55
rpool                                                                                                          555M  3.27G       24K  /rpool
rpool/ROOT                                                                                                     542M  3.27G       24K  legacy
rpool/ROOT/ramdisk                                                                                             542M  3.27G      542M  legacy
rpool/zone                                                                                                      24K  3.27G       24K  /zone
askfongjojo commented 10 months ago

The region archive is at catacomb:/staff/angela/region-0a7868bd-9778-49c5-bc05-45892a9f7ec6.tar.gz

And here are the 3 volumes for the stuck instance's disk:

               disk.id                 |                  region.id           |              dataset.id              |          ip           |      instance      |                  disk
---------------------------------------+--------------------------------------+--------------------------------------+-----------------------+--------------------+-----------------------------------------
  cb700855-07bb-4673-894b-fb97299067be | 4cfda4f3-e900-4331-b56c-85f9268f01f6 | 10b80058-9b2e-4d6c-8a1a-a61a8258c12f | fd00:1122:3344:118::9 | somethingsomething | somethingsomething-ubuntu-jammy-79a8cb
  cb700855-07bb-4673-894b-fb97299067be | 0a7868bd-9778-49c5-bc05-45892a9f7ec6 | 3b56e56b-8328-45c1-b459-1f14bbea5e36 | fd00:1122:3344:11b::8 | somethingsomething | somethingsomething-ubuntu-jammy-79a8cb
  cb700855-07bb-4673-894b-fb97299067be | 61902371-a138-4cae-945f-7af3ab1ff939 | dda0f1c6-84a5-472c-b350-a799c8d3d0eb | fd00:1122:3344:115::8 | somethingsomething | somethingsomething-ubuntu-jammy-79a8cb