Open askfongjojo opened 10 months ago
Zpool status seems fine and the disks also have plenty of space:
BRM42220024 # zpool status oxp_cf2f1644-1870-4819-8595-e183d05020db
pool: oxp_cf2f1644-1870-4819-8595-e183d05020db
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
oxp_cf2f1644-1870-4819-8595-e183d05020db ONLINE 0 0 0
c3t0014EE81000BC27Bd0s0 ONLINE 0 0 0
errors: No known data errors
BRM42220024 # zfs list -r
NAME USED AVAIL REFER MOUNTPOINT
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2 461M 724G 96K /oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/backing 192K 724G 96K /pool/int/oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/backing
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/backing/fmd 96K 500M 96K /var/fm/fmd
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/cluster 144K 724G 144K /pool/int/685e6727-9ddd-47f7-8e46-1b9cd6f796d2/cluster
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/config 116K 724G 116K /pool/int/685e6727-9ddd-47f7-8e46-1b9cd6f796d2/config
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/crash 96K 724G 96K /pool/int/685e6727-9ddd-47f7-8e46-1b9cd6f796d2/crash
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/debug 96K 100G 96K /pool/int/685e6727-9ddd-47f7-8e46-1b9cd6f796d2/debug
oxi_685e6727-9ddd-47f7-8e46-1b9cd6f796d2/install 455M 724G 455M /pool/int/685e6727-9ddd-47f7-8e46-1b9cd6f796d2/install
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457 461M 724G 96K /oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/backing 240K 724G 96K /pool/int/oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/backing
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/backing/fmd 144K 500M 144K /var/fm/fmd
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/cluster 144K 724G 144K /pool/int/7e37de81-8827-4fa3-863e-f2c6df3bb457/cluster
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/config 116K 724G 116K /pool/int/7e37de81-8827-4fa3-863e-f2c6df3bb457/config
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/crash 96K 724G 96K /pool/int/7e37de81-8827-4fa3-863e-f2c6df3bb457/crash
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/debug 96K 100G 96K /pool/int/7e37de81-8827-4fa3-863e-f2c6df3bb457/debug
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/install 455M 724G 455M /pool/int/7e37de81-8827-4fa3-863e-f2c6df3bb457/install
oxi_7e37de81-8827-4fa3-863e-f2c6df3bb457/swap 88K 724G 88K -
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58 57.2G 2.76T 24K /oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crucible 25.0G 2.76T 26K /data
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crucible/regions 25.0G 2.76T 24K /data/regions
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crucible/regions/a101225c-5829-45d3-9129-792a6b13fd06 12.6G 47.4G 6.96G /data/regions/a101225c-5829-45d3-9129-792a6b13fd06
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt 32.1G 2.76T 31.9G /pool/ext/20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/debug 99K 100G 99K /pool/ext/20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/debug
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/zone 286M 2.76T 100K /pool/ext/20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/zone
oxp_20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/zone/oxz_crucible_b8f67f67-a2e4-4e40-a0ea-a4689572372f 286M 2.76T 286M /pool/ext/20e0af7c-673a-43c6-b034-6b9ca7ff0c58/crypt/zone/oxz_crucible_b8f67f67-a2e4-4e40-a0ea-a4689572372f
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7 2.51G 2.81T 24K /oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crucible 48K 2.81T 24K /data
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crucible/regions 24K 2.81T 24K /data/regions
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt 2.50G 2.81T 2.22G /pool/ext/2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/debug 99K 100G 99K /pool/ext/2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/debug
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/zone 286M 2.81T 100K /pool/ext/2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/zone
oxp_2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/zone/oxz_crucible_e9d38f63-688a-47cd-93f6-1abe41f44cfa 286M 2.81T 286M /pool/ext/2a7db922-a80d-46db-b1f6-9e4a2af3bff7/crypt/zone/oxz_crucible_e9d38f63-688a-47cd-93f6-1abe41f44cfa
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6 35.5G 2.78T 24K /oxp_3a2616b2-81af-42d4-8c93-343db10b96f6
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crucible 3.75G 2.78T 26K /data
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crucible/regions 3.75G 2.78T 24K /data/regions
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crucible/regions/80cfef8a-4350-4b24-8f54-c083b6e8f085 1.76G 7.24G 1.76G /data/regions/80cfef8a-4350-4b24-8f54-c083b6e8f085
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crypt 31.8G 2.78T 31.8G /pool/ext/3a2616b2-81af-42d4-8c93-343db10b96f6/crypt
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crypt/debug 99K 100G 99K /pool/ext/3a2616b2-81af-42d4-8c93-343db10b96f6/crypt/debug
oxp_3a2616b2-81af-42d4-8c93-343db10b96f6/crypt/zone 99K 2.78T 99K /pool/ext/3a2616b2-81af-42d4-8c93-343db10b96f6/crypt/zone
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1 10.4G 2.81T 24K /oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crucible 48K 2.81T 24K /data
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crucible/regions 24K 2.81T 24K /data/regions
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt 10.4G 2.81T 9.84G /pool/ext/42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/debug 99K 100G 99K /pool/ext/42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/debug
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone 576M 2.81T 133K /pool/ext/42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone/oxz_crucible_0c654458-3ca3-470b-96d9-76a05e120c9f 286M 2.81T 286M /pool/ext/42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone/oxz_crucible_0c654458-3ca3-470b-96d9-76a05e120c9f
oxp_42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone/oxz_crucible_c5d44f20-3fd7-4859-9386-e81239dc413e 290M 2.81T 290M /pool/ext/42924ed1-ea9a-4abd-9bfc-bea17c511da1/crypt/zone/oxz_crucible_c5d44f20-3fd7-4859-9386-e81239dc413e
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada 7.94G 2.81T 24K /oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crucible 7.65G 2.81T 26K /data
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crucible/regions 7.65G 2.81T 25K /data/regions
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crucible/regions/736a34f2-056c-4123-9044-ff96dd042093 5.15G 6.85G 5.15G /data/regions/736a34f2-056c-4123-9044-ff96dd042093
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crucible/regions/ea9beb14-12ca-4d64-ac29-64f374fd7f48 1.08G 4.92G 1.08G /data/regions/ea9beb14-12ca-4d64-ac29-64f374fd7f48
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt 286M 2.81T 133K /pool/ext/8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/debug 99K 100G 99K /pool/ext/8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/debug
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/zone 286M 2.81T 100K /pool/ext/8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/zone
oxp_8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/zone/oxz_crucible_cfd81aac-3801-4493-831e-d9ac04001292 286M 2.81T 286M /pool/ext/8d1c245a-07d6-48ec-b4a7-3d47f65a1ada/crypt/zone/oxz_crucible_cfd81aac-3801-4493-831e-d9ac04001292
oxp_cf2f1644-1870-4819-8595-e183d05020db 26.2G 2.79T 24K /oxp_cf2f1644-1870-4819-8595-e183d05020db
oxp_cf2f1644-1870-4819-8595-e183d05020db/crucible 15.0G 2.79T 26K /data
oxp_cf2f1644-1870-4819-8595-e183d05020db/crucible/regions 15.0G 2.79T 26K /data/regions
oxp_cf2f1644-1870-4819-8595-e183d05020db/crucible/regions/0a7868bd-9778-49c5-bc05-45892a9f7ec6 3.98G 14.0G 3.98G /data/regions/0a7868bd-9778-49c5-bc05-45892a9f7ec6
oxp_cf2f1644-1870-4819-8595-e183d05020db/crucible/regions/612d3da0-8332-4483-b2a8-04e4e013dccd 4.04G 14.0G 4.04G /data/regions/612d3da0-8332-4483-b2a8-04e4e013dccd
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt 11.2G 2.79T 10.3G /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt/debug 99K 100G 99K /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt/debug
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt/zone 857M 2.79T 166K /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt/zone
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_0a2c6672-6439-4edf-9a63-50f9a25df161 285M 2.79T 285M /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_0a2c6672-6439-4edf-9a63-50f9a25df161
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_836e0c14-32fd-4d5f-919d-1b37b69bf987 286M 2.79T 286M /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_836e0c14-32fd-4d5f-919d-1b37b69bf987
oxp_cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_cedf39de-04e4-4ee9-b818-db29c3f6f7aa 286M 2.79T 286M /pool/ext/cf2f1644-1870-4819-8595-e183d05020db/crypt/zone/oxz_crucible_cedf39de-04e4-4ee9-b818-db29c3f6f7aa
oxp_ef2578b8-c062-4636-a715-182ffa3ded04 45.3G 2.77T 24K /oxp_ef2578b8-c062-4636-a715-182ffa3ded04
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crucible 23.8G 2.77T 26.5K /data
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crucible/regions 23.8G 2.77T 27K /data/regions
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crucible/regions/08f4d6d3-f230-4c81-9531-6ae14172e868 248M 17.8G 248M /data/regions/08f4d6d3-f230-4c81-9531-6ae14172e868
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crucible/regions/451f3554-33cf-479e-8a30-363fd0d6dfc4 1.76G 7.24G 1.76G /data/regions/451f3554-33cf-479e-8a30-363fd0d6dfc4
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crucible/regions/85b6cbc0-beb8-4608-9816-b6899e9a84e5 7.62M 30.0G 7.62M /data/regions/85b6cbc0-beb8-4608-9816-b6899e9a84e5
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt 21.5G 2.77T 20.7G /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt/debug 99K 100G 99K /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt/debug
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone 884M 2.77T 166K /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36 403M 2.77T 403M /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_crucible_864fdb33-a36d-4bb0-96e6-a3b3b057ba8b 290M 2.77T 290M /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_crucible_864fdb33-a36d-4bb0-96e6-a3b3b057ba8b
oxp_ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_ntp_1b49557d-8184-4556-9826-8505b043a8b5 191M 2.77T 191M /pool/ext/ef2578b8-c062-4636-a715-182ffa3ded04/crypt/zone/oxz_ntp_1b49557d-8184-4556-9826-8505b043a8b5
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2 15.7G 2.80T 24K /oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crucible 48K 2.80T 24K /data
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crucible/regions 24K 2.80T 24K /data/regions
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt 15.7G 2.80T 15.7G /pool/ext/f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt/debug 99K 100G 99K /pool/ext/f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt/debug
oxp_f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt/zone 99K 2.80T 99K /pool/ext/f3cf1065-151d-4d09-85f2-21c8ddcc16f2/crypt/zone
oxp_f59674a4-437e-4846-9abf-a71679082c82 35.9G 2.78T 24K /oxp_f59674a4-437e-4846-9abf-a71679082c82
oxp_f59674a4-437e-4846-9abf-a71679082c82/crucible 5.00G 2.78T 25.5K /data
oxp_f59674a4-437e-4846-9abf-a71679082c82/crucible/regions 5.00G 2.78T 24K /data/regions
oxp_f59674a4-437e-4846-9abf-a71679082c82/crucible/regions/14f17cc8-4857-49c4-861c-a1bb831a4dbb 4.81G 7.19G 4.81G /data/regions/14f17cc8-4857-49c4-861c-a1bb831a4dbb
oxp_f59674a4-437e-4846-9abf-a71679082c82/crypt 30.9G 2.78T 30.9G /pool/ext/f59674a4-437e-4846-9abf-a71679082c82/crypt
oxp_f59674a4-437e-4846-9abf-a71679082c82/crypt/debug 4.76M 100G 4.76M /pool/ext/f59674a4-437e-4846-9abf-a71679082c82/crypt/debug
oxp_f59674a4-437e-4846-9abf-a71679082c82/crypt/zone 99K 2.78T 99K /pool/ext/f59674a4-437e-4846-9abf-a71679082c82/crypt/zone
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048 20.8G 2.80T 24K /oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crucible 48K 2.80T 24K /data
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crucible/regions 24K 2.80T 24K /data/regions
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt 20.8G 2.80T 20.5G /pool/ext/f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/debug 99K 100G 99K /pool/ext/f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/debug
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/zone 278M 2.80T 109K /pool/ext/f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/zone
oxp_f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/zone/oxz_propolis-server_00f28d42-62e8-4401-9095-00f2eacf5b55 278M 2.80T 278M /pool/ext/f85011e9-3a6c-4925-a8e4-a56ebe8cf048/crypt/zone/oxz_propolis-server_00f28d42-62e8-4401-9095-00f2eacf5b55
rpool 555M 3.27G 24K /rpool
rpool/ROOT 542M 3.27G 24K legacy
rpool/ROOT/ramdisk 542M 3.27G 542M legacy
rpool/zone 24K 3.27G 24K /zone
The region archive is at catacomb:/staff/angela/region-0a7868bd-9778-49c5-bc05-45892a9f7ec6.tar.gz
And here are the 3 volumes for the stuck instance's disk:
disk.id | region.id | dataset.id | ip | instance | disk
---------------------------------------+--------------------------------------+--------------------------------------+-----------------------+--------------------+-----------------------------------------
cb700855-07bb-4673-894b-fb97299067be | 4cfda4f3-e900-4331-b56c-85f9268f01f6 | 10b80058-9b2e-4d6c-8a1a-a61a8258c12f | fd00:1122:3344:118::9 | somethingsomething | somethingsomething-ubuntu-jammy-79a8cb
cb700855-07bb-4673-894b-fb97299067be | 0a7868bd-9778-49c5-bc05-45892a9f7ec6 | 3b56e56b-8328-45c1-b459-1f14bbea5e36 | fd00:1122:3344:11b::8 | somethingsomething | somethingsomething-ubuntu-jammy-79a8cb
cb700855-07bb-4673-894b-fb97299067be | 61902371-a138-4cae-945f-7af3ab1ff939 | dda0f1c6-84a5-472c-b350-a799c8d3d0eb | fd00:1122:3344:115::8 | somethingsomething | somethingsomething-ubuntu-jammy-79a8cb
We have several downstairs on rack3 that hit this kind of error after the crucible zone came back from a sled reboot:
The sled in question (sled 10 on rack3) was rebooted with kmdb enabled to troubleshoot https://github.com/oxidecomputer/stlouis/issues/281. Three other regions also ran into the same kind of error when their crucible zones came back up after the sled reboot.
For now, I have disabled these downstairs to prevent the indefinite retries to bring up the SMF service:
The complete crucible zone logs are located in catacomb:/staff/augustus/rack3/instances/7f82a998-6f4f-4e53-bfde-325f339a850c ( pool/ext/*/crypt/debug/oxz_crucible_3b56e56b-8328-45c1-b459-1f14bbea5e36/oxide-crucible-downstairs:downstairs-0a7868bd-9778-49c5-bc05-45892a9f7ec6.log.*).
The region tar file is being exported from the rack. I'll add the link if we decide to upload it to catacomb for future reference..