search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.77k stars 3.76k forks source link

roachtest: disk-stalled/dmsetup failed #126452

Closed cockroach-teamcity closed 2 weeks ago

cockroach-teamcity commented 2 months ago

roachtest.disk-stalled/dmsetup failed with artifacts on release-23.1 @ fbcb992a72c3ac2a9af96f6238a24b3978bcaadf:

(disk_stall.go:301).Setup: full command output in run_121050.011821811_n1-4_echo-0-sudo-blockdev.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/disk-stalled/dmsetup/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-39926

cockroach-teamcity commented 1 month ago

roachtest.disk-stalled/dmsetup failed with artifacts on release-23.1 @ 4a21326a73225ee33350c2514ea221f21e101fe3:

(disk_stall.go:303).Setup: full command output in run_114237.102170295_n1-4_echo-0-sudo-blockdev.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/disk-stalled/dmsetup/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

jbowens commented 1 month ago 
Wraps: (3) Node 4. Command with error:
  | ```
  | echo "0 $(sudo blockdev --getsz /dev/sdb) linear /dev/sdb 0" | sudo dmsetup create data1
  | ```
  | device-mapper: reload ioctl on data1  failed: Device or resource busy
  | Command failed.
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *hintdetail.withDetail (4) errors.Cmd (5) *exec.ExitError

From n4's journalctl:

systemd[1]: Started Session 11 of user ubuntu.
sudo[11524]:   ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=roo
sudo[11525]:   ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=roo
sudo[11525]: pam_unix(sudo:session): session opened for user root
sudo[11524]: pam_unix(sudo:session): session opened for user root
sudo[11525]: pam_unix(sudo:session): session closed for user root
kernel: device-mapper: table: 253:0: linear: Device lookup failed
kernel: device-mapper: ioctl: error adding target to table
sudo[11524]: pam_unix(sudo:session): session closed for user root

and earlier during the startup script:

startup-script: tune2fs 1.45.5 (07-Jan-2020)
startup-script: Setting reserved blocks percentage to 0% (0 bloc
startup-script: + chmod 777 /mnt/data1
startup-script: + lsblk
startup-script: NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
startup-script: loop0     7:0    0  63.5M  1 loop /snap/core20/1
startup-script: loop1     7:1    0 348.5M  1 loop /snap/google-c
startup-script: loop2     7:2    0  91.9M  1 loop /snap/lxd/2406
startup-script: loop3     7:3    0  53.3M  1 loop /snap/snapd/19
startup-script: sda       8:0    0   500G  0 disk /mnt/data1
startup-script: sdb       8:16   0    32G  0 disk
startup-script: ├─sdb1    8:17   0  31.9G  0 part /
startup-script: ├─sdb14   8:30   0     4M  0 part
startup-script: └─sdb15   8:31   0   106M  0 part /boot/efi
startup-script: + df -h
startup-script: Filesystem      Size  Used Avail Use% Mounted on
startup-script: /dev/root        31G  1.9G   29G   7% /
startup-script: devtmpfs        7.9G     0  7.9G   0% /dev
startup-script: tmpfs           7.9G     0  7.9G   0% /dev/shm
startup-script: tmpfs           1.6G  1.2M  1.6G   1% /run
startup-script: tmpfs           5.0M     0  5.0M   0% /run/lock
startup-script: tmpfs           7.9G     0  7.9G   0% /sys/fs/cg
startup-script: /dev/loop0       64M   64M     0 100% /snap/core
startup-script: /dev/loop1      349M  349M     0 100% /snap/goog
startup-script: /dev/loop3       54M   54M     0 100% /snap/snap
startup-script: /dev/loop2       92M   92M     0 100% /snap/lxd/
startup-script: /dev/sdb15      105M  6.1M   99M   6% /boot/efi
startup-script: /dev/sda        492G   28K  492G   1% /mnt/data1
jbowens commented 1 month ago

The SHA 4a21326a73225ee33350c2514ea221f21e101fe3 includes #126842.

jbowens commented 1 month ago

I'm a bit perplexed. Shouldn't getDevice have found /dev/sda because that's what's mounted at /mnt/data1?