Closed cockroach-teamcity closed 2 weeks ago
roachtest.disk-stalled/dmsetup failed with artifacts on release-23.1 @ 4a21326a73225ee33350c2514ea221f21e101fe3:
(disk_stall.go:303).Setup: full command output in run_114237.102170295_n1-4_echo-0-sudo-blockdev.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/disk-stalled/dmsetup/run_1
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
See: roachtest README
See: How To Investigate (internal)
See: Grafana
Wraps: (3) Node 4. Command with error:
| ```
| echo "0 $(sudo blockdev --getsz /dev/sdb) linear /dev/sdb 0" | sudo dmsetup create data1
| ```
| device-mapper: reload ioctl on data1 failed: Device or resource busy
| Command failed.
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *hintdetail.withDetail (4) errors.Cmd (5) *exec.ExitError
From n4's journalctl:
systemd[1]: Started Session 11 of user ubuntu.
sudo[11524]: ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=roo
sudo[11525]: ubuntu : TTY=unknown ; PWD=/home/ubuntu ; USER=roo
sudo[11525]: pam_unix(sudo:session): session opened for user root
sudo[11524]: pam_unix(sudo:session): session opened for user root
sudo[11525]: pam_unix(sudo:session): session closed for user root
kernel: device-mapper: table: 253:0: linear: Device lookup failed
kernel: device-mapper: ioctl: error adding target to table
sudo[11524]: pam_unix(sudo:session): session closed for user root
and earlier during the startup script:
startup-script: tune2fs 1.45.5 (07-Jan-2020)
startup-script: Setting reserved blocks percentage to 0% (0 bloc
startup-script: + chmod 777 /mnt/data1
startup-script: + lsblk
startup-script: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
startup-script: loop0 7:0 0 63.5M 1 loop /snap/core20/1
startup-script: loop1 7:1 0 348.5M 1 loop /snap/google-c
startup-script: loop2 7:2 0 91.9M 1 loop /snap/lxd/2406
startup-script: loop3 7:3 0 53.3M 1 loop /snap/snapd/19
startup-script: sda 8:0 0 500G 0 disk /mnt/data1
startup-script: sdb 8:16 0 32G 0 disk
startup-script: ├─sdb1 8:17 0 31.9G 0 part /
startup-script: ├─sdb14 8:30 0 4M 0 part
startup-script: └─sdb15 8:31 0 106M 0 part /boot/efi
startup-script: + df -h
startup-script: Filesystem Size Used Avail Use% Mounted on
startup-script: /dev/root 31G 1.9G 29G 7% /
startup-script: devtmpfs 7.9G 0 7.9G 0% /dev
startup-script: tmpfs 7.9G 0 7.9G 0% /dev/shm
startup-script: tmpfs 1.6G 1.2M 1.6G 1% /run
startup-script: tmpfs 5.0M 0 5.0M 0% /run/lock
startup-script: tmpfs 7.9G 0 7.9G 0% /sys/fs/cg
startup-script: /dev/loop0 64M 64M 0 100% /snap/core
startup-script: /dev/loop1 349M 349M 0 100% /snap/goog
startup-script: /dev/loop3 54M 54M 0 100% /snap/snap
startup-script: /dev/loop2 92M 92M 0 100% /snap/lxd/
startup-script: /dev/sdb15 105M 6.1M 99M 6% /boot/efi
startup-script: /dev/sda 492G 28K 492G 1% /mnt/data1
The SHA 4a21326a73225ee33350c2514ea221f21e101fe3 includes #126842.
I'm a bit perplexed. Shouldn't getDevice
have found /dev/sda
because that's what's mounted at /mnt/data1?
roachtest.disk-stalled/dmsetup failed with artifacts on release-23.1 @ fbcb992a72c3ac2a9af96f6238a24b3978bcaadf:
Parameters:
ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help
See: roachtest README
See: How To Investigate (internal)
See: Grafana
/cc @cockroachdb/storageThis test on roachdash | Improve this report!
Jira issue: CRDB-39926