[Part 2] Workspace volume is read-only on cnf dev1 system

[lixuna]

Issue: file system not writeable

• Vagrant/Virtualbox not functioning because disk is read-only, which caused OpenStack failures

The dmesg log shows device-mapper requeuing since May 10th before a connection failure caused a full failure on May 29th starting at 2:40:46 UTC with multipath failures to the network storage.

[Tue May 29 02:40:46 2018] connection2:0: ping timeout of 3 secs expired, recv timeout 3, last rx 5317134756, last ping 5317135507, now 5317136321

This continued with more failures as the kernel noticed the device was offline so it was put into read-only mode at 02:41:19 UTC.

[Tue May 29 02:40:59 2018] sd 8:0:0:0: [sdh] tag#0 FAILED Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
[Tue May 29 02:41:18 2018] sd 7:0:0:0: rejecting I/O to offline device
[Tue May 29 02:41:19 2018] EXT4-fs (dm-1): Remounting filesystem read-only

Network interface to storage went down then bonded network interface disabled the failed network interface

[Tue May 29 02:44:13 2018] mlx4_en: enp2s0d1: Link Down
[Tue May 29 02:44:13 2018] bond0: link status definitely down for interface enp2s0d1, disabling it

At 02:46:04 the network interface came back up then the bonded interface was enabled. Unfortunately the bonding failed

[Tue May 29 02:46:04 2018] mlx4_en: enp2s0d1: Link Up
[Tue May 29 02:46:04 2018] bond0: link status up for interface enp2s0d1, enabling it in 200 ms
[Tue May 29 02:46:04 2018] mlx4_en: enp2s0d1: Fail to bond device

[lixuna]

Following steps from https://help.packet.net/technical/storage/packet-block-storage-linux to attach a new storage device

---

• [ ] create new volume
• [ ] attach to server from web ui
• [ ] check and ready device with packet-block-storage-attach -m queue
• [ ] format volume mkfs.ext4 /dev/mapper...
• [ ] make a mount point (eg. mkdir /cnfs01b)
• [ ] mount the volume...

Created new volume to copy existing data to https://app.packet.net/storage/2e56f22d-344a-4686-b6bf-5a27db102de0

This performance volume was created on May 30th, 2018

• id: volume-2e56f22d
• region: ams1
• name: CNFS01b
• Tier: Performance
• Location: AMS1
• Size: 300 GB

[lixuna]

Next steps:

• [ ] unmount backup storage from /cnfs01b
• [ ] review Resiliency section of https://help.packet.net/technical/storage/packet-block-storage-linux
• [ ] try using storage instructions for detach/attach: packet-block-storage-detach and packet-block-storage-attach -v -m queue listed in the docs
• [ ] if that fails, try remounting old device rw
• [ ] if that fails, try using the new disk copy
  1. otherwise stop vagrant
  2. unmount old storage device from /cnfs
  3. mount new storage device at /cnfs