MBON server outage 2017-07-20

[7yl4r]

server and network went out due to power outage last night. Coming back up required applying a few NFS mounts on the hypervisor manually. Now we are booting the VM and...

[7yl4r]

relevant errors from the log:

[7yl4r]

This is a failure of /hdb1 to mount from fstab:

/dev/hdb1    /mbon-postgres    /ext3    defaults    0    0

The relevant volume is accessible on the hypervisor:

[admin@mbonhypervisor ~]# ll -h /mnt/mbondata/mbondata_volume.qcow2 
-rw-r--r--. 1 qemu qemu 949G Jul 20 10:20 /mnt/mbondata/mbondata_volume.qcow2

so I'm not sure why it isn't mounting properly in the VM

[bbest]

Hi Tylar,

I can recreate the postgres database if you can't mount the drive. Ideally this is a fast (SSD) local drive of reasonable size (> 200 GB) for database loading and querying. I was previously having issues with queries not ever seeming to finish (to create spatial lookup tables of OBIS observations to EEZ polygons). Maybe that was because of a bad disk?

Thanks, Ben

On Thu, Jul 20, 2017 at 7:48 AM, Tylar notifications@github.com wrote:

This is a failure of /hdb1 to mount from fstab:

/dev/hdb1 /mbon-postgres /ext3 defaults 0 0

The relevant volume is accessible on the hypervisor:

[admin@mbonhypervisor ~]# ll -h /mnt/mbondata/mbondata_volume.qcow2 -rw-r--r--. 1 qemu qemu 949G Jul 20 10:20 /mnt/mbondata/mbondata_volume.qcow2

so I'm not sure why it isn't mounting properly in the VM

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/marinebon/sdg14/issues/22#issuecomment-316726816, or mute the thread https://github.com/notifications/unsubscribe-auth/ACtLCe2lisi6DkS9WeijvhGfiAH-8npRks5sP2isgaJpZM4OeNBL .

[7yl4r]

It could be a bad disk, but I believe I installed a brand new physical drive and dedicated the whole thing for this.

The drive name is not hdb1 as in the fstab, but vda1, so I have changed that but now...

[7yl4r]

It looks like /mbon-postgres has contents on the root mount that will be squashed when /vda1 is mounted there

( not a big deal I think, just a little wasted space on the other disk)

I was able to mount vda1 manually... so I guess we'll just have to remember to re-mount it after reboot since something is stopping fstab from doing that.

Let me know if you run across any other troubles.

[bbest]

I am seeing both paths /mbon and /mbon-postgres:

ben@mbon:/$ ls
bin   dev  home        initrd.img.old  lib64       mbon           media  opt   root  sbin  srv  tmp  var      vmlinuz.old
boot  etc  initrd.img  lib             lost+found  mbon-postgres  mnt    proc  run   snap  sys  usr  vmlinuz
ben@mbon:/$ ls mbon
data_big  geoserver  postgres  sdg14  shiny  shiny1  shiny-log  test.txt  www  www-conf
ben@mbon:/$ ls mbon-postgres
9.5
ben@mbon:/$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
udev                          32G     0   32G   0% /dev
tmpfs                        6.3G  9.1M  6.3G   1% /run
/dev/mapper/mbon--vg-root     19G  8.0G  9.3G  47% /
tmpfs                         27G  640K   27G   1% /dev/shm
tmpfs                        5.0M     0  5.0M   0% /run/lock
tmpfs                         27G     0   27G   0% /sys/fs/cgroup
/dev/sda1                    472M  105M  343M  24% /boot
corals.marine.usf.edu:/mbon   15T  5.2T  9.4T  36% /mbon
tmpfs                        4.8G     0  4.8G   0% /run/user/538
tmpfs                        4.8G     0  4.8G   0% /run/user/1000

Looks like /mbon-postgres is part of / which only has 9.3GB available, so will be a problem when trying to load OBIS, EEZ, WDPA, etc into PostGIS database.

[bbest]

Ah nice! Now am seeing a dedicated /mbon-postgres.

ben@mbon:/$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
udev                          32G     0   32G   0% /dev
tmpfs                        6.3G  9.1M  6.3G   1% /run
/dev/mapper/mbon--vg-root     19G  8.0G  9.3G  47% /
tmpfs                         27G  640K   27G   1% /dev/shm
tmpfs                        5.0M     0  5.0M   0% /run/lock
tmpfs                         27G     0   27G   0% /sys/fs/cgroup
/dev/sda1                    472M  105M  343M  24% /boot
corals.marine.usf.edu:/mbon   15T  5.2T  9.4T  36% /mbon
tmpfs                        4.8G     0  4.8G   0% /run/user/538
tmpfs                        4.8G     0  4.8G   0% /run/user/1000
/dev/vda1                    886G   79G  763G  10% /mbon-postgres

Isn't a mount point supposed to be an empty folder? So does the 9.5 subfolder of /mbon-postgres come from root /dev/mapper/mbon--vg-root or from /dev/vda1?

[7yl4r]

Looks like /mbon-postgres is part of / which only has 9.3GB available

I just mounted it. The contents you are seeing in /mbon-postrgres that are part of / are inaccessible while vda is mounted over that location. Ideally you would unmount and delete them b/c otherwise it is wasted space on /, but it's not critical.

Isn't a mount point supposed to be an empty folder? So does the 9.5 subfolder of /mbon-postgres come from root /dev/mapper/mbon--vg-root or from /dev/vda1?

Mount points should be empty, but if they aren't the contents get mounted over and temporarily squashed out of existence. I'm pretty sure that 9.5 subdir is coincidentally common on both / and /dev/vda1; ie they are two different dirs with the same name.

[bbest]

Hi @7yl4r,

Thanks for all your help Tylar. Unfortunately mbon.marine.usf.edu seems to be down again.

ssh ben@mbon.marine.usf.edu
ssh: connect to host mbon.marine.usf.edu port 22: Operation timed out

[bbest]

Ok, it's up again.