search

glos / myglos

Repository for MyGLOS (GLOS Data Portal)
GNU General Public License v3.0
2 stars 2 forks source link

Buoy & HABs tds netcdf not updating #255

Closed gcutrell closed 5 years ago

gcutrell commented 5 years ago

@cheryldmorse, I received alerts last night that the buoy data were not updating. While investigating I tried ssh into storage and got a "shell request failed on channel 0". It appears to be a nfs issue. Data is being saved in the database but I'm guessing it is just not getting into storage for thredds to pick up. It looks like it happened around 7PM Sunday night.

I can see the buoy data in /var/local/glos/storage/Buoy_data but it is also behind which explains why tds is not updating. Could you see what is going on? I'm not sure if this is effecting other products as well.

@kknee @tslawecki

gcutrell commented 5 years ago

@cheryldmorse After a second look it appears that the netcdf file for today in the local_cat2 folder is up-to-date, so I'm wondering if there is an issue the rsyncing to storage since there is nothing on tds?

@cheryldmorse @tslawecki

tslawecki commented 5 years ago

@cheryldmorse noted that the NFS server looked screwy and suggested a manual reboot (shutdown didn't work). @tslawecki did a backup (rsync to ./backup-storage on NFS drive, then tar to "H:\NFS\backup-storage-2.tar.gz" on mirror), then forced a reboot. Data flow is restored.