search

glos / myglos

Repository for MyGLOS (GLOS Data Portal)
GNU General Public License v3.0
2 stars 2 forks source link

TDS GLCFS time format change #171

Open gcutrell opened 6 years ago

gcutrell commented 6 years ago

@kknee @cheryldmorse Greg Lang reported that the time format attribute for the GLCFS files on TDS changed on 10/28 ~0600GMT. It changed from "hours since 1970-01-01 00:00:00 +00:00" to "hours since 1970-01-01 00:00:00.000 UTC". Do you know of anything that might have caused this change?

kknee commented 6 years ago

@gcutrell we haven't made any changes to these files on TDS. We'll poke around a bit.

gcutrell commented 6 years ago

@kknee I've alerted Tad that tredds.glos.us has hit a critical disk space usage but I'm not sure how/if the two issues are related.

kknee commented 6 years ago

thanks @gcutrell maybe that explains why I can't load the GLCFS TDS catalog page to check out the timestamp issue.

kkoch commented 6 years ago

Hey team (@kknee @gcutrell @tslawecki) - I'm not sure if this is related but just had two inquiries regarding GLCFS from users. Can we raise the poking around a bit higher and see if we can resolve? I told the users I'd get right back to them with a response.

Here are the email texts: 1) From Greg Lang Eldon, No planned outage on our end. The GLCFS nowcast and forecasts are running on schedule at GLERL. It's possible tat the GLOS/LTI scripts that grab our netcdf files either failed or is running behind. I've cc'd the GLOS DMAC. The glcfs fcast timestamp is 2017 354 12 The GLOS catalog is at 2017 353 12 http://tds.glos.us/thredds/catalog/glos/glcfs/erie/fcfmrc-2d/files/catalog.html Greg

On Wed, Dec 20, 2017 at 11:13 AM, Eldon Wong EWong@bgcengineering.ca wrote: Hi Greg, We noticed that the forecast has not updated since yesterday morning. Is there a planned outage over the holidays? Thanks. BGC ENGINEERING INC. per: Eldon Wong, B.Sc.E., P.Eng., PMP, CSM Software Development Manager

gcutrell commented 6 years ago

There is no error in the GLCFS retrieval log. It appears the data is being retrieved but it is not showing up in nfs. The status page also says that GLCFS is up-to-date (http://data.glos.us/status/).

cheryldmorse commented 6 years ago

@gcutrell I'm not seeing any new forecast files on storage.glos.us in /var/thredds/GLCFS/Forecast It seems like this is probably related to the fact that storage.glos.us ran out of disk space yesterday

gcutrell commented 6 years ago

@cheryldmorse I'm not either. I'm starting to think Gwans logs and status is based on retrieved file information (like file name) and not what is actually in storage. We also have not had a new GLCFS to retrieve since the space issue was fixed. I'll go ahead and download the missing files from GLERL.

kkoch commented 6 years ago

Thanks @gcutrell for replying to Eldon. Do we also have an answer for Jeff Houghton regarding the 6 day shift in the point query tool or should I create that as a separate issue?

gcutrell commented 6 years ago

@kkoch Can you get a little more information from Jeff. Like what is he querying? I did a quick comparison between the nowcast significant wave height and observed for 45007 and they match up.

gcutrell commented 6 years ago

@cheryldmorse The script that retrieves the GLCFS files is running without any errors in the log. The issue is it thinks it is saving data to storage but it isn't. Do you know what could have happened when storage hit capacity? Does something need a restart? The retrieval script/log is in process1 at /usr/local/glos/glcfs/ if you wanted to take a look at it. We are behind on GLCFS data but it doesn't make sense to manual update the storage again with the missing files until the retrieval is fixed.

tslawecki commented 6 years ago

Greg - good catch. Please let Greg L and others know this may be an issue again this morning.

From: gcutrell notifications@github.com Sent: Thursday, December 21, 2017 8:08 AM To: glos/myglos Cc: Tad Slawecki; Mention Subject: Re: [glos/myglos] TDS GLCFS time format change (#171)

@cheryldmorsehttps://github.com/cheryldmorse The script that retrieves the GLCFS files is running without any errors in the log. The issue is it thinks it is saving data to storage but it isn't. Do you know what could have happened when storage hit capacity? Does something need a restart? The retrieval script/log is in process1 at /usr/local/glos/glcfs/ if you wanted to take a look at it. We are behind on GLCFS data but it doesn't make sense to manual update the storage again with the missing files until the retrieval is fixed.

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/glos/myglos/issues/171#issuecomment-353346603, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AE-3PkpRpum23OjVDoiBt95Lr34yNZSsks5tClhfgaJpZM4QOIhb.

kkoch commented 6 years ago

I've moved the Jeff Houghton query to a separate ticket.