ncWMS2 dataset says "updated", but doesn't seem to be updated

[rsignell-usgs]

I've got a UGRID ocean forecast model dataset that's working with ncWMS2 (the DAP URL is: http://www.smast.umassd.edu:8080/thredds/dodsC/FVCOM/NECOFS/Forecasts/NECOFS_GOM3_FORECAST.nc) but though I've got the auto-refresh rate set to 24 hours, and the dataset shows an update time from today (March 10) in the ncWMS Admin page: When I use godiva3 to read the data I don't have the updated data. It lists the last time available as March 7:

I tried doing CTRL+F5 and clicking "refresh" in godiva3, but that didn't help. Still get March 7 listed as the end date.

I also confirmed that this dataset in fact has data starting past Mar 10. See cell [3] here: http://nbviewer.jupyter.org/gist/rsignell-usgs/484b4a6732718c863018

Is this a bug or did I configure something wrong?

[rsignell-usgs]

@guygriffiths , I realize I should have provided the ncWMS2 endpoint, here: http://geoport.whoi.edu/ncWMS2/

[cseaton]

In case a second example is useful, I'm seeing the same problem (also with a UGRID dataset accessed via OpenDAP). In addition to not showing up in Godiva with the correct time range, getMetadata requests for timesteps and getMap requests with the correct time range for the current OpenDAP source return empty set and an error, respectively.

By comparing the Godiva image with an image of the model output generated through a different method, I was able to confirm that ncWMS2 is accessing the current version of the OpenDAP source, not the version from the date that matches the date it shows in the Godiva interface.

My first thought was that it might not be updating the units attribute of the time variable, which includes the start date of the source file (e.g. 'units: seconds since 2016-03-13 00:00:00 -8:00') and changes with each forecast day. However, the NECOFS_GOM3_FORECAST.nc file that @rsignell-usgs is working with has a units value of 'units: days since 1858-11-17 00:00:00' and I don't know if that changes with each forecast day.

My ncWMS2 endpoint is: http://wms.stccmop.org:8080/ncWMS2/

My openDAP source is: http://amb6400b.stccmop.org:8080/thredds/dodsC/model_data/forecast.html

My Godiva example is: http://amb6400b.stccmop.org:8080/ncWMS2/Godiva3.html?permalinking=true&bgmap=NaturalEarth%20WMS&server=http://amb6400b.stccmop.org:8080/ncWMS2/wms&numColorBands=250&logScale=false&bbox=-124.17768859863,45.852274322511,-123.29158782959,46.561154937743&abovemaxcolor=transparent&belowmincolor=transparent&nodatacolor=transparent&layer=f33_thredds/bottom_salt&time=2016-03-11T08:00:00.000Z&palette=default&style=default-scalar&scaleRange=0,34.5&displayScaleRange=0,34.5&opacity=1

My forecast image that matches the time that Godiva claims to be showing is: http://www.stccmop.org/forecasts/f33/2016-068/images/isosal_mouth_bottom/sal_mouth_bottom_000259200.gif

and the image from the current day that matches what Godiva is actually showing is: http://www.stccmop.org/forecasts/f33/2016-074/images/isosal_mouth_bottom/sal_mouth_bottom_000259200.gif

[rsignell-usgs]

I just checked again and the problem still exists. Even though the dataset is reporting being updated every 24 hours, the start/stop times in the getCaps are unchanged: http://geoport.whoi.edu/ncWMS2/wms?SERVICE=WMS&REQUEST=GetCapabilities&VERSION=1.1.1&DATASET=FVCOM-NECOFS-GOM3

[guygriffiths]

I think that this was being caused by a caching issue. I've made some changes in the develop branch which fix the issue for local datasets, but I haven't yet tested with remote datasets (I don't have any where the update time is short and predictable). If someone can check this and confirm it works then we can close it, otherwise I should be able to test it overnight on a remote dataset sometime next week.

[rsignell-usgs]

@guygriffiths , so I can try out this branch? https://github.com/Reading-eScience-Centre/edal-java/tree/develop

[guygriffiths]

Yup, that should have the latest changes

[rsignell-usgs]

Did the changes get pushed? This commit from a day ago looks like 2D lon/lat stuff? https://github.com/Reading-eScience-Centre/edal-java/commit/d496d83a09f7fbc842799b2cb3b0e08424b729f7

[guygriffiths]

I had thought so but I guess not. I've just pushed them now.

[rsignell-usgs]

@guygriffiths , okay, I built the war file from the develop branch:

git clone https://github.com/Reading-eScience-Centre/edal-java.git
git checkout develop
mvn package

and it's working, but I guess I won't know until tomorrow if it updated properly.

[rsignell-usgs]

indeed, it seems like the updating is working now. Awesome! @cseaton , if you want to try the new war, I put a copy at https://drive.google.com/open?id=0BzAHlPEEP_ujZEZnaDJ2N3l0Zm8

[guygriffiths]

Good to hear it :-)

[rsignell-usgs]

@guygriffiths , should this be left open until the improvements from the develop branch are incorporated into a new release on master?

[rsignell-usgs]

@guygriffiths ping

[guygriffiths]

Sorry, didn't spot this. Yes, it should probably remain open in case other people search for it.

[rsignell-usgs]

@guygriffiths , any reason not to issue a new release with the improvements in the develop branch that fix this issue?

[rsignell-usgs]

@guygriffiths , actually, I thought the improvements in the develop branch fixed the issue, but they did not. Our forecast models are still not getting updated.

[rjdave]

@guygriffiths I too am having issues with my two FMRC datasets getting updated. While only one is actually a forecast model, they both have not updated since they were added to the ncWMS2 server.

http://tds.marine.rutgers.edu/ncWMS2/Godiva3.html

ESPRESSO 4DVAR V2 and Macoora 6km Codar Totals are the datasets that source an FMRC dataset from THREDDS. The ESPRESSO dataset has a later end date because it's the forecast model. The CODAR is near-realtime observations. The dods URLs I'm using in ncWMS are:

dods://tds.marine.rutgers.edu/thredds/dodsC/roms/espresso/2013_da/avg/ESPRESSO_Real-Time_v2_Averages_Best

and

dods://tds.marine.rutgers.edu/thredds/dodsC/cool/codar/totals/5Mhz_6km_realtime_fmrc/Maracoos_5MHz_6km_Totals-FMRC_best.ncd

respectively. The other two datasets on my ncWMS2 server are static and simple NcML aggregations. The datasets are from a TDS 4.6.3 server running on tomcat 8.0.32 and java 1.8.0_74. The ncWMS server uses the same tomcat and java versions.

[rsignell-usgs]

@guygriffiths just want to make sure this is still on the radar. This seems to me a pretty significant issue, since it makes ncwms2 unusable for forecast models .

[guygriffiths]

Yep, still on the radar, but I'm quite busy with other work at the moment. I should hopefully find some time next week to look into this.

[guygriffiths]

@rsignell-usgs , I'm having a look at this, but I'm unable to reproduce the error.

For all local datasets, (tested with single file, glob aggregations and NcML aggregations) I can set the auto-refresh in the admin interface, change the data, and then check it after the enough time has elapsed, after which the data is up-to-date in Godiva (after clicking the Refresh button and reloading the variable in question).

For remote datasets, I have tested: dods://tds.marine.rutgers.edu/thredds/dodsC/cool/codar/totals/5Mhz_6km_realtime_fmrc/Maracoos_5MHz_6km_Totals-FMRC_best.ncd which @rjdave mentioned. I set the refresh interval to 1min, and waited until the dataset had updated on the remote server. Again, after refreshing in Godiva the updated data is available.

Can anyone give me further details on how to reproduce this problem?

[rjdave]

@guygriffiths There aren't really any details. The admin page says it has updated but there is no new data in godiva3 even after forced refresh in the admin page and refresh on the godiva3 page. Perhaps we aren't using the same .war file you are. Could you put your .war file somewhere we could download it from like dropbox or googledrive?

[guygriffiths]

@rjdave - You can download it from here: http://www.personal.rdg.ac.uk/~qx901922/ncWMS2.war

[rjdave]

@guygriffiths That WAR file doesn't work at all for me. It throws a bunch of warnings and errors in catalina.out (Attached)

catalina.out

This log is starting from a pretty clean setup of tomcat 8.0.32 and java 1.8.0_74. I took your war file and modified WEB-INF/web.xml to allow my digested password tomcat setup to work and I modified WEB-INF/classes/config.properties to avoid collisions with my production ncWMS.

When I start up the server it unpacks the WAR and creates $HOME/.ncWMS_test (the alternate I set in config.properties) and the log directory and an empty ncwms.log. You'll notice during startup tomcat complains about auth-constraint and security-role but this doesn't seem to affect anything because I'm still able to log in to the admin section.

When I go to Godiva3 I get a pop up that says "Server Error" "Error contacting the server"

I then add the CODAR dataset through the admin panel:

10-May-2016 12:20:35.182 INFO [load-metadata-Macoora6km_fmrc] org.geotoolkit.referencing.factory.epsg.EpsgInstaller.call Creating cached EPSG database version 7.09. This operation may take a few minutes...

But when I go back to Godiva3 I get the same error pop-up.

I then shutdown the server and get the warnings about JDBC and CacheManager then the big stack traces. The tomcat shutdown never completes and I have to kill it.

Sometimes on subsequent startups catalina.out says the config is invalid and deletes it. I believe this occurs if you just happen to shutdown the server while a dataset is auto-refreshing. This seems to leave config.xml either empty or incomplete. The real issue is that, the one time I looked at this in detail, my backup config was also blank so I had nothing to fall back on. I'm certain that just before the server shutdown both config.xml and config.xml.backup were non-empty and identical.

[guygriffiths]

@rjdave I've just performed a bugfix release which includes these changes. Can you try this new release? I've tried it on clean Tomcat 8 installation on my machine at it works as expected with the dataset: dods://tds.marine.rutgers.edu/thredds/dodsC/cool/codar/totals/5Mhz_6km_realtime_fmrc/Maracoos_5MHz_6km_Totals-FMRC_best.ncd

The release can be found here: https://github.com/Reading-eScience-Centre/edal-java/releases/tag/edal-1.1.2

[rjdave]

@guygriffiths I can confirm that the codar dataset does update properly with the new WAR. I can let you know about the other FMRC dataset a little after 7am EDT (11am UTC) tomorrow (Thursday).

[rjdave]

@guygriffiths Our daily FMRC forecast model run is also updating automatically as expected. It took me a minute to realize it because ncWMS2 doesn't display the latest time record available. It appears that it shows the latest time record for the current day. I had to click the date dropdown to see that new forecast records were added.

[guygriffiths]

OK, great to hear. @rsignell-usgs - I would guess this fixes the issues you were seeing as well? If you can test it, I'll close this issue.

Re: times - yes, Godiva displays the closest time to the present by default.