Open lesserwhirls opened 8 years ago
@lesserwhirls , just a note that the latest artifact https://artifacts.unidata.ucar.edu/content/repositories/unidata-snapshots/edu/ucar/tds/5.0.0-SNAPSHOT/tds-5.0.0-20160815.201802-16.war also exhibits this problem.
So this does work on our test datasets that use the dataset element, so there is something more to this. I'll take a deeper look at your configs.
I just tested the latest TDS 5.0 artifact from Sep 21, and still godiva 3 is not working for our datasets. We are getting these errors:
erviceExceptionReport xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.3.0" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wms/1.3.0/exceptions_1_3_0.xsd">
<ServiceException>
The location file:/home/rsignell/data/models/delft3d/vs_trim2nc/00_dir.ncml doesn't refer to any existing files.
</ServiceException>
</ServiceExceptionReport>
@lesserwhirls is there anything I could do here to help?
Oh, wait, is there a workaround such that I can just eliminate the urlPath
or id
entry in the catalog?
I only ask because most of the catalogs I've tested are written this way and are not currently working with ncWMS2 in TDS5.0. Yet many of these datasets would benefit strongly from the new features in ncWMS2, so we would really like to get this going!
@lesserwhirls ?
@rsignell-usgs - sorry this has taken so long to get to. Ill be digging into and working on this for the rest of the week until (hopefully) I can find a solution. I have some ideas, so maybe it won't take too crazy long.
@lesserwhirls, any update here? Guessing either (a) ran out of time to test, or (b) hit roadblocks. Hoping it's the former. :smiley_cat:
FWIW, huge improvements for unstructured grid and all projected data in today's release of ncWMS2: https://github.com/Reading-eScience-Centre/ncwms/releases/tag/ncwms-2.2.4 I hope that version can make it into THREDDS 5.0
Hey @rsignell-usgs - I thought I had replied to this already, but I'm not seeing it. My apologies.
Working with the global_bathy.xml
catalog you sent and the and the ETOPO2 Version 2c (2 arc minute - Worldwide) dataset
, I was able to get two things working by doing two things:
bathy
, which was used in the urlpath
of the datasetAbout number 2 above, the data file was incorrectly named as etopo2_v2c.nc
in the catalog instead of bathy_etopo2_v2c.nc
(which is what I was able to download from your TDS). I am wonder though, in general, if the missing datasetRoot
element is the key issue here. Once that is defined, maybe all of the datasets will work?
@lesserwhirls , I'm trying to get this datasetRoot
thing to work.
To be frank, this always confused me, and since my catalogs worked fine without it (at least until TDS 5.0), I never used it, except in datasetScan
catalogs.
So for the bathy catalog and etopo2 dataset you mention, my catalog entry looks like:
<dataset name="ETOPO2 Version 2c (2 arc minute - Worldwide)" ID="bathy/etopo2_v2c.nc"
urlPath="bathy/etopo2_v2c.nc">
...
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
location="/usgs/data0/bathy/ETOPO2v2c_i2.nc">
<variable name="topo">
<attribute name="units" value="meters"/>
<attribute name="long_name" value="Topography"/>
</variable>
<attribute name="title" value="NGDC ETOPO2v2c Global DEM (2 min))"/>
</netcdf>
</dataset>
My understanding was that because I point to the full path name to the datafile explicitly inside the netcdf tags, the ID
and urlPath
I can pick to be whatever I want -- it's just what the user sees as the URL. Is that not correct?
So if these catalogs no longer work, it's a backwards compatibility issue. I guess that's okay since you get to break the API in a major version, but it at all possible, it would be nice if these catalogs would still work. It's not just the 30 catalogs I have, but all the folks I've helped make catalogs also...
BTW, I tried adding
<datasetRoot path="bathy" location="/usgs/data0/bathy/" />
to the top of the file but that didn't work. Perhaps you could point me toward the version you used?
According to the docs for 5.0 (not official), you should not need to define a datasetRoot when you are using NcML to point to a particular data file, like you are doing:
So, this is a bug that should be fixed before a release. Of course, bug or not, I'd say we would want this to work no matter what, as you have put a lot of effort into help others get things up and running (greatly appreciated, btw!).
In terms of adding the datasetRoot, I also had to change the name of the file used in the URL path to use the name of the file on disk - from etopo2_v2c.nc
to bathy_etopo2_v2c.nc
.
Ok, I finally found the underlying issue. Ultimately, it has to do with the handoff of datasets between netCDF-Java/TDS and edal-java when NcML is involved.
@guygriffiths - there does not appear to be a way to cleanly use CdmDatasetFactory.createDataset
when the datafile on disk is being modified by NcML. One simple way would be to create a signature for createDataset
that takes a NetcdfDataset
object. However, it seems like later in the edal-java stack, like in CdmGridDatasetFactory.generateDataset
, the dataset being used needs an actual location on disk, rather than a virtual thing like an NcML dataset. Is that correct? Any ideas on how to get edal-java and NcML datasets (when there isn't an NcML file on disk) to play nice together?
@lesserwhirls - Currently we use the location to create a NetcdfDataset object, from the NetcdfDatasetAggregator class. This works by either just opening the file from disk, or creating an in-memory NcML dataset to aggregate glob expressions. It also keeps a cache of the 20 most recently used datasets so as to avoid having to re-create them each time they are needed. The various grid dataset classes store the location and then use this class to obtain a NetcdfDataset when they need it (which may or may not retrieve it from the cache).
I propose that I create a protected method in CdmDatasetFactory called something like getDatasetFromLocation, which I then delegate to NetcdfDatasetAggregator. You'd need to create a subclass of CdmGridDatasetFactory which overrides this method for cases where the dataset has been modified by NcML (or for all cases if you prefer). From what I recall of the TDS code it's probably the case that you'll create a new ThreddsGridDatasetFactory for each dataset, and getDatasetFromLocation will just ignore the location parameter and just return the appropriate NetcdfDataset object (which you've used in the constructor to ThreddsGridDatasetFactory). Does that sound like it would work?
I was also going to suggest the following:
I create a NetcdfDatasetAggregator.addDataset(String location, NetcdfDataset dataset) method, which will add a specific location -> NetcdfDataset object mapping to the cache. I would also make it mark the dataset as "active", which means that it will not be expired until it has been finished with. In ThreddsWmsCatalogue you'd have to insert the object into the cache each time you wanted to create it.
But whilst writing it down, I came to think that has a lot more potential for weird bugs, and it feels a bit hacky.
Either would work from my point of view, but I'm always in favor of reducing the number of places weird bugs can crop up (we have too many of those already laughs). How hard would it be for you to implement the first option? I don't want to burden you.
Not hard at all. I did it earlier 😁 I should test it a bit more tomorrow, but it'll be in the next release.
Is this in the develop branch?
Oh awesome @guygriffiths!
@rsignell-usgs Not yet, it's still on my local machine until I've done some more tests. Should be there tomorrow.
Seems to all be working fine, so I've just pushed this to github on the develop branch. There's now a protected method in CdmDatasetFactory
called getNetcdfDatasetFromLocation
which takes 2 parameters - a location String and a boolean for whether or not to force a refresh (i.e. re-scan location for new files or changed data). The only place in our code that forceRefresh
will be set to true
is in our XML dataset catalogue (which THREDDS replaces), so you can safely ignore it.
Let me know if you encounter any difficulties with it.
Greetings @guygriffiths, I'm looking into this now and see that CdmGridDatasetFactory
is marked as final, so I am unable to subclass it.
@lesserwhirls Sorry, didn't spot that. I've updated it in the develop
branch - are you using a local version or do you need a release for it?
@guygriffiths - no worries! I have a local version, so we're good, and everything is working now. Thanks!
This seems to be working now. Good to close?
woohoo! Good to close!
On Tue, Dec 13, 2016 at 4:12 PM, Rich Signell notifications@github.com wrote:
This seems to be working now. Good to close?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Unidata/thredds/issues/618#issuecomment-266863551, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEGGFPU92h5m1si-Kb0Gpu-GoEzKYq2ks5rHwpZgaJpZM4JsJYU .
I'm not sure whether something broke again, but with the latest Docker thredds 5.0 snapshot, the godiva3 links on my datasets are not working, and also the WMS endpoints don't work in TerriaJS as they do with thredds 4.6.
For example on this FMRC collection of netcdf files:
The godiva3 link https://gamone.whoi.edu/thredds/Godiva.html?server=http://gamone.whoi.edu/thredds/wms/coawst_4/use/fmrc/coawst_4_use_best.ncd is returning "error getting data from server" and the developer console shows two errors:
https://gamone.whoi.edu/thredds/getconfig
is returning status 404
http://gamone.whoi.edu/thredds/wms/coawst_4/use/fmrc/coawst_4_use_best.ncd?request=GetMetadata&item=menu is returning status (blocked:mixed-content)
I also tried dropping the WMS endpoint into TerriaJS, and Terria is making getMap requests like:
which are returning 404.
I tried stripping this down and just adding parameters one by one.
This request: https://gamone.whoi.edu/thredds/wms/coawst_4/use/fmrc/coawst_4_use_best.ncd?srs=EPSG%3A3857&transparent=true&format=image%2Fpng&request=GetMap&version=1.1.1&layers=Hwave&bbox=-20037508.342789244%2C0%2C-10018754.171394622%2C10018754.171394622 returns the response: "Must provide a value for parameter WIDTH"
But when we add width and height parameters, we get the 404:
@guygriffiths ?
I'm also getting
2017-12-15T00:43:08.833 +0000 [ 110403][ 2] WARN - uk.ac.rdg.resc.edal.wms.WmsServlet - Wms Exception caught: "Problem creating dataset fx3/gbr4_rivers_simple_2017-10.nc at /usr/local/tomcat/content/thredds/public/fx3/gbr4_rivers_simple_2017-10.nc" from:uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory:115
from the 21-sept 2017 5.0-SNAPSHOT
- using a configuration that works fine with 4.6.11
opened new issue #977 for this.
So I'm looking into this today after having upgraded the code base to the latest version of edal-java
, and I get the following message when loading Godiva3:
com.google.gwt.logging.client.LogConfiguration
SEVERE: (TypeError) : Cannot read property 'length' of undefinedcom.google.gwt.core.client.JavaScriptException: (TypeError) : Cannot read property 'length' of undefined
at Unknown.hgb(godiva3-0.js)
at Unknown.new wgb(godiva3-0.js)
at Unknown.Kcb(godiva3-0.js)
at Unknown.Lcb(godiva3-0.js)
at Unknown.cdb(godiva3-0.js)
at Unknown.vo(godiva3-0.js)
at Unknown.Ho(godiva3-0.js)
at Unknown.eval(godiva3-0.js)
at Unknown.xg(godiva3-0.js)
at Unknown.Ag(godiva3-0.js)
at Unknown.eval(godiva3-0.js)
As for the wms service, I am able to successfully request images. Note that this is using Rich's example from this ticket (here)
@guygriffiths - any ideas on what could be going on with Godiva?
@lesserwhirls - Not sure what's going on from that (the Godiva3 stuff is helpfully minified and obfuscated). Is this on a public server anywhere I could get the data from using OPeNDAP?
@guygriffiths I've added links and a little info to https://github.com/Unidata/thredds/issues/977#issuecomment-361442032
@guygriffiths - sorry for the delay. Here is an example dataset:
and its corresponding OPeNDAP endpoint:
Thank you for taking a look!
@lesserwhirls - I haven't had a chance to look into this specifically because I've been working on another project which uses EDAL. However, that project threw up the same bug! I've just release a version 1.3.1 containing a fix.
Excellent - thank you for the update @guygriffiths! I'll update to 1.3.1 and let you know how it goes.
Perfect - works well locally. I'll upgrade our dev server once I get to work and will report back, but looks like 1.3.1 did the trick! Thank you once again @guygriffiths!
@lesserwhirls and @cwardgar , godiva3 still isn't working for me for netcdf files under a datasetScan.
Here is godiva2 on thredds 4.6.10 accessing one such file: http://geoport-dev.whoi.edu/thredds/godiva2/godiva2.html?server=http://geoport-dev.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc
Here is godiva3 on thredds 5.0 accessing the same file: https://gamone.whoi.edu/thredds/Godiva.html?server=http://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc
If I turn on developer tools, I can see that godiva3 is calling https://gamone.whoi.edu/thredds/getconfig which is returning a 404.
This is using the thredds 5.0 snapshot docker container from 5 days ago https://hub.docker.com/r/unidata/thredds-docker/tags/
Hmm, okay, according to Guy the getconfig
failure can be ignored.
So some other problem.
@lesserwhirls, can you try accessing a netcdf file in a datasetScan
with godiva3 and see if it works for you?
Interesting. The WMS server is working:
but godiva3 is bombing out on a null value:
Thu Feb 22 07:38:37 GMT-700 2018 com.google.gwt.logging.client.LogConfiguration
SEVERE: (TypeError) : null is not an object (evaluating 'a.db')com.google.gwt.core.client.JavaScriptException: (TypeError) : null is not an object (evaluating 'a.db')
at Unknown.bN(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
at Unknown.mdb(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
at Unknown.ucb(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
at Unknown.vcb(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
at Unknown.Ocb(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
at Unknown.vo(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
at Unknown.Ho(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
at Unknown.anonymous(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
at Unknown.xg(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
at Unknown.Ag(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
at Unknown.anonymous(https://gamone.whoi.edu/thredds/Godiva.html?server=https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc)
I have not been able to successfully debug godiva3
, as the javascript has been transformed by gwt. @guygriffiths - would you be able to take a look? The OPeNDAP
endpoint is:
https://gamone.whoi.edu/thredds/dodsC/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc
Actually, now that I have actually updated to the 5.0-SNAPSHOT
container (instead of 5.0
), my WMS endpoints are not working:
https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc?service=WMS&version=1.3.0&request=GetCapabilities
Hi @rsignell-usgs - can you clear your cache directory and retry?
Blew away the cache, but same problem: https://gamone.whoi.edu/thredds/wms/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc?service=WMS&version=1.3.0&request=GetCapabilities
I should also note that for this dataset:
unidata/thredds-docker:latest
container, just not the unidata/thredds-docker:5.0-SNAPSHOT
Ok, I see an error being generated in the ncWMS code:
018-02-23T10:07:28.483 -0700 [ 204323][ 17] WARN - uk.ac.rdg.resc.edal.dataset.cdm.NetcdfDatasetAggregator - Dataset /Users/sarms/dev/unidata/content/5.0/thredds/public/testdata/rsignell/coawst_us_20171001_01.nc is not in active dataset list but has been asked to be released! This is not harmful in itself but may indicate a coding error whereby a dataset has been marked to be released from the cache multiple times.
2018-02-23T10:07:28.484 -0700 [ 204324][ 17] WARN - uk.ac.rdg.resc.edal.wms.WmsServlet - Wms Exception caught: "Problem creating dataset testAll/rsignell/coawst_us_20171001_01.nc at /Users/sarms/dev/unidata/content/5.0/thredds/public/testdata/rsignell/coawst_us_20171001_01.nc" from:uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory:115 Cause: null
I'm digging into it to see what is happening, but it does not happen on other datasetScan
datasets that I've setup.
I'm so glad you can reproduce! The only things I saw in the logs were some H5 warnings...
So it looks like things are going wrong in uk.ac.rdg.resc.edal.dataset.cdm.CdmDatasetFactory#createDataset(java.lang.String, java.lang.String, boolean)
, perhaps due to the file following CF & SGRID and netCDF-Java not doing anything with SGRID? @guygriffiths - any thoughts on this one?
@guygriffiths - I am also seeing this kind of message in the logs:
2018-02-26T10:11:23.322 -0700 [ 1460154][ 336] WARN - uk.ac.rdg.resc.edal.dataset.cdm.NetcdfDatasetAggregator - Dataset /Users/sarms/dev/unidata/content/5.0/thredds/public/testdata/rsignell/coawst_us_20171001_01.nc is not in active dataset list but has been asked to be released! This is not harmful in itself but may indicate a coding error whereby a dataset has been marked to be released from the cache multiple times.
But I am getting this for other datasets as well (non SGRID), not just this one. Maybe the file is being prematurely released?
Not sure what's happening, I haven't seen this one before. Any chance I could get a copy of the data? Doing a download from the dataset @rsignell-usgs linked to just give be a 403 error.
@guygriffiths , I just tried the link and it worked fine for me: https://gamone.whoi.edu/thredds/fileServer/sand/usgs/users/rsignell/data/COAWST/coawst_us_20171001_01.nc
But if you don't want to download a 1.7GB file, I made a one-time-step version (170mb) you can download here: https://gamone.whoi.edu/thredds/fileServer/sand/usgs/users/rsignell/data/COAWST/coawst_1step.nc
I've had a quick look and I think it's to do with the fact that some of the variables (mask_psi
, wetdry_mask_psi
) are defined to be on a staggered grid with the attribute location = "node"
- i.e. not staggered at all.
I'll have a look at fixing it at some point tomorrow, but for the moment you can get around it by removing the grid
and location
attributes from those variables.
Excellent - thank you for your help @guygriffiths!
The issue with SGRID has been fixed in the 1.4.0 release, along with a couple of other SGRID issues.
Not sure about the dataset release one, I'm not getting that with a basic ncWMS2 installation. It's also not so important - it won't cause any actual problems in itself.
Thank you @guygriffiths!
ncWMS is not working when a thredds config catalog is using the dataset element. The issue is that the the dataset urlPath is being directly passed into the call to open a dataset, and thus there is an error as the file does not exists.
This is happening using all versions of 5.0, including the latest published snapshot at this time:
http://artifacts.unidata.ucar.edu/content/repositories/unidata-snapshots/edu/ucar/tds/5.0.0-SNAPSHOT/tds-5.0.0-20160808.141712-14.war
This is a continuation of the disccusion here
Here is the stacktrace showing the error, as found in catalina.out:
It should be noted in the dataset element, the attribute
urlPath
was set tourlPath
, andID
was set toid
.Also running into this same thing for the threddsIso services, as well as WCS.