search

simion1232006 / nctoolbox

Automatically exported from code.google.com/p/nctoolbox
0 stars 0 forks source link

Attempts to connect to datasets at http://www.esrl.noaa.gov often, but not always, fail #5

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
In matlab execute:

ds = 
ncdataset('http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis/s
urface_gauss/vwnd.10m.gauss.2009.nc')

One user reports the following stacktrace:

2010-02-08 16:19:46,408 [main] ERROR ucar.nc2.dods.DODSNetcdfFile  -
ERROR readDataDDSfromServer on ?lat,lon,time
java.io.EOFException
           at java.io.DataInputStream.readInt(Unknown Source)
           at opendap.dap.DVector.deserialize(DVector.java:219)
           at opendap.dap.DataDDS.readData(DataDDS.java:94)
           at opendap.dap.DConnect2$DataDDSCommand.process(DConnect2.java:715)
           at opendap.dap.DConnect2.openConnection(DConnect2.java:257)
           at opendap.dap.DConnect2.getData(DConnect2.java:699)
           at opendap.dap.DConnect2.getData(DConnect2.java:979)
           at ucar.nc2.dods.DODSNetcdfFile.readDataDDSfromServer(DODSNetcdfFile.java:1157)
           at ucar.nc2.dods.DODSNetcdfFile.readArrays(DODSNetcdfFile.java:1211)
           at ucar.nc2.dods.DODSNetcdfFile.<init>(DODSNetcdfFile.java:318)
           at ucar.nc2.dataset.NetcdfDataset.acquireDODS(NetcdfDataset.java:783)
           at ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:678)
           at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:434)
           at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:417)
           at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:402)
           at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:389)
Failed to open 
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis/surface_gaus
s/vwnd.10m.gauss.2009.nc

I get:
Failed to open
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/surf
ace_gauss/nbdsf.mon.ltm.nc

Caused by:
    Java exception occurred:
    java.io.IOException:
    http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/surface_gauss/nbdsf.mon.ltm.nc
    is not a valid URL.null
        at ucar.nc2.dataset.NetcdfDataset.isDODS(NetcdfDataset.java:360)
        at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:332)
        at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:175)
        at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:161)
        at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:149)

Just for grins I tried all the following and they all worked:
>> ds = 
ncdataset('http://motherlode.ucar.edu:8080/thredds/dodsC/satellite/WV/AK-NATIONA
L_8km/20100201/AK-
NATIONAL_8km_WV_20100201_2230.gini')
>> ds = 
ncdataset('http://michigan.glin.net/thredds/dodsC/bathy/crm_great_lakes')
>> ds = 
ncdataset('http://thredds1.pfeg.noaa.gov/thredds/dodsC/satellite/MB/chla/mday')

This was sent off to srl.psd.data@noaa.gov and they responded with:
"I really don't know what to tell you regarding your question #1.  I consulted 
with someone with greater expertise (at another site), and while 
he agreed with me that it was a communications problem, most likely a firewall 
issue at your end, neither of us had ever seen and EOF 
exception before.  Of course, neither of us is a Matlab user, either...."

Original issue reported on code.google.com by bschlin...@gmail.com on 9 Feb 2010 at 10:00

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
All the following failed to connect:

>> ds = 
ncdataset('http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/Timeseries/NTA.
EN')
>> ds = 
ncdataset('http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/noaa.oisst.v2/sst
.mnmean.nc')
>> ds = 
ncdataset('http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.d
erived/surface_gauss/nbdsf.mon.ltm.nc')
>> ds = 
ncdataset('http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis/s
urface/air.sig995.1957.nc')
>> ds = 
ncdataset('http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis/s
urface_gauss/vwnd.10m.gauss.2009.nc')

Original comment by bschlin...@gmail.com on 9 Feb 2010 at 10:13

GoogleCodeExporter commented 8 years ago 
Sent the following to esrl dot psd dot data at noaa dot gov:

Hi,

We're having trouble connecting to your thredds datasets at 
http://www.esrl.noaa.gov/psd/thredds/catalog.html using Netcdf-Java and 
nctoolbox (A Matlab toolbox that sits on top of Netcdf-Java). A summary of the 
issue 
is at http://code.google.com/p/nctoolbox/issues/detail?id=5.

In a nutshell, we're able to connect to other Thredds catalogs with nctoolbox 
and 
Netcdf-Java with no problems. However, we're having issues connecting to yours 
and 
we're just trying to figure out where the problem is occurring. You mentioned 
to Josh 
Plant that you suspect a firewall issue. Do you have any Matlab or Java users 
their 
that could try using nctoolbox or Netcdf-Java to open one of your datasets 
internally 
and verify that you're able to connect with those tools? Also, what OpenDAP 
clients 
are you using to access your thredds datasets? Finally, could you send me a 
opendap 
URL that you've verified works and I'll try testing with it.

Thanks!

References:
http://www.unidata.ucar.edu/software/netcdf-java/
http://code.google.com/p/nctoolbox/
http://code.google.com/p/nctoolbox/issues/detail?id=5
http://www.esrl.noaa.gov/psd/thredds/catalog.html

Original comment by bschlin...@gmail.com on 9 Feb 2010 at 10:26

GoogleCodeExporter commented 8 years ago 
Verified that the following URL works in pydap and Ferret:

http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.marine/sst.mean.nc

Original comment by bschlin...@gmail.com on 9 Feb 2010 at 10:33

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
Able to connect using Java (via a Groovy shell using netcdf-all-4.1.jar):

groovy:000> nc = 
ucar.nc2.dataset.NetcdfDataset.openDataset('http://www.esrl.noaa.gov/psd/thredds
/dodsC/Datasets/ncep.marine/sst.mean.nc')
===> netcdf 
dods://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.marine/sst.mean.nc {
 dimensions:
   lat = 90;
   lon = 180;
   time = UNLIMITED;   // (229 currently)
 variables:
   float sst(time=229, lat=90, lon=180);
     :_CoordinateAxes = "time lat lon ";
     :dataset = "NCEP Real-time Marine\nC";
     :var_desc = "Sea Surface Temperature\nS";
     :level_desc = "Surface\n0";
     :statistic = "Mean\nM";
     :parent_stat = "Individual Obs\nI";
     :actual_range = -2.8f, 36.0f; // float
     :precision = 2S; // short
     :units = "degC";
     :long_name = "Sea Surface Temperature Monthly Mean at Surface";
   float lat(lat=90);
     :long_name = "Latitude";
     :units = "degrees_north";
     :actual_range = 89.0f, -89.0f; // float
     :_CoordinateAxisType = "Lat";
   float lon(lon=180);
     :long_name = "Longitude";
     :units = "degrees_east";
     :actual_range = 1.0f, 359.0f; // float
     :_CoordinateAxisType = "Lon";
   double time(time=229);
     :units = "days since 1-1-1 00:00:0.0";
     :long_name = "Time";
     :delta_t = "0000-01-00 00:00:00";
     :avg_period = "0000-01-00 00:00:00";
     :actual_range = 726834.0, 733774.0; // double
     :_CoordinateAxisType = "Time";

 :platform = "Ship Observations";
 :title = "NCEP Real-time Marine";
 :history = "/home/hoop/crdc/oldCRDC2COARDSv3/oldCRDC2COARDS Fri Dec  8 20:50:59 1995 from sst.mean.nc\nman5tonetCDF Thu Sep  1 19:22:13 1994 from 
sst.mean\nEXTRACT.  94/09/01. 19.09.34. FROM CSMMI70-BR\nCAC-GTS  Sea surface 
temperat  Mean";
 :Conventions = "COARDS";
}

Hmmm, this makes me wonder if one of the Netcdf-Java dependencies included with 
Matlab is outdated. Why this is failing with only NOAA's THREDDS catalog is a 
mystery.

Original comment by bschlin...@gmail.com on 9 Feb 2010 at 10:52

GoogleCodeExporter commented 8 years ago 
Ahhh, I just noticed that there is some redirect going on. In the above, I 
submitted a 
'http' url, but the response is actually a 'dods' url. I tried the following 
using 
nctoolbox (notice I replaced 'http' with 'dods'):

>> ds = 
ncdataset('dods://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.marine/sst.
mean.nc')

ds = 

  ncdataset handle

  Properties:
       netcdf: [1x1 ucar.nc2.dataset.NetcdfDataset]
    variables: {4x1 cell}

  Methods, Events, Superclasses

Original comment by bschlin...@gmail.com on 9 Feb 2010 at 10:57

GoogleCodeExporter commented 8 years ago 
Sent the following to esrl dot psd dot data at noaa dot gov:

OK, it looks like you've got a redirect setup somewhere. Submitting an 'http' 
url 
results in a redirect to 'dods'. So when we try 
http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.marine/sst.mean.nc 
with nctoolbox, it fails but using
dods://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.marine/sst.mean.nc 
works. I think this may be failing in the Matlab environment because Matlab 
ships 
with a bunch of Java dependencies, and some may be outdated and have a bug that 
is failing on the redirect.

Original comment by bschlin...@gmail.com on 9 Feb 2010 at 11:02

GoogleCodeExporter commented 8 years ago 
Apparently, this redirect bug occurs in Matlab 2009a. Redirect works fine in 
Matlab 
2009b.

Original comment by bschlin...@gmail.com on 9 Feb 2010 at 11:14

GoogleCodeExporter commented 8 years ago 
At this point, the fix would be to find the offending library used by matlab, 
perhaps 
commons-net, and do the following:

1. Execute the following command at the MATLAB command prompt:
    edit classpath.txt

2. Find the line for the offending:
    $matlabroot/java/jarext/commons-net-1.0.0.jar

3. Copy a newer version of commons-net.jar to $NCTOOLBOX\java

3. Save the file and restart MATLAB.

However, given that this is resolved in the latest version of Matlab, I'm not 
going to 
spend time on this issue.

Original comment by bschlin...@gmail.com on 9 Feb 2010 at 11:21