search

MetcalfeB / nctoolbox

Automatically exported from code.google.com/p/nctoolbox
0 stars 0 forks source link

Issue with JGOFS OPenDAP dataset. #3

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
I’m trying to use the ‘Data URL’ address from the following DODS Dataset 
Access Form webpage:

http://usjgofs.whoi.edu/dods-bin/nph-
dods/jgofs/merged_objects/US_JGOFS/Equatorial_Pacific/bottle_eqpac.html

I thought I could use ncdataset.  When I look at the variables it just gives me 
one, ‘Level_0’, and 
when I try to use the ‘data’ method it gives me errors: 
opendap.dap.DAP2Exception: "Could not 
open ctd_eqpac(Level_0): l_jdbopen, &x Bad Name: Level_0: "

Am I not using this correctly?  Is this not a dataset that I can use with 
ncdataset?  I don’t 
understand why there is only one variable.  The ascii output from the DODS page 
has variable 
names like ‘Level_0.depth’, can this be causing a problem?

Original issue reported on code.google.com by bschlin...@gmail.com on 24 Nov 2009 at 7:36

GoogleCodeExporter commented 8 years ago 
Information regarding JGOFS data:
http://opendap.org/api/ws-html/writing_server_2.html
http://opendap.org/user/guide-html/guide_57.html
http://usjgofs.whoi.edu/datasys/jgdb_docs/jgsys.html

Original comment by bschlin...@gmail.com on 24 Nov 2009 at 9:16

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
DDL for offending dataset:

netcdf dods://usjgofs.whoi.edu/dods-bin/nph-
dods/jgofs/merged_objects/US_JGOFS/Equatorial_Pacific/bottle_eqpac {
 variables:

   Structure {
     String object_ID;
     String project;
     String cruise_type;
     String cruise_id;
     String date;
     String event;
     String sta;
     String cast;
     String lon;
     String lat;
     String MLD_0d03_dp;
     String MLD_0d125_dp;
     String depth_start_dens;
     String depth;
     String press;
     String bot;
     String qflag;
     String sal_bot;
     String sal_ctd;
     String temp;
     String potemp;
     String sigma_t;
     String sigma_0;
     String PMC_vol_filt;
     String PMC_kg;
     String NH4;
     String NO2;
     String NO3;
     String O2_ml_L;
     String pic;
     String PO4;
     String POC;
     String PON;
     String SiO4;
     String TOC;
     String TOC_kg;
     String TPC;
     String Urea;
     String pCO2_20;
     String pCO2_is;
     String TALK;
     String TCO2;
     String TCO2_man;
     String allox;
     String carotene_a;
     String carotene_b;
     String chlide_a;
     String chl_a;
     String chl_a_allo;
     String chl_a_prime;
     String chl_b;
     String chl_c1_c2;
     String chl_c3;
     String chl_c4;
     String diadinox;
     String diatox;
     String fucox;
     String fucox_but;
     String fucox_hex;
     String peridinin;
     String phyco;
     String prasinox;
     String violax;
     String zeax;
     String chl_a_fluor;
     String fluor_vol_filt;
     String Fo_to_Fa;
     String phaeo;
     String foram_lg_C;
     String acanth;
     String acanth_C;
     String cocc_gt15;
     String cocc_gt15_C;
     String foram;
     String foram_C;
     String foram_lg;
     String rad;
     String rad_C;
     String rad_lg;
     String rad_lg_C;
     String bact_het_cyt;
     String bact_het_mic;
     String bact_het_cellv;
     String coccus_p;
     String coccus_s;
     String coccus_s_mic;
     String coccus_s_mic_biov;
     String coccus_s_mic_C;
     String coccus_p_cyt;
     String coccus_s_cyt;
     String ciliates;
     String ciliates_C;
     String diatom_gt15_C;
     String diatom_dead;
     String diatom_gt15;
     String dino_gt15;
     String dino_gt15_C;
     String dino_het;
     String dino_het_C;
     String dino_het_lt20;
     String dino_het_lt20_C;
     String dino_het_lt20_cellv;
     String dino_het_t;
     String dino_het_t_C;
     String phyto_e_n;
     String phyto_e_u;
     String phyto_e_u_cyt;
     String phyto_oth_gt15;
     String phyto_oth_gt15_C;
     String hepnp_mic;
     String hepnp_mic_biov;
     String hepnp_mic_C;
     String pepnp_mic;
     String pepnp_mic_biov;
     String pepnp_mic_C;
     String leuc_incorp;
     String thy_incorp;
     String delta13C;
     String delta18O;
   } Level_0(*);

 :object_ID.width = "14";
 :project.width = "18";
 :cruise_type.width = "18";
 :cruise_id.width = "9";
 :date.units = "YYYYMMDD";
 :date.width = "8";
 :event.units = "MMDDhhmm";
 :event.width = "8";
 :sta.width = "3";
 :cast.width = "4";
 :lon.units = "decimal_degrees";
 :lon.width = "9";
 :lat.units = "decimal_degrees";
 :lat.width = "8";
 :MLD_0d03_dp.units = "decibars";
 :MLD_0d03_dp.width = "11";
 :MLD_0d125_dp.units = "decibars";
 :MLD_0d125_dp.width = "12";
 :depth_start_dens.units = "decibars";
 :depth_start_dens.width = "16";
 :depth.units = "meters";
 :depth.width = "6";
 :press.units = "decibars";
 :press.width = "6";
 :bot.width = "3";
 :qflag.width = "5";
 :sal_bot.width = "7";
 :sal_ctd.width = "7";
 :temp.units = "degrees_C";
 :temp.width = "7";
 :potemp.units = "degrees_C";
 :potemp.width = "9";
 :sigma_t.units = "kilograms/meter^3";
 :sigma_t.width = "8";
 :sigma_0.units = "kilograms/meter^3";
 :sigma_0.width = "9";
 :PMC_vol_filt.units = "kilograms";
 :PMC_vol_filt.width = "12";
 :PMC_kg.units = "micrograms/kilogram";
 :PMC_kg.width = "6";
 :NH4.units = "micromoles_N/liter";
 :NH4.width = "4";
 :NO2.units = "micromoles/liter";
 :NO2.width = "4";
 :NO3.units = "micromoles/liter";
 :NO3.width = "7";
 :O2_ml_L.units = "mls/liter";
 :O2_ml_L.width = "7";
 :pic.units = "ugC/l";
 :pic.width = "5";
 :PO4.units = "micromoles/liter";
 :PO4.width = "4";
 :POC.units = "micromoles/liter";
 :POC.width = "5";
 :PON.units = "micromoles/liter";
 :PON.width = "5";
 :SiO4.units = "micromoles/liter";
 :SiO4.width = "6";
 :TOC.units = "micromoles_C/liter";
 :TOC.width = "5";
 :TOC_kg.units = "micromoles_C/kilogram";
 :TOC_kg.width = "6";
 :TPC.units = "micromoles/liter";
 :TPC.width = "5";
 :Urea.units = "micromoles_N/liter";
 :Urea.width = "5";
 :pCO2_20.units = "microatmospheres";
 :pCO2_20.width = "7";
 :pCO2_is.units = "microatmospheres";
 :pCO2_is.width = "7";
 :TALK.units = "micromoles/kilogram";
 :TALK.width = "7";
 :TCO2.units = "micromoles/kilogram";
 :TCO2.width = "6";
 :TCO2_man.units = "micromoles_C/kilogram";
 :TCO2_man.width = "8";
 :allox.units = "nanograms/liter";
 :allox.width = "5";
 :carotene_a.units = "nanograms/liter";
 :carotene_a.width = "10";
 :carotene_b.units = "nanograms/liter";
 :carotene_b.width = "10";
 :chlide_a.units = "nanograms/liter";
 :chlide_a.width = "8";
 :chl_a.units = "nanograms/liter";
 :chl_a.width = "5";
 :chl_a_allo.units = "nanograms/liter";
 :chl_a_allo.width = "10";
 :chl_a_prime.units = "nanograms/liter";
 :chl_a_prime.width = "11";
 :chl_b.units = "nanograms/liter";
 :chl_b.width = "5";
 :chl_c1_c2.units = "nanograms/liter";
 :chl_c1_c2.width = "9";
 :chl_c3.units = "nanograms/liter";
 :chl_c3.width = "6";
 :chl_c4.units = "nanograms/liter";
 :chl_c4.width = "6";
 :diadinox.units = "nanograms/liter";
 :diadinox.width = "8";
 :diatox.units = "nanograms/liter";
 :diatox.width = "6";
 :fucox.units = "nanograms/liter";
 :fucox.width = "5";
 :fucox_but.units = "nanograms/liter";
 :fucox_but.width = "9";
 :fucox_hex.units = "nanograms/liter";
 :fucox_hex.width = "9";
 :peridinin.units = "nanograms/liter";
 :peridinin.width = "9";
 :phyco.units = "nanograms/liter";
 :phyco.width = "5";
 :prasinox.units = "nanograms/liter";
 :prasinox.width = "8";
 :violax.units = "nanograms/liter";
 :violax.width = "6";
 :zeax.units = "nanograms/liter";
 :zeax.width = "4";
 :chl_a_fluor.units = "micrograms/liter";
 :chl_a_fluor.width = "11";
 :fluor_vol_filt.units = "liters";
 :fluor_vol_filt.width = "14";
 :Fo_to_Fa.width = "8";
 :phaeo.units = "micrograms/liter";
 :phaeo.width = "5";
 :foram_lg_C.units = "nanograms_C/liter";
 :foram_lg_C.width = "10";
 :acanth.units = "cells/liter";
 :acanth.width = "6";
 :acanth_C.units = "nanograms_C/liter";
 :acanth_C.width = "8";
 :cocc_gt15.units = "cells/liter";
 :cocc_gt15.width = "9";
 :cocc_gt15_C.units = "picograms_C/liter";
 :cocc_gt15_C.width = "11";
 :foram.units = "cells/liter";
 :foram.width = "5";
 :foram_C.units = "nanograms_C/liter";
 :foram_C.width = "7";
 :foram_lg.units = "cells/liter";
 :foram_lg.width = "8";
 :rad.units = "cells/liter";
 :rad.width = "4";
 :rad_C.units = "nanograms_C/liter";
 :rad_C.width = "5";
 :rad_lg.units = "cells/liter";
 :rad_lg.width = "6";
 :rad_lg_C.units = "nanograms_C/liter";
 :rad_lg_C.width = "8";
 :bact_het_cyt.units = "cells/ml";
 :bact_het_cyt.width = "12";
 :bact_het_mic.units = "cells/ml";
 :bact_het_mic.width = "12";
 :bact_het_cellv.units = "cubic_microns";
 :bact_het_cellv.width = "14";
 :coccus_p.units = "cells/ml";
 :coccus_p.width = "8";
 :coccus_s.units = "cells/ml";
 :coccus_s.width = "8";
 :coccus_s_mic.units = "cells/ml";
 :coccus_s_mic.width = "12";
 :coccus_s_mic_biov.units = "cubic_microns";
 :coccus_s_mic_biov.width = "17";
 :coccus_s_mic_C.units = "micrograms_C/liter";
 :coccus_s_mic_C.width = "14";
 :coccus_p_cyt.units = "cells/ml";
 :coccus_p_cyt.width = "12";
 :coccus_s_cyt.units = "cells/ml";
 :coccus_s_cyt.width = "12";
 :ciliates.units = "cells/liter";
 :ciliates.width = "8";
 :ciliates_C.units = "nanograms_C/liter";
 :ciliates_C.width = "10";
 :diatom_gt15_C.units = "picograms_C/liter";
 :diatom_gt15_C.width = "13";
 :diatom_dead.units = "cells/liter";
 :diatom_dead.width = "11";
 :diatom_gt15.units = "cells/liter";
 :diatom_gt15.width = "11";
 :dino_gt15.units = "cells/liter";
 :dino_gt15.width = "9";
 :dino_gt15_C.units = "picograms_C/liter";
 :dino_gt15_C.width = "11";
 :dino_het.units = "cells/liter";
 :dino_het.width = "8";
 :dino_het_C.units = "micrograms_C/liter";
 :dino_het_C.width = "10";
 :dino_het_lt20.units = "cells/ml";
 :dino_het_lt20.width = "13";
 :dino_het_lt20_C.units = "micrograms_C/liter";
 :dino_het_lt20_C.width = "15";
 :dino_het_lt20_cellv.units = "cubic_microns/cell";
 :dino_het_lt20_cellv.width = "19";
 :dino_het_t.units = "cells_liter";
 :dino_het_t.width = "10";
 :dino_het_t_C.units = "nanograms_C/liter";
 :dino_het_t_C.width = "12";
 :phyto_e_n.units = "cells/ml";
 :phyto_e_n.width = "9";
 :phyto_e_u.units = "cells/ml";
 :phyto_e_u.width = "9";
 :phyto_e_u_cyt.units = "cells/ml";
 :phyto_e_u_cyt.width = "13";
 :phyto_oth_gt15.units = "cells/liter";
 :phyto_oth_gt15.width = "14";
 :phyto_oth_gt15_C.units = "picograms_C/liter";
 :phyto_oth_gt15_C.width = "16";
 :hepnp_mic.units = "cells/liter";
 :hepnp_mic.width = "9";
 :hepnp_mic_biov.units = "cubic_microns";
 :hepnp_mic_biov.width = "14";
 :hepnp_mic_C.units = "micrograms_C/liter";
 :hepnp_mic_C.width = "11";
 :pepnp_mic.units = "cells/liter";
 :pepnp_mic.width = "9";
 :pepnp_mic_biov.units = "cubic_microns";
 :pepnp_mic_biov.width = "14";
 :pepnp_mic_C.units = "micrograms_C/liter";
 :pepnp_mic_C.width = "11";
 :leuc_incorp.units = "picomoles/liter/hour";
 :leuc_incorp.width = "11";
 :thy_incorp.units = "picomoles/liter/hour";
 :thy_incorp.width = "10";
 :delta13C.units = "per_mil";
 :delta13C.width = "8";
 :delta18O.units = "no_units";
 :delta18O.width = "8";
}

Original comment by bschlin...@gmail.com on 24 Nov 2009 at 9:39

GoogleCodeExporter commented 8 years ago 
In matlab I tried the following:

ds = cfdataset('http://usjgofs.whoi.edu/dods-bin/nph-
dods/jgofs/merged_objects/US_JGOFS/Equatorial_Pacific/bottle_eqpac');
vAll = ds.netcdf.getVariables();
v = vAll.get(0); % 'Level_0' Structure
vForam = v.findVariable('foram');
data = vForam.read()

And it throws the following Exception:
opendap.dap.DAP2Exception: "Could not open bottle_eqpac(Level_0): l_jdbopen, &x 
Bad Name: Level_0: "
    at opendap.dap.DConnect2.openConnection(DConnect2.java:233)
    at opendap.dap.DConnect2.getData(DConnect2.java:699)
    at opendap.dap.DConnect2.getData(DConnect2.java:979)
    at ucar.nc2.dods.DODSNetcdfFile.readDataDDSfromServer(DODSNetcdfFile.java:1125)
    at ucar.nc2.dods.DODSNetcdfFile.readData(DODSNetcdfFile.java:1289)
    at ucar.nc2.Variable._read(Variable.java:927)
    at ucar.nc2.Variable.read(Variable.java:791)
    at ucar.nc2.Variable._read(Variable.java:914)
    at ucar.nc2.Variable.read(Variable.java:791)
    at ucar.nc2.dataset.VariableDS._read(VariableDS.java:469)
    at ucar.nc2.Variable.read(Variable.java:791)
??? Java exception occurred:
java.io.IOException: "Could not open bottle_eqpac(Level_0): l_jdbopen, &x Bad 
Name: Level_0: "
    at ucar.nc2.dods.DODSNetcdfFile.readData(DODSNetcdfFile.java:1296)
    at ucar.nc2.Variable._read(Variable.java:927)
    at ucar.nc2.Variable.read(Variable.java:791)
    at ucar.nc2.Variable._read(Variable.java:914)
    at ucar.nc2.Variable.read(Variable.java:791)
    at ucar.nc2.dataset.VariableDS._read(VariableDS.java:469)
    at ucar.nc2.Variable.read(Variable.java:791)

Original comment by bschlin...@gmail.com on 24 Nov 2009 at 9:47

GoogleCodeExporter commented 8 years ago 
Message sent to netcdf-java at unidata dot ucar dot edu and support at opendap 
dot org:

Hi All,

I'm trying to access JGOFS bottle data using NetCDF-Java. The JGOFS data is 
being served via OPeNDAP at the 
URL: http://usjgofs.whoi.edu/dods-bin/nph-
dods/jgofs/merged_objects/US_JGOFS/Equatorial_Pacific/bottle_eqpac. The DDL can 
be viewed at 
http://usjgofs.whoi.edu/dods-bin/nph-
dods/jgofs/merged_objects/US_JGOFS/Equatorial_Pacific/bottle_eqpac

Here's the issue:

---- I'm actually using the NetCDF-Java inside of Matlab (see 
http://code.google.com/p/nctoolbox/). The 
versions of the Java jars are:
    netcdf-4.0
    opendap-2.1
---- Here's the steps I'm taking in Matlab. (It looks like Java pseudo-code) :
    url = 'http://usjgofs.whoi.edu/dods-bin/nph-
dods/jgofs/merged_objects/US_JGOFS/Equatorial_Pacific/bottle_eqpac'
    netcdf = ucar.nc2.dataset.NetcdfDataset.openDataset(url);
     vars = netcdf.getVariables();
     v = vars.get(0);
     foram = v.findVariable('foram');
     data = foram.read();

When I make the call to 'read()' the following Exception is thrown:

opendap.dap.DAP2Exception: "Could not open bottle_eqpac(Level_0): l_jdbopen, &x 
Bad Name: Level_0: "
    at opendap.dap.DConnect2.openConnection(DConnect2.java:233)
    at opendap.dap.DConnect2.getData(DConnect2.java:699)
    at opendap.dap.DConnect2.getData(DConnect2.java:979)
    at ucar.nc2.dods.DODSNetcdfFile.readDataDDSfromServer(DODSNetcdfFile.java:1125)
    at ucar.nc2.dods.DODSNetcdfFile.readData(DODSNetcdfFile.java:1289)
    at ucar.nc2.Variable._read(Variable.java:927)
    at ucar.nc2.Variable.read(Variable.java:791)
    at ucar.nc2.Variable._read(Variable.java:914)
    at ucar.nc2.Variable.read(Variable.java:791)
    at ucar.nc2.dataset.VariableDS._read(VariableDS.java:469)
    at ucar.nc2.Variable.read(Variable.java:791)
??? Java exception occurred:
java.io.IOException: "Could not open bottle_eqpac(Level_0): l_jdbopen, &x Bad 
Name:
Level_0: "
    at ucar.nc2.dods.DODSNetcdfFile.readData(DODSNetcdfFile.java:1296)
    at ucar.nc2.Variable._read(Variable.java:927)
    at ucar.nc2.Variable.read(Variable.java:791)
    at ucar.nc2.Variable._read(Variable.java:914)
    at ucar.nc2.Variable.read(Variable.java:791)
    at ucar.nc2.dataset.VariableDS._read(VariableDS.java:469)
    at ucar.nc2.Variable.read(Variable.java:791)

Note: This happens both inside of Matlab and in just plain Java. 

Am I accessing the JGOFS data structure correctly? Any help is appreciated!!

Original comment by bschlin...@gmail.com on 24 Nov 2009 at 10:17

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
Tried the same thing in a Groovy Shell and got the exact same error; so it's 
not Matlab specific. Looks like a 
bug 
in the OpenDap java libraries. To run this in Groovy (I'm assuming you've 
installed Groovy from 
http://groovy.codehaus.org/):
1) Download all the unbundled netcdf library from 
https://www.unidata.ucar.edu/software/netcdf-java/.
2) Extract the zip file and copy ALL the jars to ~/.groovy/lib
3) Run 'groovysh'
4) In groovy shell, run the following code:
url = 'http://usjgofs.whoi.edu/dods-bin/nph-
dods/jgofs/merged_objects/US_JGOFS/Equatorial_Pacific/bottle_eqpac'
netcdf = ucar.nc2.dataset.NetcdfDataset.openDataset(url); 
vars = netcdf.getVariables()
v = vars.get(0)
foram = v.findVariable('foram')
foram.read()

Original comment by bschlin...@gmail.com on 24 Nov 2009 at 10:25

GoogleCodeExporter commented 8 years ago 
From John Caron at Unidata:

This server is an older one:

DODS server core software: DODS DAP++/3.4.8
Server version: jg-dods/3.4.3

It doesnt appear to conform to the opendap spec, since it wont accept ?Level_0 
query, although its possible that the netcdf java library isnt doing the right 
thing with Sequences. Heres the HTTP Live Header dump:

http://usjgofs.whoi.edu/dods-bin/nph-dods/jgofs/merged_objects/US_JGOFS/Equatori
al_Pacific/bottle_eqpac?Level_0

GET 
/dods-bin/nph-dods/jgofs/merged_objects/US_JGOFS/Equatorial_Pacific/bottle_eqpac
?Level_0 HTTP/1.1
Host: usjgofs.whoi.edu
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1.5) 
Gecko/20091102 Firefox/3.5.5 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
If-Modified-Since: Tue Nov 24 22:42:17 2009 GMT

HTTP/1.x 200 OK
XDODS-Server: /DAP/3.8.0
Date: Tue Nov 24 22:42:28 2009 GMT
Last-Modified: Tue Nov 24 22:42:28 2009 GMT
Content-Type: text/plain
Content-Description: dods_error
Cache-Control: no-cache
----------------------------------------------------------

It also is returning everything as a String, which is pretty odd. Its likely 
that

   foram = v.findVariable('foram');
   data = foram.read();

isnt quite right, but i cant test until we see whats up with the server.

You should probably ask about this server at the opendap tech support and see 
what its status is.

Original comment by bschlin...@gmail.com on 3 Dec 2009 at 8:20

GoogleCodeExporter commented 8 years ago 
Looks like the issue is with JGOFS server. At this point, I'll tag this as 
'won't fix'

Original comment by bschlin...@gmail.com on 10 Feb 2010 at 5:50