Problem accessing data stored as Strings. (was Problem getting data from NARR 3D dataset)

[GoogleCodeExporter]

This is failing for me with the latest NCTOOLBOX

url='http://nomads.ncdc.noaa.gov/thredds/dodsC/aggs/narrA';
ds = cfdataset(url);

% Grab the variable of interest. No data is being read yet.
sz = ds.size('Sensible_heat_flux');

% Grab a subset of the data. Data is now being pulled across the network
t = ds.struct('Sensible_heat_flux', [sz(1) 1 1], [sz(1) sz(2) sz(3)]);

Original issue reported on code.google.com by rsignell on 5 Aug 2010 at 12:28

[GoogleCodeExporter]

The exception that's being tossed is:

== START EXCEPTION ==
??? Java exception occurred:
java.lang.NullPointerException
    at ucar.ma2.Array.reflectArrayCopyOut(Array.java:265)
    at ucar.ma2.Array.reflectArrayCopyOut(Array.java:270)
    at ucar.ma2.Array.copyToNDJavaArray(Array.java:761)

Error in ==> ncdataset>ncdataset.data at 174
                d = array.copyToNDJavaArray();

Error in ==> ncvariable>ncvariable.somedata at 313
                data.(name) = obj.dataset.data(name, vFirst, vLast, vStride);

Error in ==> ncvariable>ncvariable.data at 140
                d = somedata(obj, 1, first, last, stride);

Error in ==> ncvariable>ncvariable.subsref at 202
                    sref = builtin('subsref',obj,s);

Error in ==> cfdataset>cfdataset.struct at 195
                s = v.data(first, last, stride);
== END EXCEPTION

The data is being read successfully into a ucar.ma2.Array object. However, the 
call to 'copyToNDJavaArray', which converts it to an array that Matlab can 
handle, is bombing. Looking a variable,v , dimensions I get:

K>> v.axes

ans = 

    'time'
    'y'
    'x'

When I take a look at the CDL for the dataset I notice that the time is being 
stored as a String rather than a number:

String time(time=173304);
     :_CoordinateAxisType = "Time";
     :long_name = "time coordinate";
     :standard_name = "time";

I suspect that's the reason for the crash. The copyToNDJavaArray doesn't seem 
to like strings ( I tested this, yes it doesn't like strings) I'd have to dive 
into the NetCDF Java code to see what's going on. I won't be able to dive into 
that until about 8/18 or so. 

So that IDs a hole in ncdataset API, handling String data. Here's a hack to 
fetch a time value from that file:

>> v = ds.netcdf.findVariable('time')
>> r = ucar.ma2.Range(sz(1) - 1, sz(1) - 1, 1)
>> jl = java.util.ArrayList()
>> jl.add(r)
>> a = v.read(jl)
>> timeString = a.get(0)

At soon as I can get to it. I'll add a more graceful workaround for string data.

Original comment by bschlin...@gmail.com on 5 Aug 2010 at 3:44

• Changed title: Problem accessing data stored as Strings. (was Problem getting data from NARR 3D dataset)
• Added labels: Usability

[GoogleCodeExporter]

From rsignell:

Brian,
So you aren't using "getTimeDates"?
This would handle it for you.

http://www.unidata.ucar.edu/software/netcdf-java/v4.0/javadocAll/ucar/nc2/datase
t/CoordinateAxis1DTime.html

Original comment by bschlin...@gmail.com on 5 Aug 2010 at 4:03

[GoogleCodeExporter]

Rich thanks for the tip.

Here's a note for converting v (a ucar.nc2.dataset.CoordinateAxis1D) to a 
ucar.nc2.dataset.CoordinateAxis1DTime.

>> v = ds.netcdf.findVariable('time')
>> v2 = ucar.nc2.dataset.CoordinateAxis1DTime.factory(ds.netcdf, v, 
java.util.Formatter())
>> t = v2.getTimeDate(0)

This will work, especially when using ncdataset.time. For other calls though, 
you have to know apriori if the variable is a time variable. I can have the 
code make 'educated' quesses but I suspect those will be the wrong quesses 
under certain conditions. Also, getTimeDate uses udunits to conver to time. 
We've already identified cases where the udunits packages fails to parse time 
units; so the getTimeDate call will introduce a number of headaches.

Original comment by bschlin...@gmail.com on 5 Aug 2010 at 4:09

[GoogleCodeExporter]

So are you not using "getCoordinateSystem"?
You don't have to guess whether the variable is time or not -- NJ figures it 
out.

GridDataset = GridDataset.open(uri);

% get the grid associated with the variable name "temp" (temperature)
Grid = GridDataset.findGridByName(var);

% get the coordinate system for this grid:
GridCoordSys = Grid.getCoordinateSystem();

%get coordinate axes
latj=GridCoordSys.getYHorizAxis.read();
lonj=GridCoordSys.getXHorizAxis.read();
zj = GridCoordSys.getVerticalTransform.getCoordinateArray(iTime); % vertical 
transform 
% read times from file as gregorian dates
timeDate = GridCoordSys.getTimeDates();

Original comment by rsignell on 5 Aug 2010 at 4:32

[GoogleCodeExporter]

I haven't used GridDataset; it looks like we don't use formats here at MBARI 
that would require it, so there hasn't been a need. It looks very cool though. 
I'll have to play with it a bit to see how it would fit into nctoolbox. At 
first glance it strikes me that we might need to create another dataset type 
(ncdataset, cfdataset, griddataset?). 

Thanks for the pointer! 

BTW, You know far too much to be considered a 'normal' user. ;-) , maybe I 
should get you to write this part? 

Original comment by bschlin...@gmail.com on 5 Aug 2010 at 4:42

[GoogleCodeExporter]

Hello, I do not know if this is the same issue as above, but I can't retrieve 
strings variables from my netcdf files either. I include a trivial example as 
an attachment.

Basically, I have a variable 
string some_strings(station);

The content of which is
some_strings[0] = 'string_one'
some_strings[1] = 'string_two'

in matlab, I get
>> nc = ncdataset('string_test.nc')
>> var = nc.netcdf.findVariable('some_strings')
>> var.read()
??? Java exception occurred:
java.lang.IllegalStateException
    at ucar.nc2.iosp.IospHelper.readData(IospHelper.java:390)
    at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:287)
    at ucar.nc2.iosp.hdf5.H5iosp.readData(H5iosp.java:143)
    at ucar.nc2.iosp.hdf5.H5iosp.readData(H5iosp.java:121)
    at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1765)
    at ucar.nc2.Variable.reallyRead(Variable.java:812)
    at ucar.nc2.Variable._read(Variable.java:784)
    at ucar.nc2.Variable.read(Variable.java:662)
    at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:512)
    at ucar.nc2.dataset.VariableDS._read(VariableDS.java:496)
    at ucar.nc2.Variable.read(Variable.java:662)

Tested with nctoolbox 20110818-alpha3, Java 1.6.0_04-b12
I can't find any way to access those strings, including the hack posted in 
comment #1, any ideas?

Original comment by bstde...@gmail.com on 23 Aug 2011 at 6:26

Attachments:

• string_test.nc

[GoogleCodeExporter]

I'm posting a short groovy script for attempts to read Strings from the netcdf 
file using the straight Netcdf-Java API. All attempts fail. Might be a bug in 
the latest netcdf stuff. I'll look into it more next week.

Original comment by bschlin...@gmail.com on 25 Aug 2011 at 8:49

Attachments:

• stringtest.groovy