Clarify logicalSection in docs / Use logicalSection to hyperslice subsets?

[drf5n]

In the documentation at https://www.unidata.ucar.edu/software/thredds/v4.5/netcdf-java/ncml/AnnotatedSchema4.html the "logical view Elements" section seems to indicate that the logicalSection, logicalSlice, and logicalReduce elements enable subsets of the data:

These allow a variable to be a logical view of the original variable. Only one of the logical views can be used per variable.

 <!-- logical view: use only a section of original  -->
....

There is an example of using the logicalReduce element, and the logicalSlice seems straightforward ( you fill in the dimName and index attributes and get a single slice.) However, the logicalSection documentation seems unclear. Could someone show an example of using it?

Maybe something applied to the logicalReduce example's dataset set like:

<variable name="temperature_day1" orgName="temperature">
  <logicalSection value="(1:24,:,:,:)" />

Is this what logicalSection is for? Is this a way to provide view of a hyperslice subset in NCML?

Also, https://www.unidata.ucar.edu/software/thredds/v4.5/netcdf-java/ncml/Cookbook.html has a logicalReduce example, but no logicalSlice or logicalSection example.

[cofinoa]

@drf5n in the thredds mail list there is an example: http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2016/msg00251.html

in that example I'm changing the range of lonfitude coordinate from 0 360 range to -180 180 and, of course, the corresponding data.

The example creates 2 slices (logicalSection) from the same dataset and then a tiled aggregationof them:

<?xml version='1.0' encoding='UTF-8'?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
   <variable name='lon' type='double' shape='lon' >
    <attribute name='standard_name' value='longitude' />
    <attribute name='long_name' value='longitude' />
    <attribute name='units' value='degrees_east' />
    <attribute name='axis' value='X' />
    <attribute name='bounds' value='lon_bnds' />
    <attribute name='original_units' value='degrees_east' />
    <values>
      -180.0, -176.25, -172.5, -168.75, -165.0, -161.25, -157.5, -153.75, -150.0, -146.25, -142.5, -138.75, -135.0, -131.25, -127.5, -123.75, -120.0, -116.25, -112.5, -108.75, -105.0, -101.25, -97.5, -93.75, -90.0, -86.25, -82.5, -78.75, -75.0, -71.25, -67.5, -63.75, -60.0, -56.25, -52.5, -48.75, -45.0, -41.25, -37.5, -33.75, -30.0, -26.25, -22.5, -18.75, -15.0, -11.25, -7.5, -3.75, 0.0, 3.75, 7.5, 11.25, 15.0, 18.75, 22.5, 26.25, 30.0, 33.75, 37.5, 41.25, 45.0, 48.75, 52.5, 56.25, 60.0, 63.75, 67.5, 71.25, 75.0, 78.75, 82.5, 86.25, 90.0, 93.75, 97.5, 101.25, 105.0, 108.75, 112.5, 116.25, 120.0, 123.75, 127.5, 131.25, 135.0, 138.75, 142.5, 146.25, 150.0, 153.75, 157.5, 161.25, 165.0, 168.75, 172.5, 176.25
    </values>
  </variable>
  <variable name='lon_bnds' type='double' shape='lon bnds'>
    <values>
      -181.875 -178.125 -178.125 -174.375 -174.375 -170.625 -170.625 -166.875 -166.875 -163.125 -163.125 -159.375 -159.375 -155.625 -155.625 -151.875 -151.875 -148.125 -148.125 -144.375 -144.375 -140.625 -140.625 -136.875 -136.875 -133.125 -133.125 -129.375 -129.375 -125.625 -125.625 -121.875 -121.875 -118.125 -118.125 -114.375 -114.375 -110.625 -110.625 -106.875 -106.875 -103.125 -103.125 -99.375 -99.375 -95.625 -95.625 -91.875 -91.875 -88.125 -88.125 -84.375 -84.375 -80.625 -80.625 -76.875 -76.875 -73.125 -73.125 -69.375 -69.375 -65.625 -65.625 -61.875 -61.875 -58.125 -58.125 -54.375 -54.375 -50.625 -50.625 -46.875 -46.875 -43.125 -43.125 -39.375 -39.375 -35.625 -35.625 -31.875 -31.875 -28.125 -28.125 -24.375 -24.375 -20.625 -20.625 -16.875 -16.875 -13.125 -13.125 -9.375 -9.375 -5.625 -5.625 -1.875 -1.875 1.875 1.875 5.625 5.625 9.375 9.375 13.125 13.125 16.875 16.875 20.625 20.625 24.375 24.375 28.125 28.125 31.875 31.875 35.625 35.625 39.375 39.375 43.125 43.125 46.875 46.875 50.625 50.625 54.375 54.375 58.125 58.125 61.875 61.875 65.625 65.625 69.375 69.375 73.125 73.125 76.875 76.875 80.625 80.625 84.375 84.375 88.125 88.125 91.875 91.875 95.625 95.625 99.375 99.375 103.125 103.125 106.875 106.875 110.625 110.625 114.375 114.375 118.125 118.125 121.875 121.875 125.625 125.625 129.375 129.375 133.125 133.125 136.875 136.875 140.625 140.625 144.375 144.375 148.125 148.125 151.875 151.875 155.625 155.625 159.375 159.375 163.125 163.125 166.875 166.875 170.625 170.625 174.375 174.375 178.125 
    </values>
  </variable>

  <aggregation type="tiled" dimName="lon">
    <netcdf section="(0:47)">
      <dimension name="lon" length="48"/>
      <variable name="tasmax" shape="time lat lon"/>
      <remove name="lon" type="variable"/>
      <remove name="lon_bnds" type="variable"/>
      <aggregation type="union">
        <netcdf location="dods://www-lscedods.cea.fr/cgi-bin/nph-dods/pmip2_dbext/pmip2_21k_oa/atm/mo/tasmax/tasmax_A_MO_pmip2_21k_oa_IPSL-CM4-V1-MR_0200-0299.nc">
          <variable name="tasmax">
            <logicalSection section="(:,:,48:95)"/>
          </variable>
        </netcdf>
      </aggregation>
    </netcdf>
    <netcdf section="(48:95)">
      <dimension name="lon" length="48"/>
      <variable name="tasmax" shape="time lat lon"/>
      <remove name="lon" type="variable"/>
      <remove name="lon_bnds" type="variable"/>
      <aggregation type="union">
        <netcdf location="dods://www-lscedods.cea.fr/cgi-bin/nph-dods/pmip2_dbext/pmip2_21k_oa/atm/mo/tasmax/tasmax_A_MO_pmip2_21k_oa_IPSL-CM4-V1-MR_0200-0299.nc">
          <variable name="tasmax">
            <logicalSection section="(:,:,0:47)"/>
          </variable>
        </netcdf>
      </aggregation>
    </netcdf>
  </aggregation>
</netcdf>

Antonio

[drf5n]

Cool. Perhaps I can use this scheme to construct an FMRC-like aggregation of UGRID slabs by tiling them along time.

I see mention of type="tiled" in https://www.unidata.ucar.edu/software/thredds/v4.5/netcdf-java/ncml/AnnotatedSchema4.html but no indication of "tiled" or "forecastModelRunSingleCollection" on https://www.unidata.ucar.edu/software/thredds/v4.5/netcdf-java/ncml/Aggregation.html

It seems to work with the "joinExisting" aggregation:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation type="joinExisting" dimName="time">
    <netcdf  location="/ches/data10/pub/drf/SCHISM/NETCDF/elev_20170204.nc">
      <variable name="time">
        <logicalSection section="0:71"/>
      </variable>
    </netcdf>
    <netcdf  location="/ches/data10/pub/drf/SCHISM/NETCDF/elev_20170205.nc">
      <variable name="time">
        <logicalSection section="0:71"/>
      </variable>
    </netcdf>
  </aggregation>
</netcdf>

and

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation type="joinExisting" dimName="time">
    <scan  location="/ches/data10/pub/drf/SCHISM/NETCDF/" regExp="elev_.*\.nc$">
      <variable name="time">
        <logicalSection section="0:71"/>
      </variable>
    </scan>
  </aggregation>
</netcdf>

Looks like I need to clean up my time:units to translate them all onto the same origin with timeUnitsChange="true":

<aggregation type="joinExisting" dimName="time" timeUnitsChange="true">

This is great.

[cofinoa]

yes, if your aggregation dimension is the outer one, the joinExisting is a better option. The tiled one is the general case, and is the only one you can use if you want to aggregate on none-outer dimesions.

[cwardgar]

@drf5n, it looks like you've got something workable. Thanks for the help, @cofinoa. I'm gonna go ahead and close this. I've created separate issues for the missing documentation that you identified: #784 and #785.

Thanks for the heads-up!

[drf5n]

Thanks. There's some amazing stuff in THREDDS & NetCDF-Java.

[drf5n]

I spoke too early.

I haven't quite found a solution. <aggregation type="joinExisting ...> seems to only deliver one of the explicit <netcdf> sets with <logicalSection ...> , and with the `<scan ...> element it appears to ignore the subsetting completely.

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation type="joinExisting" dimName="time" >
    <netcdf location="http://web.vims.edu/%7Edrf/NETCDF/elev_20170319.nc">
      <variable name="elev">
        <logicalSection section="0:23,:"/>
      </variable>
      <variable name="time">
        <logicalSection section="0:23"/>
      </variable>
    </netcdf>
    <netcdf location="http://web.vims.edu/%7Edrf/NETCDF/elev_20170320.nc">
      <variable name="elev">
        <logicalSection section="24:47,:"/>
      </variable>
      <variable name="time">
        <logicalSection section="24:47"/>
      </variable>
    </netcdf>
  </aggregation>
</netcdf>

... delivers one 24-length section.

and

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation type="joinExisting" dimName="time">
     <scan location="/ches/data10/pub/drf/SCHISM/NETCDF/" regExp="elev_.*\.nc$" >
      <variable name="elev">
        <logicalSection section="0:23,:"/>
      </variable>
      <variable name="time">
        <logicalSection section="0:23"/>
      </variable>
    </scan>
  </aggregation>
</netcdf>

aggregates the full, unsliced sets together.

[cwardgar]

Thanks for the report. I've created another, separate issue: #787.