NcML logical view elements within aggregations

[cwardgar]

From https://github.com/Unidata/thredds/issues/778#issuecomment-288433894

The logical view elements don't behave as expected when they're used within aggregations. For example:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation type="joinExisting" dimName="time" >
    <netcdf location="http://web.vims.edu/%7Edrf/NETCDF/elev_20170319.nc">
      <variable name="elev">
        <logicalSection section="0:23,:"/>
      </variable>
      <variable name="time">
        <logicalSection section="0:23"/>
      </variable>
    </netcdf>
    <netcdf location="http://web.vims.edu/%7Edrf/NETCDF/elev_20170320.nc">
      <variable name="elev">
        <logicalSection section="24:47,:"/>
      </variable>
      <variable name="time">
        <logicalSection section="24:47"/>
      </variable>
    </netcdf>
  </aggregation>
</netcdf>

delivers one 24-length section. And

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation type="joinExisting" dimName="time">
     <scan location="/ches/data10/pub/drf/SCHISM/NETCDF/" regExp="elev_.*\.nc$" >
      <variable name="elev">
        <logicalSection section="0:23,:"/>
      </variable>
      <variable name="time">
        <logicalSection section="0:23"/>
      </variable>
    </scan>
  </aggregation>
</netcdf>

aggregates the full, unsliced sets together.

Given that TestLogicalReduce is the only test we have for the logical view elements, it's clear that this combination wasn't considered at all.

[drf5n]

I think a part of it might be the 'anonymous dimensions' indicated in the comment here and shown in this ToolsUI ncdump. and CoordSys screenshot.

Note that the dump shows no named 'dimensions' with size 24, and the time and elev variables have dimensions of simple integers with no names. This probably interferes with interpretation of what <aggregation type="joinExisting" dimName="time"> would mean. I fiddled a bit with dimName="" and dimName="anon" with joinExisting and joinNew got errors.

netcdf dods://tds.vims.edu:8080/thredds/dodsC/SCHISM/NETCDFfix/section_test.ncml {
  dimensions:
    nele = 2437197;
    nface = 3;
    node = 1243146;
    sigma = 2;
    one = 1;
    time = 144;
  variables:
    int ele(nele=2437197, nface=3);
      :long_name = "Horizontal Triangular Element Table";
      :units = "non-dimensional";
      :cf_role = "face_node_connectivity";
      :start_index = 1; // int

    float x(node=1243146);
      :long_name = "x-coordinates";
      :units = "meters";
      :mesh = "mesh";
      :location = "node";

    float y(node=1243146);
      :long_name = "y-coordinates";
      :units = "meters";
      :mesh = "mesh";
      :location = "node";

    float depth(node=1243146);
      :long_name = "Bathymetry";
      :units = "meters";
      :positive = "down";
      :mesh = "mesh";
      :location = "node";

    float sigma(sigma=2);
      :long_name = "S coordinates at whole levels";
      :units = "1";
      :standard_name = "ocean_s_coordinate";
      :positive = "up";
      :h_s = 1000000.0f; // float
      :h_c = 30.0f; // float
      :theta_b = 0.0f; // float
      :theta_f = 1.0E-4f; // float
      :formula_terms = "s: sigma eta: elev depth: depth a: sigma_theta_f b: sigma_theta_b depth_c: sigma_h_c";

    float sigma_h_c(one=1);
      :long_name = "ocean_s_coordinate h_c constant";
      :units = "meters";
      :positive = "down";

    float sigma_theta_b(one=1);
      :long_name = "ocean_s_coordinate theta_b constant";

    float sigma_theta_f(one=1);
      :long_name = "ocean_s_coordinate theta_f constant";

    float sigma_maxdepth(one=1);
      :long_name = "ocean_s_coordinate maximum depth cutoff (mixed s over z boundary";
      :units = "meters";
      :positive = "down";

    float Cs(sigma=2);
      :long_name = "Function C(s) at whole levels";
      :units = "non-dimensional";
      :positive = "up";

    int mesh;
      :cf_role = "mesh_topology";
      :long_name = "Topology data of 2d unstructured mesh";
      :topology_dimension = 2; // int
      :node_coordinates = "x y";
      :face_node_connectivity = "ele";

    float elev(24, node=1243146);
      :long_name = "elev.61";
      :mesh = "mesh";
      :location = "node";

    float time(25);
      :long_name = "Time";
      :units = "seconds since 2017-02-04 00:00:00 +00";
      :base_date = "02/04/2017 00:00:00 UTC                         ";
      :standard_name = "time";

  // global attributes:
  :Conventions = "CF-1.0";
  :history = "Mon Mar 20 01:15:49 2017: ncatted -O -a units,time,o,c,\'seconds since 2017-02-04  00:00:00 +00\' NETCDF/elev_20170204.nc\nSat Feb  4 10:08:00 2017: ncrcat -O ARCHIVE/20170204/outputs/1_elev.nc ARCHIVE/20170204/outputs/2_elev.nc ARCHIVE/20170204/outputs/3_elev.nc NETCDF/elev_20170204.nc";
  :nco_openmp_thread_number = 1; // int
  :Yoyo = "ma";
}

[cofinoa]

@drf5n,

In the second JoinExisting aggregation example, the inner elements to the scan element are completely ignored by the NcMLReader.

In the first on some manipulation is required. This is the working version:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
  <aggregation type="joinExisting" dimName="time" >
    <netcdf ncoords="24">
      <aggregation type="union">
        <dimension name="time" length="24" unlimited="false"/>
        <variable  name="time" shape="time"/>
        <variable  name="elev" shape="time node"/>
        <netcdf location="drf/SCHISM/NETCDF/elev_20170319.nc">
          <remove name="time" type="dimension"/>
          <variable name="elev">
            <logicalSection section="0:23,:"/>
          </variable>
          <variable name="time">
            <logicalSection section="0:23"/>
          </variable>
        </netcdf>
      </aggregation>
    </netcdf>
    <netcdf  ncoords="24">
      <aggregation type="union">
        <dimension name="time" length="24" unlimited="false"/>
        <variable  name="time" shape="time"/>
        <variable  name="elev" shape="time node"/>
        <netcdf location="drf/SCHISM/NETCDF/elev_20170320.nc">
          <remove name="time" type="dimension"/>
          <variable name="elev">
            <logicalSection section="24:47,:"/>
          </variable>
          <variable name="time">
            <logicalSection section="24:47"/>
          </variable>
        </netcdf>
      </aggregation>
    </netcdf>
  </aggregation>
</netcdf>

You need to "rename" the anonymous dimensions (both are different ones with same name), but in outer element to the logicalSection. The union aggregation is a dummy container to rename those dimensions.