search

dash-project / dash

DASH, the C++ Template Library for Distributed Data Structures with Support for Hierarchical Locality for HPC and Data-Driven Science
http://www.dash-project.org/
Other
154 stars 44 forks source link

StoreHDF::write for 1D compile error #692

Open Goon83 opened 4 years ago

Goon83 commented 4 years ago

Hi, I am testing the HDF5 support in DASH. It reports compile error for 1D data at compile time. Could you help to look at it ? See below for the code and error output.

Bests, Bin

Code:

#include <libdash.h>
#include <iostream>

using dash::io::hdf5::hdf5_options;
using dash::io::hdf5::StoreHDF;

int main(int argc, char *argv[])
{
    dash::init(&argc, &argv);

    dash::Matrix<int, 1> *h5matrix = new dash::Matrix<int, 1>(dash::SizeSpec<1>(8));
    StoreHDF::write(*h5matrix, "testf.h5", "testg/testd");

    //2D works
    //dash::Matrix<int, 2> *h5matrix = new dash::Matrix<int, 2>(dash::SizeSpec<2>(8, 8));
    //StoreHDF::write(*h5matrix, "testf.h5", "testg/testd");

    dash::finalize();

    return EXIT_SUCCESS;
}

Error info:

In file included from h5-test.cpp:8:
In file included from /Users/dbin/opt/dash-0.4.0/include/libdash.h:71:
In file included from /Users/dbin/opt/dash-0.4.0/include/dash/io/HDF5.h:4:
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:477:39: error:
      no member named 'local_blockspec' in 'dash::TilePattern<1,
      dash::ROW_MAJOR, long>'
    const auto & lblockspec = pattern.local_blockspec();
                              ~~~~~~~ ^
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:582:12: note:
      in instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::_get_hdf_slab_body<dash::TilePattern<1,
      dash::ROW_MAJOR, long> >' requested here
        1, _get_hdf_slab_body(pattern, {}));
           ^
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/internal/DriverImplZeroCopy.h:32:21: note:
      in instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::_get_hdf_slabs<dash::TilePattern<1,
      dash::ROW_MAJOR, long> >' requested here
  auto hyperslabs = _get_hdf_slabs(container.pattern());
                    ^
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:733:5: note: in
      instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::_process_dataset_impl_zero_copy<dash::Matrix<int,
      1, long, dash::TilePattern<1, dash::ROW_MAJOR, long>, dash::HostSpace> >'
      requested here
    _process_dataset_impl_zero_copy(StoreHDF::Mode::WRITE, container, h5dset,
    ^
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:234:5: note: in
      instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::_write_dataset_impl<dash::Matrix<int, 1, long,
      dash::TilePattern<1, dash::ROW_MAJOR, long>, dash::HostSpace> >' requested
      here
    _write_dataset_impl(array, h5dset, internal_type);
    ^
h5-test.cpp:26:12: note: in instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::write<dash::Matrix<int, 1, long,
      dash::TilePattern<1, dash::ROW_MAJOR, long>, dash::HostSpace> >' requested
      here
        StoreHDF::write(*h5matrix, "testf.h5", "testg/testd");
                  ^
1 error generated.
#include <libdash.h>
#include <iostream>

using dash::io::hdf5::hdf5_options;
using dash::io::hdf5::StoreHDF;

int main(int argc, char *argv[])
{
    dash::init(&argc, &argv);

    dash::Matrix<int, 1> *h5matrix = new dash::Matrix<int, 1>(dash::SizeSpec<1>(8));
    StoreHDF::write(*h5matrix, "testf.h5", "testg/testd");

    //2D works
    //dash::Matrix<int, 2> *h5matrix = new dash::Matrix<int, 2>(dash::SizeSpec<2>(8, 8));
    //StoreHDF::write(*h5matrix, "testf.h5", "testg/testd");

    dash::finalize();

    return EXIT_SUCCESS;
}
./h5-test.compile
In file included from h5-test.cpp:8:
In file included from /Users/dbin/opt/dash-0.4.0/include/libdash.h:71:
In file included from /Users/dbin/opt/dash-0.4.0/include/dash/io/HDF5.h:4:
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:477:39: error:
      no member named 'local_blockspec' in 'dash::TilePattern<1,
      dash::ROW_MAJOR, long>'
    const auto & lblockspec = pattern.local_blockspec();
                              ~~~~~~~ ^
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:582:12: note:
      in instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::_get_hdf_slab_body<dash::TilePattern<1,
      dash::ROW_MAJOR, long> >' requested here
        1, _get_hdf_slab_body(pattern, {}));
           ^
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/internal/DriverImplZeroCopy.h:32:21: note:
      in instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::_get_hdf_slabs<dash::TilePattern<1,
      dash::ROW_MAJOR, long> >' requested here
  auto hyperslabs = _get_hdf_slabs(container.pattern());
                    ^
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:733:5: note: in
      instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::_process_dataset_impl_zero_copy<dash::Matrix<int,
      1, long, dash::TilePattern<1, dash::ROW_MAJOR, long>, dash::HostSpace> >'
      requested here
    _process_dataset_impl_zero_copy(StoreHDF::Mode::WRITE, container, h5dset,
    ^
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:234:5: note: in
      instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::_write_dataset_impl<dash::Matrix<int, 1, long,
      dash::TilePattern<1, dash::ROW_MAJOR, long>, dash::HostSpace> >' requested
      here
    _write_dataset_impl(array, h5dset, internal_type);
    ^
h5-test.cpp:26:12: note: in instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::write<dash::Matrix<int, 1, long,
      dash::TilePattern<1, dash::ROW_MAJOR, long>, dash::HostSpace> >' requested
      here
        StoreHDF::write(*h5matrix, "testf.h5", "testg/testd");
                  ^
1 error generated.
devreal commented 4 years ago

Looking into this right now. It seems that the 1D specialization of the TilePattern does not offer local_blockspec. Will try to come up with a patch soon.

devreal commented 4 years ago

@Goon83 Can you give the patch at #693 a try?

The 1D specialization of the TilePattern was missing the local_blockspec member function. I should note that the CSRPattern and the LoadBalancingPattern are missing that method, too, but I don't know enough about them to fix it immediately. I hope this helps though.

Goon83 commented 4 years ago

Hi devreal, Thanks for working to resolve this issue. It compiles now without any error.

Bests, Bin

Goon83 commented 4 years ago

hi @devreal Another error comes up for the "StoreHDF::read" at compile time. It says that

"TeamSpec<[...], (default) long>' to 'const TeamSpec<[...], unsigned long>' for 3rd argument".

Looks wired that the "StoreHDF::read" works.

It might be the same as the previous error https://github.com/dash-project/dash/issues/688

BTW, my system is MacOS 10.14.6.

Bests, Bin

Test code:

#include <libdash.h>
#include <iostream>

using dash::io::hdf5::hdf5_options;
using dash::io::hdf5::StoreHDF;

int main(int argc, char *argv[])
{
    dash::init(&argc, &argv);

    dash::Matrix<double, 1, unsigned long> *h5matrix = new dash::Matrix<double, 1, unsigned long>(dash::SizeSpec<1>(8));

         //This line works
         StoreHDF::write(*h5matrix, "testf.h5", "testg/testd1D");
    //This line fails
         StoreHDF::read(*h5matrix, "testf.h5", "testg/testd1D");

    dash::finalize();

    return EXIT_SUCCESS;
}

Error at compile time:

In file included from h5-test.cpp:8:
In file included from /Users/dbin/opt/dash-0.4.0/include/libdash.h:71:
In file included from /Users/dbin/opt/dash-0.4.0/include/dash/io/HDF5.h:4:
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:364:23: error: no matching constructor
      for initialization of 'const pattern_t' (aka 'const dash::TilePattern<1, dash::ROW_MAJOR,
      unsigned long>')
      const pattern_t pattern(dash::SizeSpec<ndim>(size_extents),
                      ^       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
h5-test.cpp:20:12: note: in instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::read<dash::Matrix<double, 1, unsigned long, dash::TilePattern<1,
      dash::ROW_MAJOR, unsigned long>, dash::HostSpace> >' requested here
        StoreHDF::read(*h5matrix, "testf.h5", "testg/testd1D");
                  ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:200:3: note: candidate constructor not
      viable: no known conversion from 'TeamSpec<[...], (default) long>' to 'const TeamSpec<[...],
      unsigned long>' for 3rd argument
  TilePattern(
  ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:159:3: note: candidate constructor
      template not viable: no known conversion from 'dash::SizeSpec<ndim>' to 'dash::TilePattern<1,
      dash::ROW_MAJOR, unsigned long>::SizeType' (aka 'unsigned long') for 1st argument
  TilePattern(
  ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:263:3: note: candidate constructor not
      viable: requires at most 3 arguments, but 4 were provided
  TilePattern(
  ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:292:13: note: candidate constructor
      not viable: requires single argument 'other', but 4 arguments were provided
  constexpr TilePattern(const self_t & other) = default;
            ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:300:13: note: candidate constructor
      not viable: requires single argument 'other', but 4 arguments were provided
  constexpr TilePattern(self_t & other)
            ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:1042:3: note: candidate constructor
      not viable: requires single argument 'arguments', but 4 arguments were provided
  TilePattern(const PatternArguments_t & arguments)
  ^
In file included from h5-test.cpp:8:
In file included from /Users/dbin/opt/dash-0.4.0/include/libdash.h:71:
In file included from /Users/dbin/opt/dash-0.4.0/include/dash/io/HDF5.h:4:
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:701:21: error: no matching constructor
      for initialization of 'const pattern_t' (aka 'const dash::TilePattern<1, dash::ROW_MAJOR,
      unsigned long>')
    const pattern_t pattern(dash::SizeSpec<ndim>(size_extents),
                    ^       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/dbin/opt/dash-0.4.0/include/dash/io/hdf5/StorageDriver.h:353:7: note: in instantiation of
      function template specialization
      'dash::io::hdf5::StoreHDF::_restore_pattern<dash::Matrix<double, 1, unsigned long,
      dash::TilePattern<1, dash::ROW_MAJOR, unsigned long>, dash::HostSpace> >' requested here
      _restore_pattern(matrix, h5dset, foptions);
      ^
h5-test.cpp:20:12: note: in instantiation of function template specialization
      'dash::io::hdf5::StoreHDF::read<dash::Matrix<double, 1, unsigned long, dash::TilePattern<1,
      dash::ROW_MAJOR, unsigned long>, dash::HostSpace> >' requested here
        StoreHDF::read(*h5matrix, "testf.h5", "testg/testd1D");
                  ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:200:3: note: candidate constructor not
      viable: no known conversion from 'TeamSpec<[...], (default) long>' to 'const TeamSpec<[...],
      unsigned long>' for 3rd argument
  TilePattern(
  ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:159:3: note: candidate constructor
      template not viable: no known conversion from 'dash::SizeSpec<ndim>' to 'dash::TilePattern<1,
      dash::ROW_MAJOR, unsigned long>::SizeType' (aka 'unsigned long') for 1st argument
  TilePattern(
  ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:263:3: note: candidate constructor not
      viable: requires at most 3 arguments, but 4 were provided
  TilePattern(
  ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:292:13: note: candidate constructor
      not viable: requires single argument 'other', but 4 arguments were provided
  constexpr TilePattern(const self_t & other) = default;
            ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:300:13: note: candidate constructor
      not viable: requires single argument 'other', but 4 arguments were provided
  constexpr TilePattern(self_t & other)
            ^
/Users/dbin/opt/dash-0.4.0/include/dash/pattern/TilePattern1D.h:1042:3: note: candidate constructor
      not viable: requires single argument 'arguments', but 4 arguments were provided
  TilePattern(const PatternArguments_t & arguments)
  ^
2 errors generated.
devreal commented 4 years ago

@Goon83 I think there is a problem with the explicit specification of the index type in your Matrix instance. This is certainly a bug in how we pass around this type to dependent templates. As a workaround, can you try removing the unsigned long from dash::Matrix<double, 1, unsigned long>?

@anindex can you take a look at this? We should include the cases brought up here in the DASH test suite (and extend it some more).

anindex commented 4 years ago

Hi @devreal,

Sure I will look into it tomorrow :)