search

Weiming-Hu / AnalogsEnsemble

The C++ and R packages for parallel ensemble forecasts using Analog Ensemble
https://weiming-hu.github.io/AnalogsEnsemble/
MIT License
18 stars 5 forks source link

Linking error when integrating PyTorch #110

Closed Weiming-Hu closed 3 years ago

Weiming-Hu commented 3 years ago

I have been working on the branch with integration of PyTorch libTorch. When I turn on the option ENABLE_AI, I have got the following error message.

[100%] Linking CXX executable anen_netcdf
cd /sapphire/s0/wuh20/github/AnalogsEnsemble/build/apps/anen_netcdf && /sapphire/s0/wuh20/packages/release/bin/cmake -E cmake_link_script CMakeFiles/anen_netcdf.dir/link.txt --verbose=1
/usr/bin/c++    -rdynamic CMakeFiles/anen_netcdf.dir/anen_netcdf.cpp.o  -o anen_netcdf  -Wl,-rpath,/home/graduate/wuh20/tmp/libtorch/lib: ../../CAnEnIO/libAnEnIO.a /usr/lib/x86_64-linux-gnu/libboost_program_options.so ../../CAnEn/libAnEn.a /home/graduate/wuh20/tmp/libtorch/lib/libtorch.so -Wl,--no-as-needed,"/home/graduate/wuh20/tmp/libtorch/lib/libtorch_cpu.so" -Wl,--as-needed /home/graduate/wuh20/tmp/libtorch/lib/libc10.so -lpthread -Wl,--no-as-needed,"/home/graduate/wuh20/tmp/libtorch/lib/libtorch.so" -Wl,--as-needed /home/graduate/wuh20/tmp/libtorch/lib/libc10.so /usr/lib/x86_64-linux-gnu/libboost_serialization.so /usr/lib/x86_64-linux-gnu/libboost_date_time.so -lgomp -lpthread /usr/lib/x86_64-linux-gnu/libboost_date_time.so -lnetcdf_c++4 -lnetcdf /usr/lib/x86_64-linux-gnu/libboost_filesystem.so /usr/lib/x86_64-linux-gnu/libboost_system.so -leccodes 
../../CAnEnIO/libAnEnIO.a(AnEnReadNcdf.cpp.o): In function `AnEnReadNcdf::readForecasts(std::string const&, Forecasts&, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<unsigned long, std::allocator<unsigned long> >) const':
AnEnReadNcdf.cpp:(.text+0xc0e): undefined reference to `netCDF::NcFile::NcFile(std::string const&, netCDF::NcFile::FileMode)'
AnEnReadNcdf.cpp:(.text+0x1187): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
AnEnReadNcdf.cpp:(.text+0x11fd): undefined reference to `netCDF::NcGroup::getGroup(std::string const&, netCDF::NcGroup::GroupLocation) const'
../../CAnEnIO/libAnEnIO.a(AnEnReadNcdf.cpp.o): In function `AnEnReadNcdf::readObservations(std::string const&, Observations&, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<unsigned long, std::allocator<unsigned long> >) const':
AnEnReadNcdf.cpp:(.text+0x21ac): undefined reference to `netCDF::NcFile::NcFile(std::string const&, netCDF::NcFile::FileMode)'
AnEnReadNcdf.cpp:(.text+0x2594): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
AnEnReadNcdf.cpp:(.text+0x260a): undefined reference to `netCDF::NcGroup::getGroup(std::string const&, netCDF::NcGroup::GroupLocation) const'
../../CAnEnIO/libAnEnIO.a(AnEnReadNcdf.cpp.o): In function `AnEnReadNcdf::readAnalogs(std::string const&, Array4D&, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<unsigned long, std::allocator<unsigned long> >)':
AnEnReadNcdf.cpp:(.text+0x2d70): undefined reference to `netCDF::NcFile::NcFile(std::string const&, netCDF::NcFile::FileMode)'
AnEnReadNcdf.cpp:(.text+0x2ebf): undefined reference to `netCDF::NcGroup::getDim(std::string const&, netCDF::NcGroup::Location) const'
AnEnReadNcdf.cpp:(.text+0x2f06): undefined reference to `netCDF::NcGroup::getDim(std::string const&, netCDF::NcGroup::Location) const'
AnEnReadNcdf.cpp:(.text+0x2f4d): undefined reference to `netCDF::NcGroup::getDim(std::string const&, netCDF::NcGroup::Location) const'
AnEnReadNcdf.cpp:(.text+0x2f94): undefined reference to `netCDF::NcGroup::getDim(std::string const&, netCDF::NcGroup::Location) const'
AnEnReadNcdf.cpp:(.text+0x30c2): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(AnEnReadNcdf.cpp.o): In function `AnEnReadNcdf::read(netCDF::NcGroup const&, Parameters&, unsigned long, unsigned long) const':
AnEnReadNcdf.cpp:(.text+0x3488): undefined reference to `netCDF::NcGroup::getDim(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(AnEnReadNcdf.cpp.o): In function `AnEnReadNcdf::read(netCDF::NcGroup const&, Stations&, unsigned long, unsigned long, std::string const&, std::string const&) const':
AnEnReadNcdf.cpp:(.text+0x3d03): undefined reference to `netCDF::NcGroup::getDim(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(AnEnReadNcdf.cpp.o): In function `AnEnReadNcdf::read(netCDF::NcGroup const&, Times&, std::string const&, unsigned long, unsigned long) const':
AnEnReadNcdf.cpp:(.text+0x4c96): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(AnEnReadNcdf.cpp.o): In function `void AnEnReadNcdf::read<double>(netCDF::NcGroup const&, double*, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<unsigned long, std::allocator<unsigned long> >) const':
AnEnReadNcdf.cpp:(.text._ZNK12AnEnReadNcdf4readIdEEvRKN6netCDF7NcGroupEPT_RKSsSt6vectorImSaImEESB_[_ZNK12AnEnReadNcdf4readIdEEvRKN6netCDF7NcGroupEPT_RKSsSt6vectorImSaImEESB_]+0x9a): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(AnEnReadNcdf.cpp.o): In function `void Ncdf::readVector<double>(netCDF::NcGroup const&, std::string, std::vector<double, std::allocator<double> >&, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<unsigned long, std::allocator<unsigned long> >)':
AnEnReadNcdf.cpp:(.text._ZN4Ncdf10readVectorIdEEvRKN6netCDF7NcGroupESsRSt6vectorIT_SaIS6_EES5_ImSaImEESB_[_ZN4Ncdf10readVectorIdEEvRKN6netCDF7NcGroupESsRSt6vectorIT_SaIS6_EES5_ImSaImEESB_]+0x98): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(AnEnReadNcdf.cpp.o): In function `void Ncdf::readVector<unsigned int>(netCDF::NcGroup const&, std::string, std::vector<unsigned int, std::allocator<unsigned int> >&, std::vector<unsigned long, std::allocator<unsigned long> >, std::vector<unsigned long, std::allocator<unsigned long> >)':
AnEnReadNcdf.cpp:(.text._ZN4Ncdf10readVectorIjEEvRKN6netCDF7NcGroupESsRSt6vectorIT_SaIS6_EES5_ImSaImEESB_[_ZN4Ncdf10readVectorIjEEvRKN6netCDF7NcGroupESsRSt6vectorIT_SaIS6_EES5_ImSaImEESB_]+0x98): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(Ncdf.cpp.o): In function `Ncdf::readDimLength(std::string const&, std::string)':
Ncdf.cpp:(.text+0x45): undefined reference to `netCDF::NcFile::NcFile(std::string const&, netCDF::NcFile::FileMode)'
Ncdf.cpp:(.text+0x139): undefined reference to `netCDF::NcGroup::getDim(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(Ncdf.cpp.o): In function `Ncdf::checkExists(std::string const&, bool, bool)':
Ncdf.cpp:(.text+0x46f): undefined reference to `netCDF::NcFile::NcFile(std::string const&, netCDF::NcFile::FileMode)'
../../CAnEnIO/libAnEnIO.a(Ncdf.cpp.o): In function `Ncdf::checkVarShape(netCDF::NcGroup const&, std::string const&, std::vector<std::string, std::allocator<std::string> > const&)':
Ncdf.cpp:(.text+0xf1b): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
Ncdf.cpp:(.text+0x1145): undefined reference to `netCDF::NcDim::getName() const'
Ncdf.cpp:(.text+0x1233): undefined reference to `netCDF::NcDim::getName() const'
../../CAnEnIO/libAnEnIO.a(Ncdf.cpp.o): In function `Ncdf::dimExists(netCDF::NcGroup const&, std::string const&)':
Ncdf.cpp:(.text+0x149e): undefined reference to `netCDF::NcGroup::getDim(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(Ncdf.cpp.o): In function `Ncdf::varExists(netCDF::NcGroup const&, std::string const&)':
Ncdf.cpp:(.text+0x1572): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(Ncdf.cpp.o): In function `Ncdf::readStringVector(netCDF::NcGroup const&, std::string, std::vector<std::string, std::allocator<std::string> >&, unsigned long, unsigned long, std::string const&)':
Ncdf.cpp:(.text+0x168e): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
Ncdf.cpp:(.text+0x192a): undefined reference to `netCDF::NcGroup::getDim(std::string const&, netCDF::NcGroup::Location) const'
../../CAnEnIO/libAnEnIO.a(Ncdf.cpp.o): In function `Ncdf::getDimension(netCDF::NcGroup&, std::string const&, bool, unsigned long)':
Ncdf.cpp:(.text+0x1fed): undefined reference to `netCDF::NcGroup::getDim(std::string const&, netCDF::NcGroup::Location) const'
Ncdf.cpp:(.text+0x2029): undefined reference to `netCDF::NcGroup::addDim(std::string const&) const'
Ncdf.cpp:(.text+0x2052): undefined reference to `netCDF::NcGroup::addDim(std::string const&, unsigned long) const'
../../CAnEnIO/libAnEnIO.a(Ncdf.cpp.o): In function `Ncdf::writeStringVector(netCDF::NcGroup&, std::string const&, std::string const&, std::vector<std::string, std::allocator<std::string> > const&, bool)':
Ncdf.cpp:(.text+0x22e6): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
Ncdf.cpp:(.text+0x243a): undefined reference to `netCDF::NcGroup::addVar(std::string const&, netCDF::NcType const&, netCDF::NcDim const&) const'
Ncdf.cpp:(.text+0x2505): undefined reference to `netCDF::NcVar::putVar(std::vector<unsigned long, std::allocator<unsigned long> > const&, std::string const&) const'
../../CAnEnIO/libAnEnIO.a(Ncdf.cpp.o): In function `Ncdf::writeArray4D(netCDF::NcGroup&, Array4D const&, std::string const&, std::array<std::string, 4ul> const&, std::array<bool, 4ul> const&)':
Ncdf.cpp:(.text+0x28b0): undefined reference to `netCDF::NcGroup::getVar(std::string const&, netCDF::NcGroup::Location) const'
Ncdf.cpp:(.text+0x2d0c): undefined reference to `netCDF::NcGroup::addVar(std::string const&, netCDF::NcType const&, std::vector<netCDF::NcDim, std::allocator<netCDF::NcDim> > const&) const'
../../CAnEn/libAnEn.a(Functions.cpp.o): In function `unsigned short boost::date_time::month_str_to_ushort<boost::gregorian::greg_month>(std::string const&)':
Functions.cpp:(.text._ZN5boost9date_time19month_str_to_ushortINS_9gregorian10greg_monthEEEtRKSs[_ZN5boost9date_time19month_str_to_ushortINS_9gregorian10greg_monthEEEtRKSs]+0xab): undefined reference to `boost::gregorian::greg_month::get_month_map_ptr()'
collect2: error: ld returned 1 exit status
apps/anen_netcdf/CMakeFiles/anen_netcdf.dir/build.make:116: recipe for target 'apps/anen_netcdf/anen_netcdf' failed
make[3]: *** [apps/anen_netcdf/anen_netcdf] Error 1
make[3]: Leaving directory '/sapphire/s0/wuh20/github/AnalogsEnsemble/build'
CMakeFiles/Makefile2:369: recipe for target 'apps/anen_netcdf/CMakeFiles/anen_netcdf.dir/all' failed
make[2]: *** [apps/anen_netcdf/CMakeFiles/anen_netcdf.dir/all] Error 2
make[2]: Leaving directory '/sapphire/s0/wuh20/github/AnalogsEnsemble/build'
CMakeFiles/Makefile2:376: recipe for target 'apps/anen_netcdf/CMakeFiles/anen_netcdf.dir/rule' failed
make[1]: *** [apps/anen_netcdf/CMakeFiles/anen_netcdf.dir/rule] Error 2
make[1]: Leaving directory '/sapphire/s0/wuh20/github/AnalogsEnsemble/build'
Makefile:244: recipe for target 'anen_netcdf' failed
make: *** [anen_netcdf] Error 2
Weiming-Hu commented 3 years ago

Although this looks like a linking error from other packages, this is indeed caused by the inclusion of libTorch library. As mentioned in this ticket, this issue is caused by the use of _GLIBCXX_USE_CXX11_ABI=0 inserted by libTorch while building. The error messages are very misleading.

Weiming-Hu commented 3 years ago

I need to make sure that I download the cxx-11 ABS version of libTorch as provided here

Weiming-Hu commented 3 years ago

The CXX11 ABS version is now available from the official website. That is very convenient.