gmake: *** [Makefile:166: all] Error 2

[itslittman]

I saw there were some updates to the code recently, so I figured I'd try building dorado from source. I noticed the README only provided specific instructions for CUDA:

cmake -S . -B cmake-build -DCMAKE_CUDA_COMPILER=/nvcc

So I tried this since I have a mac:

cmake -S . -B cmake-build

It seemed to work, but after moving on to

cmake --build cmake-build --config Release -j

it ran for a while and then eventually gave me a cryptic error:

gmake: *** [Makefile:166: all] Error 2

How should I proceed?

[iiSeymour]

Hey @itslittman

Yes, there are some nice osx performance improvements on the head of master right now. I suspect the issue you are seeing is because of the htslib requirement on autoconf@2.69 - see https://github.com/nanoporetech/dorado/blob/master/DEV.md#osx

HTH

Chris.

[itslittman]

@iiSeymour I linked autoconf 2.69/made sure that was the one in the PATH, and tried again, and I'm still getting the same gcc error

are these the correct compiler flags?

-- The C compiler identification is AppleClang 14.0.0.14000029 -- The CXX compiler identification is AppleClang 14.0.0.14000029 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Submodule update

[iiSeymour]

@itslittman can you delete the build directory, rerun the generation and build step again and post the full output.

$ rm -rf cmake-build
$ cmake -S . -B cmake-build
$ cmake --build cmake-build --config Release -j

[itslittman]

@iiSeymour I already tried that, same result - should it be using the Clang that comes with Xcode? or the homebrew gcc/g++? Or is this even a compiler issue at all?

This is the error log:

Edit: Oh boy, github did not like that formatting. I just attached the file. CMakeError.log

[iiSeymour]

Clang is fine. It's still not clear to me atm without the cmake output - can you paste the output from the terminal?

[itslittman]

Oh yes sorry:

Here's the output after the first command:

noahlittman@Noahs-MacBook-Pro-118 dorado % cmake -S . -B cmake-build
-- The C compiler identification is AppleClang 14.0.0.14000029
-- The CXX compiler identification is AppleClang 14.0.0.14000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Submodule update
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- Found HDF5: /Users/noahlittman/biology_programs/nanopolish/lib/libhdf5.a;/opt/homebrew/lib/libhdf5_cpp.a;/Users/noahlittman/biology_programs/nanopolish/lib/libhdf5.a (found version "1.12.2") found components: C CXX HL 
-- Found pod5-0.0.41-Darwin
-- Found torch-1.12.1-Darwin
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found Torch: /Users/noahlittman/dorado/dorado/3rdparty/torch-1.12.1-Darwin/torch/lib/libtorch.dylib  
-- Found OpenSSL: /opt/homebrew/opt/openssl@3/lib/libcrypto.a (found version "3.0.7")  
-- Building htslib
-- Done Building htslib
-- Using CMake version 3.25.1
CMake Deprecation Warning at dorado/3rdparty/elzip/extlibs/minizip/CMakeLists.txt:56 (cmake_policy):
  The OLD behavior for policy CMP0074 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.

-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - failed
-- Looking for fseeko
-- Looking for fseeko - found
-- Found PkgConfig: /opt/homebrew/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'openssl'
--   Found openssl, version 3.0.7
-- Using OpenSSL 3.0.7
-- Character encoding support requires iconv
-- The following features have been enabled:

 * MZ_COMPAT, Enables compatibility layer
 * MZ_LIBCOMP, Enables Apple compression
 * MZ_FETCH_LIBS, Enables fetching third-party libraries if not found
 * MZ_OPENSSL, Enables OpenSSL for encryption
 * MZ_LIBBSD, Build with libbsd for crypto random

-- The following features have been disabled:

 * MZ_ZLIB, Enables ZLIB compression
 * MZ_BZIP2, Enables BZIP2 compression
 * MZ_LZMA, Enables LZMA & XZ compression
 * MZ_ZSTD, Enables ZSTD compression
 * MZ_FORCE_FETCH_LIBS, Enables fetching third-party libraries always
 * MZ_PKCRYPT, Enables PKWARE traditional encryption
 * MZ_WZAES, Enables WinZIP AES encryption
 * MZ_SIGNING, Enables zip signing support
 * MZ_ICONV, Enables iconv string encoding conversion library
 * MZ_COMPRESS_ONLY, Only support compression
 * MZ_DECOMPRESS_ONLY, Only support decompression
 * MZ_FILE32_API, Builds using posix 32-bit file api
 * MZ_BUILD_TESTS, Builds minizip test executable
 * MZ_BUILD_UNIT_TESTS, Builds minizip unit test project
 * MZ_BUILD_FUZZ_TESTS, Builds minizip fuzzer executables
 * MZ_CODE_COVERAGE, Builds with code coverage flags

Setting warning flags
-- Performing Test WOLD_STYLE_CAST
-- Performing Test WOLD_STYLE_CAST - Success
-- Performing Test WSHADOW
-- Performing Test WSHADOW - Success
-- Found metal-cpp
-- Building version 1.0.3
-- No CMAKE_BUILD_TYPE set - defaulting to Debug
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- Found HDF5: /Users/noahlittman/biology_programs/nanopolish/lib/libhdf5.a (found suitable version "1.12.2", minimum required is "1.8.16")  
-- Found zstd: /opt/homebrew/lib/libzstd.dylib (found suitable version "1.5.2", minimum required is "1.3.1") 
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Configuring done
CMake Warning (dev) at dorado/3rdparty/edlib/CMakeLists.txt:54 (add_library):
  Policy CMP0063 is not set: Honor visibility properties for all target
  types.  Run "cmake --help-policy CMP0063" for policy details.  Use the
  cmake_policy command to set the policy and suppress this warning.

  Target "edlib" of type "STATIC_LIBRARY" has the following visibility
  properties set for CXX:

    CXX_VISIBILITY_PRESET
    VISIBILITY_INLINES_HIDDEN

  For compatibility CMake is not honoring them for this target.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Generating done
-- Build files have been written to: /Users/noahlittman/dorado/cmake-build

[itslittman]

@iiSeymour output 2: output.txt

[iiSeymour]

Okay thanks @itslittman - these are the relavent lines from the build step and it is indeed the compilation of htslib however it's not something I've seen and isn't imediately obvious what the solution is just yet.

Can't locate Autom4te/ChannelDefs.pm in @INC (you may need to install the Autom4te::ChannelDefs module) (@INC contains: /opt/homebrew/Cellar/autoconf@2.69/2.69/share/autoconf /Library/Perl/5.30/darwin-thread-multi-2level /Library/Perl/5.30 /Network/Library/Perl/5.30/darwin-thread-multi-2level /Network/Library/Perl/5.30 /Library/Perl/Updates/5.30.3 /System/Library/Perl/5.30/darwin-thread-multi-2level /System/Library/Perl/5.30 /System/Library/Perl/Extras/5.30/darwin-thread-multi-2level /System/Library/Perl/Extras/5.30) at /opt/homebrew/bin/autoheader line 41.
BEGIN failed--compilation aborted at /opt/homebrew/bin/autoheader line 41.
gmake[2]: *** [CMakeFiles/htslib_project.dir/build.make:92: 3rdparty/htslib/src/htslib_project-stamp/htslib_project-configure] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:328: CMakeFiles/htslib_project.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....

[itslittman]

@iiSeymour i googled it and it seems like it generally has to do with autoconf permissions. Should I have sudo'd something?

[iiSeymour]

You should never need to sudo to build. I think what I would do is remove all installs of autoconf.

$ brew uninstall autoconf

Verify

$ autoconf --version
zsh: command not found: autoconf

Reinstall and link autoconf

$ brew install autoconf@2.69
$ brew link autoconf@2.69

Start a new terminal and verifiy

$ autoconf --version
$ autoconf (GNU Autoconf) 2.69

If the above shows 2.69 then remove the old build directory and try building again.

$ rm -rf cmake-build
$ cmake -S . -B cmake-build
$ cmake --build cmake-build --config Release -j

[itslittman]

@iiSeymour 2.69 is definitely linked; I've uninstalled/reinstalled it a few times now

[itslittman]

And Autom4te/ChannelDefs.pm is definitely there where it is supposed to be in /opt/homebrew/Cellar/autoconf@2.69/2.69/share/autoconf/Autom4te

[itslittman]

@iiSeymour okay...I linked autoconf weird. So I relinstalled/relinked it. And now I get this: 2.txt

[iiSeymour]

The next issue appears to be linking HDF5.. your cmake generation step shows it's using a version shipped with your nanopolish build and not the prebuild version dorado provides.

Found HDF5: /Users/noahlittman/biology_programs/nanopolish/lib/libhdf5.a

[itslittman]

@iiSeymour ahh... how do I fix that without messing up nanopolish? Can I just temorarily remove that link and then relink the nanopolish one after I build dorado?

[iiSeymour]

I can't comment for nanopolish directly but yes temorarily cleaning up your paths should work. Make sure to remove the build directory and check the cmake generation step output for dorado to ensure all the paths look sensible.

Dorado sets the rpath so you don't need to set any paths for dynamic linking once built.

[itslittman]

@iiSeymour I cleaned up my path and it worked, thanks!!!

Noah