search

nanoporetech / dorado

Oxford Nanopore's Basecaller
https://nanoporetech.com/
Other
477 stars 59 forks source link

Assertion failed: (ptr != nullptr), function mtl_for_tensor, file metal_utils.cpp, line 138. #7

Closed mjfos2r closed 1 year ago

mjfos2r commented 2 years ago

Greetings! I am currently attempting to use Dorado on my Macbook Pro and running into the title'd issue. Computer Specs:

❯ sw_vers
ProductName:    macOS
ProductVersion: 12.4
BuildVersion:   21F79

Model Name: MacBook Pro Model Identifier: MacBookPro18,3 Chip: Apple M1 Pro Total Number of Cores: 10 (8 performance and 2 efficiency) Memory: 16 GB System Firmware Version: 7459.121.3 OS Loader Version: 7459.121.3 Apple M1 Pro:

Chipset Model: Apple M1 Pro Type: GPU Bus: Built-In Total Number of Cores: 16 Vendor: Apple (0x106b) Metal Family: Supported, Metal GPUFamily Apple 7

cmake version: 

❯ cmake --version
cmake version 3.23.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

and path of cmake

❯ which cmake
/opt/homebrew/bin/cmake

I have been attempting to run basecalling on a run I did last week. my first step was basecalling the barcode for the known lambdaphage control I added. I moved all of the fast5 files for the barcoded lambda control into a directory for testing and got the following error:

❯ ./dorado basecaller models/dna_r9.4.1_e8.1_hac@v3.3 /Users/michaelfoster/sequencing/dorado_test/ >lambda-dorado-test.sam
> Creating basecall pipeline
Assertion failed: (ptr != nullptr), function mtl_for_tensor, file metal_utils.cpp, line 138.
[1]    52214 abort      ./dorado basecaller models/dna_r9.4.1_e8.1_hac@v3.3  > lambda-dorado-test.sam

okay, just gonna rm -rf the entire dorado directory and do it from the start.

here is the entire process starting from git clone:

❯ git clone git@github.com:nanoporetech/dorado.git
Cloning into 'dorado'...
remote: Enumerating objects: 1172, done.
remote: Counting objects: 100% (1172/1172), done.
remote: Compressing objects: 100% (428/428), done.
remote: Total 1172 (delta 762), reused 1138 (delta 735), pack-reused 0
Receiving objects: 100% (1172/1172), 366.55 KiB | 1.68 MiB/s, done.
Resolving deltas: 100% (762/762), done.
❯ cd dorado
❯ ls
CMakeLists.txt          LICENCE.txt             dorado                  make_koi_archive_win.sh
DEV.md                  README.md               make_koi_archive.sh     tests
❯ cmake -S . -B cmake-build
-- The C compiler identification is AppleClang 13.1.6.13160021
-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Submodule update
Submodule 'dorado/3rdparty/HighFive' (https://github.com/BlueBrain/HighFive.git) registered for path 'dorado/3rdparty/HighFive'
Submodule 'dorado/3rdparty/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'dorado/3rdparty/cpp-httplib'
Submodule 'dorado/3rdparty/11Zip' (https://github.com/Sygmei/11Zip.git) registered for path 'dorado/3rdparty/elzip'
Submodule 'dorado/3rdparty/hdf_plugins' (https://github.com/nanoporetech/vbz_compression.git) registered for path 'dorado/3rdparty/hdf_plugins'
Submodule 'dorado/3rdparty/toml11' (https://github.com/ToruNiina/toml11.git) registered for path 'dorado/3rdparty/toml11'
Submodule 'dorado/3rdparty/zstd' (https://github.com/facebook/zstd.git) registered for path 'dorado/3rdparty/zstd'
Cloning into '/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/HighFive'...
Cloning into '/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/cpp-httplib'...
Cloning into '/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip'...
Cloning into '/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/hdf_plugins'...
Cloning into '/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/toml11'...
Cloning into '/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/zstd'...
From https://github.com/BlueBrain/HighFive
 * branch            d3afd218ff04c3a2c6fbbd2a26b076715428bd57 -> FETCH_HEAD
Submodule path 'dorado/3rdparty/HighFive': checked out 'd3afd218ff04c3a2c6fbbd2a26b076715428bd57'
Submodule 'deps/catch2' (https://github.com/catchorg/Catch2.git) registered for path 'dorado/3rdparty/HighFive/deps/catch2'
Cloning into '/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/HighFive/deps/catch2'...
Submodule path 'dorado/3rdparty/HighFive/deps/catch2': checked out '216713a4066b79d9803d374f261ccb30c0fb451f'
Submodule path 'dorado/3rdparty/cpp-httplib': checked out 'fee8e97b4eeb34fe2e6e6294413d84e9e7a072a7'
Submodule path 'dorado/3rdparty/elzip': checked out '94a125161e4acab2638d8becd99af352d515b793'
Submodule 'extlibs/minizip' (https://github.com/zlib-ng/minizip-ng.git) registered for path 'dorado/3rdparty/elzip/extlibs/minizip'
Cloning into '/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip'...
From https://github.com/zlib-ng/minizip-ng
 * branch            99d39015e29703af2612277180ea586805f655ea -> FETCH_HEAD
Submodule path 'dorado/3rdparty/elzip/extlibs/minizip': checked out '99d39015e29703af2612277180ea586805f655ea'
Submodule path 'dorado/3rdparty/hdf_plugins': checked out '02fb8f50b93921ffa3c040106e809aaf8adbe5c5'
Submodule 'third_party/streamvbyte' (https://github.com/lemire/streamvbyte.git) registered for path 'dorado/3rdparty/hdf_plugins/third_party/streamvbyte'
Cloning into '/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/hdf_plugins/third_party/streamvbyte'...
Submodule path 'dorado/3rdparty/hdf_plugins/third_party/streamvbyte': checked out '1813d4ec3d732f3f615821fd8d1f725204c15ecc'
Submodule path 'dorado/3rdparty/toml11': checked out '59243256528d4133321e845c3193db2d2725e6ee'
Submodule path 'dorado/3rdparty/zstd': checked out '97a3da1df009d4dc67251de0c4b1c9d7fe286fc1'
-- Downloading metal-cpp
-- Downloading metal-cpp - done
-- Extracting metal-cpp
-- Extracting metal-cpp - done
-- Downloading pod5-0.0.14-Darwin
-- Downloading pod5-0.0.14-Darwin - done
-- Extracting pod5-0.0.14-Darwin
-- Extracting pod5-0.0.14-Darwin - done
-- Building version 1.0.3
-- No CMAKE_BUILD_TYPE set - defaulting to Debug
-- Found HDF5: /opt/homebrew/Cellar/hdf5/1.12.2/lib/libhdf5.dylib;/opt/homebrew/opt/libaec/lib/libsz.dylib;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libz.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libdl.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libm.tbd (found suitable version "1.12.2", minimum required is "1.8.16")  
-- Found zstd: /opt/homebrew/lib/libzstd.dylib (found suitable version "1.5.2", minimum required is "1.3.1") 
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Success
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Success
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Downloading torch-1.10.2-Darwin
-- Downloading torch-1.10.2-Darwin - done
-- Extracting torch-1.10.2-Darwin
-- Extracting torch-1.10.2-Darwin - done
CMake Warning at dorado/3rdparty/torch-1.10.2-Darwin/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  dorado/3rdparty/torch-1.10.2-Darwin/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
  CMakeLists.txt:195 (find_package)

-- Found Torch: /Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/torch-1.10.2-Darwin/torch/lib/libtorch.dylib  
-- Found HDF5: /opt/homebrew/Cellar/hdf5/1.12.2/lib/libhdf5.dylib;/opt/homebrew/opt/libaec/lib/libsz.dylib;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libz.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libdl.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libm.tbd;/opt/homebrew/Cellar/hdf5/1.12.2/lib/libhdf5_cpp.a;/opt/homebrew/Cellar/hdf5/1.12.2/lib/libhdf5.a;/opt/homebrew/opt/libaec/lib/libsz.dylib;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libz.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libdl.tbd;/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk/usr/lib/libm.tbd (found version "1.12.2") found components: C CXX HL 
-- Found OpenSSL: /opt/homebrew/opt/openssl@3/lib/libcrypto.a (found version "3.0.2")  
-- Using CMake version 3.23.1
CMake Deprecation Warning at dorado/3rdparty/elzip/extlibs/minizip/CMakeLists.txt:56 (cmake_policy):
  The OLD behavior for policy CMP0074 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.

-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of off64_t
-- Check size of off64_t - failed
-- Looking for fseeko
-- Looking for fseeko - found
-- Found PkgConfig: /opt/homebrew/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'openssl'
--   No package 'openssl' found
-- Using OpenSSL 3.0.2
-- Character encoding support requires iconv
-- The following features have been enabled:

 * MZ_COMPAT, Enables compatibility layer
 * MZ_LIBCOMP, Enables Apple compression
 * MZ_FETCH_LIBS, Enables fetching third-party libraries if not found
 * MZ_OPENSSL, Enables OpenSSL for encryption
 * MZ_LIBBSD, Build with libbsd for crypto random

-- The following features have been disabled:

 * MZ_ZLIB, Enables ZLIB compression
 * MZ_BZIP2, Enables BZIP2 compression
 * MZ_LZMA, Enables LZMA & XZ compression
 * MZ_ZSTD, Enables ZSTD compression
 * MZ_FORCE_FETCH_LIBS, Enables fetching third-party libraries always
 * MZ_PKCRYPT, Enables PKWARE traditional encryption
 * MZ_WZAES, Enables WinZIP AES encryption
 * MZ_SIGNING, Enables zip signing support
 * MZ_ICONV, Enables iconv string encoding conversion library
 * MZ_COMPRESS_ONLY, Only support compression
 * MZ_DECOMPRESS_ONLY, Only support decompression
 * MZ_FILE32_API, Builds using posix 32-bit file api
 * MZ_BUILD_TESTS, Builds minizip test executable
 * MZ_BUILD_UNIT_TESTS, Builds minizip unit test project
 * MZ_BUILD_FUZZ_TESTS, Builds minizip fuzzer executables
 * MZ_CODE_COVERAGE, Builds with code coverage flags

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/michaelfoster/sequencing/basecalling/dorado/cmake-build

okay seems to have worked. it printed it couldn't find openssl but then used it anyway despite having everything installed ahead of time. I can list the steps taken to prep/download deps if needed.

ran the next command:

❯ cmake --build cmake-build --config Release -- -j
[  1%] Creating directories for 'streamvbyte'
[  4%] Compiling metal kernels
[  7%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_crypt.c.o
[  7%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm.c.o
[  8%] Building CXX object dorado/3rdparty/hdf_plugins/vbz_plugin/hdf_test_utils/CMakeFiles/hdf_test_utils.dir/hdf_id_helper.cpp.o
[  8%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm_buf.c.o
[ 10%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm_mem.c.o
[ 13%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_os.c.o
[ 13%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_zip.c.o
[ 14%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm_split.c.o
[ 16%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_zip_rw.c.o
[ 17%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm_libcomp.c.o
[ 19%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_crypt_openssl.c.o
[ 20%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_os_posix.c.o
[ 22%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_strm_os_posix.c.o
[ 23%] Building C object dorado/3rdparty/elzip/extlibs/minizip/CMakeFiles/minizip.dir/mz_compat.c.o
[ 25%] No download step for 'streamvbyte'
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_strm_libcomp.c:120:13: warning: variable 'total_in' set but not used [-Wunused-but-set-variable]
    int32_t total_in = 0;
            ^
1 warning generated.
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_zip.c:192:17: warning: unused function 'mz_zip_get_pk_verify' [-Wunused-function]
static uint16_t mz_zip_get_pk_verify(uint32_t dos_date, uint64_t crc, uint16_t flag)
                ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_compat.c:166:13: warning: unused variable 'written' [-Wunused-variable]
    int32_t written = 0;
            ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_compat.c:167:11: warning: unused variable 'opaque' [-Wunused-variable]
    void *opaque = NULL;
          ^
[ 26%] No patch step for 'streamvbyte'
2 warnings generated.
[ 27%] Performing configure step for 'streamvbyte'
1 warning generated.
-- No build type selected, default to Release
[ 29%] Linking CXX static library ../../../../../lib/libhdf_test_utils.a
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:40:9: warning: 'ERR_load_BIO_strings' is deprecated [-Wdeprecated-declarations]
        ERR_load_BIO_strings();
        ^
/opt/homebrew/opt/openssl@3/include/openssl/cryptoerr_legacy.h:31:1: note: 'ERR_load_BIO_strings' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int ERR_load_BIO_strings(void);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:43:9: warning: 'ENGINE_load_builtin_engines' is deprecated [-Wdeprecated-declarations]
        ENGINE_load_builtin_engines();
        ^
/opt/homebrew/opt/openssl@3/include/openssl/engine.h:358:1: note: 'ENGINE_load_builtin_engines' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 void ENGINE_load_builtin_engines(void);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:44:9: warning: 'ENGINE_register_all_complete' is deprecated [-Wdeprecated-declarations]
        ENGINE_register_all_complete();
        ^
/opt/homebrew/opt/openssl@3/include/openssl/engine.h:415:1: note: 'ENGINE_register_all_complete' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int ENGINE_register_all_complete(void);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:93:18: warning: 'SHA1_Init' is deprecated [-Wdeprecated-declarations]
        result = SHA1_Init(&sha->ctx1);
                 ^
/opt/homebrew/opt/openssl@3/include/openssl/sha.h:49:1: note: 'SHA1_Init' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int SHA1_Init(SHA_CTX *c);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:95:18: warning: 'SHA256_Init' is deprecated [-Wdeprecated-declarations]
        result = SHA256_Init(&sha->ctx256);
                 ^
/opt/homebrew/opt/openssl@3/include/openssl/sha.h:73:1: note: 'SHA256_Init' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int SHA256_Init(SHA256_CTX *c);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:114:18: warning: 'SHA1_Update' is deprecated [-Wdeprecated-declarations]
        result = SHA1_Update(&sha->ctx1, buf, size);
                 ^
/opt/homebrew/opt/openssl@3/include/openssl/sha.h:50:1: note: 'SHA1_Update' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int SHA1_Update(SHA_CTX *c, const void *data, size_t len);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:116:18: warning: 'SHA256_Update' is deprecated [-Wdeprecated-declarations]
        result = SHA256_Update(&sha->ctx256, buf, size);
                 ^
/opt/homebrew/opt/openssl@3/include/openssl/sha.h:74:1: note: 'SHA256_Update' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int SHA256_Update(SHA256_CTX *c,
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:136:18: warning: 'SHA1_Final' is deprecated [-Wdeprecated-declarations]
        result = SHA1_Final(digest, &sha->ctx1);
                 ^
/opt/homebrew/opt/openssl@3/include/openssl/sha.h:51:1: note: 'SHA1_Final' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int SHA1_Final(unsigned char *md, SHA_CTX *c);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:140:18: warning: 'SHA256_Final' is deprecated [-Wdeprecated-declarations]
        result = SHA256_Final(digest, &sha->ctx256);
                 ^
/opt/homebrew/opt/openssl@3/include/openssl/sha.h:76:1: note: 'SHA256_Final' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int SHA256_Final(unsigned char *md, SHA256_CTX *c);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:208:5: warning: 'AES_encrypt' is deprecated [-Wdeprecated-declarations]
    AES_encrypt(buf, buf, &aes->key);
    ^
/opt/homebrew/opt/openssl@3/include/openssl/aes.h:56:1: note: 'AES_encrypt' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:220:5: warning: 'AES_decrypt' is deprecated [-Wdeprecated-declarations]
    AES_decrypt(buf, buf, &aes->key);
    ^
/opt/homebrew/opt/openssl@3/include/openssl/aes.h:59:1: note: 'AES_decrypt' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:237:14: warning: 'AES_set_encrypt_key' is deprecated [-Wdeprecated-declarations]
    result = AES_set_encrypt_key(key, key_bits, &aes->key);
             ^
/opt/homebrew/opt/openssl@3/include/openssl/aes.h:50:1: note: 'AES_set_encrypt_key' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:258:14: warning: 'AES_set_decrypt_key' is deprecated [-Wdeprecated-declarations]
    result = AES_set_decrypt_key(key, key_bits, &aes->key);
             ^
/opt/homebrew/opt/openssl@3/include/openssl/aes.h:53:1: note: 'AES_set_decrypt_key' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:326:5: warning: 'HMAC_CTX_free' is deprecated [-Wdeprecated-declarations]
    HMAC_CTX_free(hmac->ctx);
    ^
/opt/homebrew/opt/openssl@3/include/openssl/hmac.h:35:1: note: 'HMAC_CTX_free' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 void HMAC_CTX_free(HMAC_CTX *ctx);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:344:17: warning: 'HMAC_CTX_new' is deprecated [-Wdeprecated-declarations]
    hmac->ctx = HMAC_CTX_new();
                ^
/opt/homebrew/opt/openssl@3/include/openssl/hmac.h:33:1: note: 'HMAC_CTX_new' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 HMAC_CTX *HMAC_CTX_new(void);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:351:14: warning: 'HMAC_Init_ex' is deprecated [-Wdeprecated-declarations]
    result = HMAC_Init_ex(hmac->ctx, key, key_length, evp_md, NULL);
             ^
/opt/homebrew/opt/openssl@3/include/openssl/hmac.h:43:1: note: 'HMAC_Init_ex' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int HMAC_Init_ex(HMAC_CTX *ctx, const void *key, int len,
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:367:14: warning: 'HMAC_Update' is deprecated [-Wdeprecated-declarations]
    result = HMAC_Update(hmac->ctx, buf, size);
             ^
/opt/homebrew/opt/openssl@3/include/openssl/hmac.h:45:1: note: 'HMAC_Update' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int HMAC_Update(HMAC_CTX *ctx, const unsigned char *data,
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:387:18: warning: 'HMAC_Final' is deprecated [-Wdeprecated-declarations]
        result = HMAC_Final(hmac->ctx, digest, (uint32_t *)&digest_size);
                 ^
/opt/homebrew/opt/openssl@3/include/openssl/hmac.h:47:1: note: 'HMAC_Final' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int HMAC_Final(HMAC_CTX *ctx, unsigned char *md,
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:391:18: warning: 'HMAC_Final' is deprecated [-Wdeprecated-declarations]
        result = HMAC_Final(hmac->ctx, digest, (uint32_t *)&digest_size);
                 ^
/opt/homebrew/opt/openssl@3/include/openssl/hmac.h:47:1: note: 'HMAC_Final' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 int HMAC_Final(HMAC_CTX *ctx, unsigned char *md,
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:418:23: warning: 'HMAC_CTX_new' is deprecated [-Wdeprecated-declarations]
        target->ctx = HMAC_CTX_new();
                      ^
/opt/homebrew/opt/openssl@3/include/openssl/hmac.h:33:1: note: 'HMAC_CTX_new' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 HMAC_CTX *HMAC_CTX_new(void);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
/Users/michaelfoster/sequencing/basecalling/dorado/dorado/3rdparty/elzip/extlibs/minizip/mz_crypt_openssl.c:420:14: warning: 'HMAC_CTX_copy' is deprecated [-Wdeprecated-declarations]
    result = HMAC_CTX_copy(target->ctx, source->ctx);
             ^
/opt/homebrew/opt/openssl@3/include/openssl/hmac.h:49:1: note: 'HMAC_CTX_copy' has been explicitly marked deprecated here
OSSL_DEPRECATEDIN_3_0 __owur int HMAC_CTX_copy(HMAC_CTX *dctx, HMAC_CTX *sctx);
^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:182:49: note: expanded from macro 'OSSL_DEPRECATEDIN_3_0'
#   define OSSL_DEPRECATEDIN_3_0                OSSL_DEPRECATED(3.0)
                                                ^
/opt/homebrew/opt/openssl@3/include/openssl/macros.h:62:52: note: expanded from macro 'OSSL_DEPRECATED'
#     define OSSL_DEPRECATED(since) __attribute__((deprecated))
                                                   ^
21 warnings generated.
[ 30%] Linking C static library libminizip.a
[ 30%] Built target hdf_test_utils
[ 30%] Built target minizip
[ 35%] Building CXX object dorado/3rdparty/elzip/CMakeFiles/elzip.dir/src/zipper.cpp.o
[ 35%] Building CXX object dorado/3rdparty/elzip/CMakeFiles/elzip.dir/src/elzip.cpp.o
[ 35%] Building CXX object dorado/3rdparty/elzip/CMakeFiles/elzip.dir/src/unzipper.cpp.o
-- The C compiler identification is AppleClang 13.1.6.13160021
-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Deprecation Warning at CMakeLists.txt:9 (cmake_policy):
  The OLD behavior for policy CMP0065 will be removed from a future version
  of CMake.

  The cmake-policies(7) manual explains that the OLD behaviors of all
  policies are deprecated and that a policy should be set to OLD only under
  specific short-term circumstances.  Projects should be ported to the NEW
  behavior and not rely on setting a policy to OLD.

-- CMAKE_SYSTEM_PROCESSOR: arm64
-- CMAKE_BUILD_TYPE: Release
-- CMAKE_C_COMPILER: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- CMAKE_C_FLAGS: "-std=c99"     
-- CMAKE_C_FLAGS_DEBUG: -g
-- CMAKE_C_FLAGS_RELEASE: -O3 -DNDEBUG
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/michaelfoster/sequencing/basecalling/dorado/cmake-build/streamvbyte/src/streamvbyte-build
[ 36%] Performing build step for 'streamvbyte'
[  4%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbyte_encode.c.o
[ 12%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbyte_encode.c.o
[ 12%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbyte_zigzag.c.o
[ 16%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbyte_decode.c.o
[ 20%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbytedelta_decode.c.o
[ 25%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbyte_0124_encode.c.o
[ 29%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbyte_0124_decode.c.o
[ 38%] Creating metallib
[ 37%] Building C object CMakeFiles/streamvbyte.dir/src/streamvbytedelta_encode.c.o
[ 37%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbyte_decode.c.o
[ 45%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbyte_zigzag.c.o
[ 45%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbytedelta_encode.c.o
[ 54%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbyte_0124_decode.c.o
[ 54%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbytedelta_decode.c.o
[ 58%] Building C object CMakeFiles/streamvbyte_static.dir/src/streamvbyte_0124_encode.c.o
[ 39%] Linking CXX static library libelzip.a
[ 39%] Built target metal-lib
[ 62%] Linking C shared library libstreamvbyte.dylib
[ 39%] Built target elzip
[ 66%] Linking C static library libstreamvbyte_static.a
[ 66%] Built target streamvbyte_static
[ 66%] Built target streamvbyte
[ 75%] Building C object CMakeFiles/writeseq.dir/tests/writeseq.c.o
[ 79%] Building C object CMakeFiles/example.dir/example.c.o
[ 79%] Building C object CMakeFiles/unit.dir/tests/unit.c.o
[ 83%] Building C object CMakeFiles/perf.dir/tests/perf.c.o
[ 87%] Linking C executable example
[ 91%] Linking C executable writeseq
Apple clang version 13.1.6 (clang-1316.0.21.2.3)
Target: arm64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -dynamic -arch arm64 -platform_version macos 12.0.0 12.3 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -o example -search_paths_first -headerpad_max_install_names CMakeFiles/example.dir/example.c.o libstreamvbyte_static.a -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.1.6/lib/darwin/libclang_rt.osx.a
Apple clang version 13.1.6 (clang-1316.0.21.2.3)
Target: arm64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -dynamic -arch arm64 -platform_version macos 12.0.0 12.3 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -o writeseq -search_paths_first -headerpad_max_install_names CMakeFiles/writeseq.dir/tests/writeseq.c.o libstreamvbyte_static.a -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.1.6/lib/darwin/libclang_rt.osx.a
[ 95%] Linking C executable perf
Apple clang version 13.1.6 (clang-1316.0.21.2.3)
Target: arm64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -dynamic -arch arm64 -platform_version macos 12.0.0 12.3 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -o perf -search_paths_first -headerpad_max_install_names CMakeFiles/perf.dir/tests/perf.c.o libstreamvbyte_static.a -lm -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.1.6/lib/darwin/libclang_rt.osx.a
[100%] Linking C executable unit
[100%] Built target example
[100%] Built target writeseq
Apple clang version 13.1.6 (clang-1316.0.21.2.3)
Target: arm64-apple-darwin21.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -dynamic -arch arm64 -platform_version macos 12.0.0 12.3 -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -o unit -search_paths_first -headerpad_max_install_names CMakeFiles/unit.dir/tests/unit.c.o libstreamvbyte_static.a -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.1.6/lib/darwin/libclang_rt.osx.a
[100%] Built target perf
[100%] Built target unit
[ 41%] Performing install step for 'streamvbyte'
Consolidate compiler generated dependencies of target streamvbyte_static
Consolidate compiler generated dependencies of target streamvbyte
[ 66%] Built target streamvbyte_static
[ 66%] Built target streamvbyte
Consolidate compiler generated dependencies of target example
Consolidate compiler generated dependencies of target unit
Consolidate compiler generated dependencies of target writeseq
Consolidate compiler generated dependencies of target perf
[ 75%] Built target example
[ 87%] Built target unit
[100%] Built target writeseq
[100%] Built target perf
Install the project...
-- Install configuration: "Release"
-- Installing: /Users/michaelfoster/sequencing/basecalling/dorado/cmake-build/streamvbyte_lib/include/streamvbyte.h
-- Installing: /Users/michaelfoster/sequencing/basecalling/dorado/cmake-build/streamvbyte_lib/include/streamvbytedelta.h
-- Installing: /Users/michaelfoster/sequencing/basecalling/dorado/cmake-build/streamvbyte_lib/include/streamvbyte_zigzag.h
-- Installing: /Users/michaelfoster/sequencing/basecalling/dorado/cmake-build/streamvbyte_lib/lib/libstreamvbyte.dylib
-- Installing: /Users/michaelfoster/sequencing/basecalling/dorado/cmake-build/streamvbyte_lib/lib/libstreamvbyte_static.a
[ 42%] Completed 'streamvbyte'
[ 42%] Built target streamvbyte
[ 45%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/CMakeFiles/vbz.dir/v0/vbz_streamvbyte.cpp.o
[ 45%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/CMakeFiles/vbz.dir/v1/vbz_streamvbyte.cpp.o
[ 47%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/CMakeFiles/vbz.dir/vbz.cpp.o
[ 48%] Linking CXX static library ../../../../lib/libvbz.a
[ 48%] Built target vbz
[ 52%] Building CXX object dorado/3rdparty/hdf_plugins/vbz_plugin/CMakeFiles/vbz_hdf_plugin.dir/vbz_plugin.cpp.o
[ 52%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/test/CMakeFiles/vbz_test.dir/streamvbyte_test.cpp.o
[ 52%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/test/CMakeFiles/vbz_test.dir/vbz_test.cpp.o
[ 54%] Building CXX object dorado/3rdparty/hdf_plugins/vbz/test/CMakeFiles/vbz_test.dir/main.cpp.o
[ 55%] Linking CXX static library ../../../../lib/libvbz_hdf_plugin.a
[ 55%] Built target vbz_hdf_plugin
[ 57%] Building CXX object dorado/3rdparty/hdf_plugins/vbz_plugin/test/CMakeFiles/vbz_hdf_plugin_test.dir/main.cpp.o
[ 60%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/ScalerNode.cpp.o
[ 60%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/cli/basecaller.cpp.o
[ 61%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/cli/download.cpp.o
[ 63%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/nn/CRFModel.cpp.o
[ 67%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/BasecallerNode.cpp.o
[ 67%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/ReadPipeline.cpp.o
[ 67%] Building CXX object dorado/3rdparty/hdf_plugins/vbz_plugin/test/CMakeFiles/vbz_hdf_plugin_test.dir/vbz_hdf_plugin_test.cpp.o
[ 69%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/data_loader/DataLoader.cpp.o
[ 70%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/read_pipeline/WriterNode.cpp.o
[ 72%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/decode/beam_search.cpp.o
[ 75%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/decode/GPUDecoder.cpp.o
[ 75%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/compat_utils.cpp.o
[ 76%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/decode/CPUDecoder.cpp.o
[ 77%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/decode/fast_hash.cpp.o
[ 80%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/sequence_utils.cpp.o
[ 82%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/nn/MetalCRFModel.cpp.o
[ 83%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/stitch.cpp.o
[ 80%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/tensor_utils.cpp.o
[ 85%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/utils/metal_utils.cpp.o
[ 86%] Building CXX object CMakeFiles/dorado_lib.dir/dorado/decode/MTLDecoder.cpp.o
[ 88%] Linking CXX executable ../../../../../bin/vbz_test
[ 88%] Built target vbz_test
[ 89%] Linking CXX executable ../../../../../bin/vbz_hdf_plugin_test
[ 89%] Built target vbz_hdf_plugin_test
[ 91%] Linking CXX static library lib/libdorado_lib.a
[ 91%] Built target dorado_lib
[ 92%] Building CXX object CMakeFiles/dorado.dir/dorado/main.cpp.o
[ 94%] Building CXX object tests/CMakeFiles/dorado_tests.dir/main.cpp.o
[ 97%] Building CXX object tests/CMakeFiles/dorado_tests.dir/Fast5DataLoaderTest.cpp.o
[ 97%] Building CXX object tests/CMakeFiles/dorado_tests.dir/ReadTest.cpp.o
[ 98%] Linking CXX executable bin/dorado
[ 98%] Built target dorado
[100%] Linking CXX executable dorado_tests
[100%] Built target dorado_tests

looks to have worked but only error is: 

CMake Warning at dorado/3rdparty/torch-1.10.2-Darwin/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  dorado/3rdparty/torch-1.10.2-Darwin/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)
  CMakeLists.txt:195 (find_package)

anyway, ran the tests and they worked fine.

❯ ctest --test-dir cmake-build
Internal ctest changing into directory: /Users/michaelfoster/sequencing/basecalling/dorado/cmake-build
Test project /Users/michaelfoster/sequencing/basecalling/dorado/cmake-build
    Start 1: vbz_test
1/3 Test #1: vbz_test .........................   Passed    4.26 sec
    Start 2: vbz_hdf_plugin_test
2/3 Test #2: vbz_hdf_plugin_test ..............   Passed    0.12 sec
    Start 3: dorado_tests
3/3 Test #3: dorado_tests .....................   Passed    1.06 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) =   5.44 sec

okay cool. lets run the test binaries in cmake-build/bin/

❯ ./vbz_hdf_plugin_test
===============================================================================
All tests passed (12 assertions in 6 test cases)

that worked, the next one:

❯ ./vbz_test
===============================================================================
All tests passed (101 assertions in 13 test cases)

cool, that one worked as well.

now to download the models. will just download them to the bin folder as it downloads to . as default.

❯ ./dorado download
 - downloading dna_r10.4.1_e8.2_fast@v3.5.1 [200]
 - downloading dna_r10.4.1_e8.2_hac@v3.5.1 [200]
 - downloading dna_r10.4.1_e8.2_sup@v3.5.1 [200]
 - downloading dna_r10.4_e8.1_fast@v3.4 [200]
 - downloading dna_r10.4_e8.1_hac@v3.4 [200]
 - downloading dna_r10.4_e8.1_sup@v3.4 [200]
 - downloading dna_r9.4.1_e8.1_fast@v3.4 [200]
 - downloading dna_r9.4.1_e8.1_hac@v3.3 [200]
 - downloading dna_r9.4.1_e8.1_sup@v3.3 [200]
 - downloading dna_r9.4.1_e8_fast@v3.4 [200]
 - downloading dna_r9.4.1_e8_hac@v3.3 [200]
 - downloading dna_r9.4.1_e8_sup@v3.3 [200]

cool, lets basecall the lambda barcode now.

❯ ./dorado basecaller dna_r9.4.1_e8.1_hac@v3.3 /Users/michaelfoster/sequencing/dorado_test/ >dorado-test-lambda.sam
> Creating basecall pipeline
Assertion failed: (ptr != nullptr), function mtl_for_tensor, file metal_utils.cpp, line 138.
[1]    53532 abort      ./dorado basecaller dna_r9.4.1_e8.1_hac@v3.3  > dorado-test-lambda.sam

same error. Where do I go next for troubleshooting? Is it an issue with running it directly from the cmake-build/bin folder? Honestly I have no idea what the issue is but hopefully I've provided enough information to assist in troubleshooting.

Thanks again. Looking forward to being able to do entire sequencing runs on my macbook pro without having to ssh into the cluster.

iiSeymour commented 2 years ago

Hey @mjfos2r

Well done on getting a successful build. You are the first to try the experimental metal support afaik 👏🏻 .

The metal support is still very much a work in progress so has some sharp edges as you have found. Currently, we have a limitation of allocating tensors >4GB on macos which is what you are seeing here with the default batch size. Try running with 2 runners and a batch size of 384 (we plan to set parameters to their optimal values soon so the user doesn't need to worry about this).

./dorado basecaller -r 2 -b 384 dna_r9.4.1_e8.1_hac@v3.3 /Users/michaelfoster/sequencing/dorado_test/ >dorado-test-lambda.sam

Note: We have the model running well on the GPU but the decoder is still on the CPU and is currently a bottleneck. We are working on resolving this atm.

mjfos2r commented 2 years ago

@iiSeymour worked! thanks for the help.

❯ ./dorado basecaller -r 2 -b 384 --emit-fastq dna_r9.4.1_e8.1_hac@v3.3 /Users/michaelfoster/sequencing/dorado_test/ >dorado-test-lambda.sam
> Creating basecall pipeline
> Loaded 132453 reads
> Reads basecalled: 132453
> Samples/s: 4.066346e+04
> Finished

took almost 2 hours but still. Super excited to see how it improves from here!

vdejager commented 2 years ago

nice. Just to benchmark: a run with amplicon data (reads max 2kb) with 3.625.247 reads and the dna_r9.2.1_e8.1_sup@v3.3 model took about 210 minutes on the following system:

ThinkSystem SD650-N v2 Intel Xeon Platinum 8360Y (2x),36 Cores/Socket, 2.4 GHz (Speed Select SKU), 250W NVIDIA A100 (4x),40 GiB HMB2 memory with 5 active memory stacks per GPU 16x32 GiB,3200 MHz, DDR4 512GiB160GiB HMB2(7.111 GiB) 2xHDR100 ConnectX-6 single port2x25GbE SFP28 LOM1x1GbE RJ45 LOM

I get 1/4 a share of it, so 18 cores and 1 A100 I will checkout the M1 build as well.

iiSeymour commented 2 years ago

@mjfos2r @thomasvangurp we've just dropped a performance patch for m1 10bf86e7c4e94b66fef68a9aa59a5267aeca0385 . Should be a nice improvement with hac models - suggested params right now are -r 2 -b 1536 for fast and -r 2 -b 768 -c 4000 for hac.

More to come.

Chris.

mjfos2r commented 2 years ago

@iiSeymour so I used the same data as the successful run above and attempted to output as fastq and it segfaulted after 30ish minutes. 

❯ ./dorado basecaller -r 2 -b 768 -c 4000 dna_r9.4.1_e8.1_hac@v3.3 /Users/michaelfoster/sequencing/dorado_test/ >/Users/michaelfoster/sequencing/dorado_test/dorado-lambda-output.fastq
> Creating basecall pipeline
[1]    28894 segmentation fault  ./dorado basecaller -r 2 -b 768 -c 4000 dna_r9.4.1_e8.1_hac@v3.3  >

re-running now with output as .sam, will update with results. mem usage was consistently 80-85% and core util was less than 40% for any one core the entire time.

UPDATE:

Completely forgot that --emit-fastq was an option. anyways it worked this time. 

❯ ./dorado basecaller -r 2 -b 768 -c 4000 dna_r9.4.1_e8.1_hac@v3.3 /Users/michaelfoster/sequencing/dorado_test/fast5/ >/Users/michaelfoster/sequencing/dorado_test/called/dorado-lambda-output.sam
> Creating basecall pipeline
> Loaded 132453 reads
> Reads basecalled: 132453
> Samples/s: 1.384669e+06
> Finished

took 55 minutes. wonderful speed increase. great job so far, can't wait to see how much better it gets. Will see how fast running a fast model is and update.

UPDATE:

fast model is aptly named. using the same lambdaphage data it took 8 minutes.

❯ ./dorado basecaller -r 2 -b 1536 dna_r9.4.1_e8.1_fast@v3.4 /Users/michaelfoster/sequencing/dorado_test/fast5/ >/Users/michaelfoster/sequencing/dorado_test/called/dorado-lambda-output-fast.sam
> Creating basecall pipeline
> Loaded 132453 reads
> Reads basecalled: 132453
> Samples/s: 8.976293e+06
> Finished

difference between the fast .sam and HAC .sam is 9,220,712 bytes. very impressed with the fast performance.

iiSeymour commented 1 year ago

The -r, -b- c parameters are all set correctly in the latest release along with some performance optimisation.