Closed mjfos2r closed 1 year ago
Hey @mjfos2r
Well done on getting a successful build. You are the first to try the experimental metal support afaik 👏🏻 .
The metal support is still very much a work in progress so has some sharp edges as you have found. Currently, we have a limitation of allocating tensors >4GB on macos which is what you are seeing here with the default batch size. Try running with 2 runners and a batch size of 384 (we plan to set parameters to their optimal values soon so the user doesn't need to worry about this).
./dorado basecaller -r 2 -b 384 dna_r9.4.1_e8.1_hac@v3.3 /Users/michaelfoster/sequencing/dorado_test/ >dorado-test-lambda.sam
Note: We have the model running well on the GPU but the decoder is still on the CPU and is currently a bottleneck. We are working on resolving this atm.
@iiSeymour worked! thanks for the help.
❯ ./dorado basecaller -r 2 -b 384 --emit-fastq dna_r9.4.1_e8.1_hac@v3.3 /Users/michaelfoster/sequencing/dorado_test/ >dorado-test-lambda.sam
> Creating basecall pipeline
> Loaded 132453 reads
> Reads basecalled: 132453
> Samples/s: 4.066346e+04
> Finished
took almost 2 hours but still. Super excited to see how it improves from here!
nice. Just to benchmark: a run with amplicon data (reads max 2kb) with 3.625.247 reads and the dna_r9.2.1_e8.1_sup@v3.3 model took about 210 minutes on the following system:
ThinkSystem SD650-N v2 Intel Xeon Platinum 8360Y (2x),36 Cores/Socket, 2.4 GHz (Speed Select SKU), 250W NVIDIA A100 (4x),40 GiB HMB2 memory with 5 active memory stacks per GPU 16x32 GiB,3200 MHz, DDR4 512GiB160GiB HMB2(7.111 GiB) 2xHDR100 ConnectX-6 single port2x25GbE SFP28 LOM1x1GbE RJ45 LOM
I get 1/4 a share of it, so 18 cores and 1 A100 I will checkout the M1 build as well.
@mjfos2r @thomasvangurp we've just dropped a performance patch for m1 10bf86e7c4e94b66fef68a9aa59a5267aeca0385 . Should be a nice improvement with hac
models - suggested params right now are -r 2 -b 1536
for fast
and -r 2 -b 768 -c 4000
for hac
.
More to come.
Chris.
@iiSeymour so I used the same data as the successful run above and attempted to output as fastq and it segfaulted after 30ish minutes.
❯ ./dorado basecaller -r 2 -b 768 -c 4000 dna_r9.4.1_e8.1_hac@v3.3 /Users/michaelfoster/sequencing/dorado_test/ >/Users/michaelfoster/sequencing/dorado_test/dorado-lambda-output.fastq
> Creating basecall pipeline
[1] 28894 segmentation fault ./dorado basecaller -r 2 -b 768 -c 4000 dna_r9.4.1_e8.1_hac@v3.3 >
re-running now with output as .sam, will update with results. mem usage was consistently 80-85% and core util was less than 40% for any one core the entire time.
Completely forgot that --emit-fastq was an option. anyways it worked this time.
❯ ./dorado basecaller -r 2 -b 768 -c 4000 dna_r9.4.1_e8.1_hac@v3.3 /Users/michaelfoster/sequencing/dorado_test/fast5/ >/Users/michaelfoster/sequencing/dorado_test/called/dorado-lambda-output.sam
> Creating basecall pipeline
> Loaded 132453 reads
> Reads basecalled: 132453
> Samples/s: 1.384669e+06
> Finished
took 55 minutes. wonderful speed increase. great job so far, can't wait to see how much better it gets. Will see how fast running a fast model is and update.
fast model is aptly named. using the same lambdaphage data it took 8 minutes.
❯ ./dorado basecaller -r 2 -b 1536 dna_r9.4.1_e8.1_fast@v3.4 /Users/michaelfoster/sequencing/dorado_test/fast5/ >/Users/michaelfoster/sequencing/dorado_test/called/dorado-lambda-output-fast.sam
> Creating basecall pipeline
> Loaded 132453 reads
> Reads basecalled: 132453
> Samples/s: 8.976293e+06
> Finished
difference between the fast .sam and HAC .sam is 9,220,712 bytes. very impressed with the fast performance.
The -r, -b- c
parameters are all set correctly in the latest release along with some performance optimisation.
Greetings! I am currently attempting to use Dorado on my Macbook Pro and running into the title'd issue. Computer Specs:
Model Name: MacBook Pro Model Identifier: MacBookPro18,3 Chip: Apple M1 Pro Total Number of Cores: 10 (8 performance and 2 efficiency) Memory: 16 GB System Firmware Version: 7459.121.3 OS Loader Version: 7459.121.3 Apple M1 Pro:
Chipset Model: Apple M1 Pro Type: GPU Bus: Built-In Total Number of Cores: 16 Vendor: Apple (0x106b) Metal Family: Supported, Metal GPUFamily Apple 7
cmake version:
and path of cmake
I have been attempting to run basecalling on a run I did last week. my first step was basecalling the barcode for the known lambdaphage control I added. I moved all of the fast5 files for the barcoded lambda control into a directory for testing and got the following error:
okay, just gonna rm -rf the entire dorado directory and do it from the start.
here is the entire process starting from git clone:
okay seems to have worked. it printed it couldn't find openssl but then used it anyway despite having everything installed ahead of time. I can list the steps taken to prep/download deps if needed.
ran the next command:
looks to have worked but only error is:
anyway, ran the tests and they worked fine.
okay cool. lets run the test binaries in cmake-build/bin/
that worked, the next one:
cool, that one worked as well.
now to download the models. will just download them to the bin folder as it downloads to . as default.
cool, lets basecall the lambda barcode now.
same error. Where do I go next for troubleshooting? Is it an issue with running it directly from the cmake-build/bin folder? Honestly I have no idea what the issue is but hopefully I've provided enough information to assist in troubleshooting.
Thanks again. Looking forward to being able to do entire sequencing runs on my macbook pro without having to ssh into the cluster.