Closed bede closed 8 months ago
Rolling back to bowtie2-2.4.4-linux-x86_64 solved the issue for me. More recent versions failed to use the number of specified threads. On a machine with 224 CPUs, only 2 were being used using the latest version.
Thank you @snayfach, rolling back to 2.4.4 also worked for me. Great news.
Hello all,
Thank you for your patience. My initial hunch is that this issue was introduced in 2.5.0 with the changes async changes that we made to input reading an writing. @bede, since you are able to reproduce this issue would you be willing to test v2.4.5 and v2.5.0?
Thanks @ch4rr0, I tested v2.4.5 and v2.5.0 as you suggested on a 32 core machine specifying 32 threads. 2.4.5 consistently used ~3200% CPU (reported by top) as expected.
However 2.5.0 was erratic, jumping between 400% and 3200%. I haven't compared 2.5.0 and 2.5.1, but it is clear to me that this issue began with 2.5.0.
@bede, I pushed a change-set to the bug_fixes
branch. Would be willing to test whether it has any effect on thread behavior?
@snayfach -- would you be willing to test since I have not heard back from bede yet?
Sorry for delay @ch4rr0, I just got time to test
I'm afraid the same behaviour remains with the bug_fixes
branch in my testing. 800-1600% CPU usage with 32 physical cores, whereas 2.4.5 gives >3100%
$ git status
On branch bug_fixes
Your branch is up to date with 'origin/bug_fixes'.
nothing to commit, working tree clean
$ bowtie2 --version
/data16/bowtie2-bug_fixes_2023-08-02/bowtie2-align-s version 2.5.1
64-bit
Built on pikachu
Wed Aug 2 13:27:40 UTC 2023
Compiler: gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
Options: -O3 -msse2 -funroll-loops -g3 -std=c++11 -DPOPCNT_CAPABILITY -DNO_SPINLOCK -DWITH_QUEUELOCK=1
Sizeof {int, long, long long, void*, size_t, off_t}: {4, 8, 8, 8, 8, 8}
I have been able to recreate this issue and indeed my latest push does not resolve it. I have found the cause and I am currently working on a solution. In the meantime increasing the --reads-per-batch
seems to keep the threads busy for longer and decreasing contention on the producer. The current default is 16, increasing that number to 1024 or 2048 seemed like a sweet spot on my hardware.
Thank you all for your patience.
I committed a few changes to bug_fixes that, in my testing, seem to resolve the issue. Would you be willing to test these changes?
Thanks @ch4rr0, your changes in 6f6458c5dae6ef8931c6024b1a20b5c2e275607d have resolved the issue for my test case. Looks good to me!
I'm reasonably satisfied that this issue has been fixed in 2.5.2, thanks! 🙏
Firstly, thank you for developing and maintaining Bowtie2! I noticed that 2.5.1 performs strangely when varying the
--threads
parameter. Beyond a certain number of threads, execution time increases considerably and CPU utilisation decreases. I've observed this with multiple read datasets using both an x86_64 Ubuntu VM and my arm64 MacOS machine, both using the appropriate GitHub release binaries. The behaviour is reproducible on any one machine, although a given read dataset will not necessarily trigger the problem on both my laptop and the VM.Here I used ~20m paired mixed bacterial and human reads with an index built from the human T2T reference + HLA sequences.
VM info
Ubuntu 22.04 LTS