Memory allocation failure during mremap call for MemoryMapped::Vector.

[c2997108]

Hi, I am trying to run Shasta on Rocky Linux 9. I have tried using Shasta from version 0.8 - 0.12 and none of them work with the following message. "Memory allocation failure during mremap call for MemoryMapped::Vector."

I wondered if my setup was correct because the input file is small and it worked when I ran 0.8 on CentOS7 before, so I tried to assemble it using the file r94_ec_rad2.181119.60x-10kb.fasta.gz from the following demonstration page. https://paoloshasta.github.io/shasta/QuickStart.html

However, I got an error message “The specified bucket does not exist” and could not download the file. Is there somewhere I can download it?

Thank you.

[paoloshasta]

Sorry about the demonstration file - I did not own its location and the owner must have decided that it was time to declare it obsolete. I should modify the quick start information to reflect that.

For your current purpose, you can download this very small test input dataset from the Shasta GitHub repository: https://github.com/paoloshasta/shasta/blob/main/tests/TinyTest.fasta.gz. (You need to click the download button on the top right of that page).

This is a very small dataset that will not produce a meaningful assembly but should be sufficient to test your memory problems. Of course, you need to gunzip it before presenting it as input to Shasta, because Shasta does not accept compressed files as input.

If this still gives you the same error message, please provide the following information:

• The exact command line you used.
• The output of uname -a
• The output of ulimit -a
• The output of cat /proc/meminfo

[c2997108]

Thank you for the new test file. I tried it and got the same error message.

$ shasta-Linux-0.12.0 --input TinyTest.fasta --config Nanopore-May2022
Shasta Release 0.12.0
2024-May-31 23:42:05.062055 Assembly begins.
Command line:
shasta-Linux-0.12.0 --input TinyTest.fasta --config Nanopore-May2022
For options in use for this assembly, see shasta.conf in the assembly directory.
This run uses options "--memoryBacking 4K --memoryMode anonymous".
This could result in longer run time.
For faster assembly, use "--memoryBacking 2M --memoryMode filesystem"
(root privilege via sudo required).
Therefore the results of this run should not be used
for the purpose of benchmarking assembly time.
However the memory options don't affect assembly results in any way.
This assembly will use 40 threads.
Setting up consensus caller Bayesian:guppy-5.0.7-b
Using predefined Bayesian consensus caller guppy-5.0.7-b
Bayesian consensus caller configuration name is HG005_wg_guppy_5.0.7_with_10_pseudocount_4-6-2022
2024-May-31 23:42:05.082213 A runtime error occurred in thread 1:
Memory allocation failure  during mremap call for MemoryMapped::Vector.
This assembly requires more memory than available.
Rerun on a larger machine.

I show the result of the following commands. It would be appreciated if you could tell me the cause of the error.

$ uname -a
Linux m1024 5.14.0-284.25.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 2 14:53:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 4122664
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 65536
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 4122664
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

$ cat /proc/meminfo
MemTotal:       1055465996 kB
MemFree:         4214976 kB
MemAvailable:   1024055932 kB
Buffers:            8068 kB
Cached:         998777312 kB
SwapCached:        92076 kB
Active:         74859004 kB
Inactive:       937648336 kB
Active(anon):    1203364 kB
Inactive(anon): 13268988 kB
Active(file):   73655640 kB
Inactive(file): 924379348 kB
Unevictable:        6892 kB
Mlocked:               0 kB
SwapTotal:      367628284 kB
SwapFree:       360214012 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:             74172 kB
Writeback:             0 kB
AnonPages:      10681012 kB
Mapped:          2063276 kB
Shmem:            754572 kB
KReclaimable:   28040564 kB
Slab:           31678764 kB
SReclaimable:   28040564 kB
SUnreclaim:      3638200 kB
KernelStack:       55104 kB
PageTables:       143188 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    895361280 kB
Committed_AS:   52718916 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      834220 kB
VmallocChunk:          0 kB
Percpu:           265536 kB
HardwareCorrupted:     4 kB
AnonHugePages:   8437760 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:    20223152 kB
DirectMap2M:    277221376 kB
DirectMap1G:    776994816 kB

[paoloshasta]

Please also attach the contents of performance.log in the failing assembly directory. If that still does not reveal the cause of the problem I will create an instrumented version of the code that will print out more information.

[c2997108]

This is the performance.log file. I sincerely appreciate your support.

$ more ShastaRun/performance.log
2024-May-31 23:42:05.061921 Assembly begins.
2024-May-31 23:42:05.063736 Begin loading reads from 1 files.
2024-May-31 23:42:05.063763 Loading reads from /tmp/fu/TinyTest.fasta
File size: 891697 bytes.
Buffer allocate time: 0.00144798 s.
Read time: 0.00294244 s.
Read rate: 3.03047e+08 bytes/s.

[paoloshasta]

Please try running it again, adding the following to your command line:

--memoryMode filesystem --memoryBacking disk

These options cause the assembly to use memory mapped to files on disk rather than standard dynamically allocated memory. This is not something I recommend in general because it slows down the assembly, but it will help us diagnose.

[c2997108]

It seems to have worked well with that option.

$ shasta-Linux-0.12.0 --input TinyTest.fasta --config Nanopore-May2022 --memoryMode filesystem --memoryBacking disk
Shasta Release 0.12.0
2024-Jun-01 00:46:40.283183 Assembly begins.
Command line:
shasta-Linux-0.12.0 --input TinyTest.fasta --config Nanopore-May2022 --memoryMode filesystem --memoryBacking disk
For options in use for this assembly, see shasta.conf in the assembly directory.
This assembly will use 40 threads.
Setting up consensus caller Bayesian:guppy-5.0.7-b
Using predefined Bayesian consensus caller guppy-5.0.7-b
Bayesian consensus caller configuration name is HG005_wg_guppy_5.0.7_with_10_pseudocount_4-6-2022
Discarded read statistics for file /tmp/fu/TinyTest.fasta:
    Discarded 0 reads containing invalid bases for a total 0 valid bases.
    Discarded 0 reads shorter than 10000 bases for a total 0 bases.
    Discarded 0 reads containing repeat counts 256 or more for a total 0 bases.
Discarded read statistics for all input files:
    Discarded 0 reads containing invalid bases for a total 0 valid bases.
    Discarded 0 short reads for a total 0 bases.
    Discarded 0 reads containing repeat counts 256 or more for a total 0 bases.
Read statistics for reads that will be used in this assembly:
    Total number of reads is 20.
    Total number of raw bases is 891443.
    Average read length is 44572.2 bases.
    N50 for read length is 81086 bases.
Found 0 reads with duplicate names.
Discarded from the assembly 0 reads with duplicate names.
Flagged 0 reads as palindromic out of 20 total.
Palindromic fraction is 0
LowHash0 algorithm will use 2^16 = 65536 buckets.
Alignment candidates after lowhash iteration 0: high frequency 27, total 153, capacity 153.
Alignment candidates after lowhash iteration 1: high frequency 93, total 172, capacity 231.
Alignment candidates after lowhash iteration 2: high frequency 111, total 180, capacity 241.
Alignment candidates after lowhash iteration 3: high frequency 123, total 182, capacity 253.
Alignment candidates after lowhash iteration 4: high frequency 143, total 184, capacity 254.
Alignment candidates after lowhash iteration 5: high frequency 147, total 184, capacity 254.
Alignment candidates after lowhash iteration 6: high frequency 157, total 184, capacity 254.
Alignment candidates after lowhash iteration 7: high frequency 159, total 184, capacity 254.
Alignment candidates after lowhash iteration 8: high frequency 160, total 184, capacity 254.
Alignment candidates after lowhash iteration 9: high frequency 162, total 184, capacity 254.
Found 162 alignment candidates.
Average number of alignment candidates per oriented read is 8.1.
Number of alignment candidates before suppression is 162
Suppressed 0 alignment candidates.
Number of alignment candidates after suppression is 162
2024-Jun-01 00:46:40.653046 Alignment computation begins.
2024-Jun-01 00:46:40.741996 Alignment computation completed.
Found and stored 159 good alignments.
Automatically selected alignment criteria:
        minAlignedFraction:     0.355
        minAlignedMarkerCount:          105
        maxDrift:               13
        maxSkip:                32
        maxTrim:                33
Keeping 87 alignments of 159
Of 40 vertices in the read graph, 0 are within distance 6 of their reverse complement.
Found 0 strand jump regions.
Marked 0 read graph edges out of 174 total as cross-strand.
Flagged 0 reads as chimeric out of 20 total.
Chimera rate is 0
Automatically selected value of MarkerGraph.minCoverage is 4
Kept 7342 disjoint sets with coverage in the requested range.
Found 10 disjoint sets with more than one marker on a single oriented read or with less than 0 supporting oriented reads on each strand.
Found 14202 edges for 7332 vertices.
The marker graph has 7332 vertices and 14202 edges.
Maximum edge coverage is 15
Flagged as weak 0 edges with coverage 1 and marker skip greater than 100
Transitive reduction removed 6852 marker graph edges out of 14202 total.
The marker graph has 7332 vertices and 7350 strong edges.
Pruned 4 edges at prune iteration 0.
Pruned 4 edges at prune iteration 1.
Pruned 4 edges at prune iteration 2.
Pruned 4 edges at prune iteration 3.
Pruned 4 edges at prune iteration 4.
Pruned 4 edges at prune iteration 5.
The original marker graph had 7332 vertices and 14202 edges.
The number of surviving edges is 7326.
Begin simplifyMarkerGraph iteration 0 with maxLength = 10
Before iteration 0 part 1, the assembly graph has 44 vertices and 62 edges.
Before iteration 0 part 2, the assembly graph has 4 vertices and 2 edges.
Begin simplifyMarkerGraph iteration 1 with maxLength = 100
Before iteration 1 part 1, the assembly graph has 4 vertices and 2 edges.
Before iteration 1 part 2, the assembly graph has 4 vertices and 2 edges.
Begin simplifyMarkerGraph iteration 2 with maxLength = 1000
Before iteration 2 part 1, the assembly graph has 4 vertices and 2 edges.
Before iteration 2 part 2, the assembly graph has 4 vertices and 2 edges.
Begin simplifyMarkerGraph iteration 3 with maxLength = 10000
Before iteration 3 part 1, the assembly graph has 4 vertices and 2 edges.
Before iteration 3 part 2, the assembly graph has 4 vertices and 2 edges.
Begin simplifyMarkerGraph iteration 4 with maxLength = 100000
Before iteration 4 part 1, the assembly graph has 4 vertices and 2 edges.
Before iteration 4 part 2, the assembly graph has 4 vertices and 2 edges.
Removed 0 low coverage cross edges of the assembly graph and 0 corresponding marker graph edges.
Before detangling, the assembly graph has 4 vertices and 2 edges.
Found 0 tangles.
The detangled assembly graph has 4 vertices.
The detangled assembly graph has 2 edges.
Removed 0 low coverage cross edges of the assembly graph and 0 corresponding marker graph edges.
Using 40 threads.
Assembly begins for 2 edges of the assembly graph.
Assembled a total 71485 bases for 2 assembly graph edges of which 1 were assembled.
The assembly graph has 4 vertices and 2 edges of which 1 were assembled.
Total length of assembled sequence is 71485
N50 for assembly segments is 71485
Shasta Release 0.12.0
2024-Jun-01 00:46:41.379558 Assembly ends.

Here is performance.log

[paoloshasta]

This allows you at least to run small assemblies for now, and also larger assemblies with a performance penalty.

This behavior indicates that your system has some problems allocating memory. If you tell me the exact version of Rocky Linux you are using I can try and reproduce the problem to get a better idea of the cause.

[c2997108]

Thanks to your help, I am now able to start it. The OS version is as follows.

$ cat /etc/os-release
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"

[paoloshasta]

I was not able to successfully install this older version of Rocky Linux and therefore I will not be able to reproduce. Some web posts reference possible problems in the Linux mremap call, used by Shasta, in some Linux kernel versions. I suggest upgrading to a newer version of Rocky Linux (currently at 9.4).

[c2997108]

Thank you for your support. I updated the OS to Rocky Linux 9.4 and the error is gone!