search

gmarcais / Jellyfish

A fast multi-threaded k-mer counter
Other
471 stars 136 forks source link

Can't merge hash with different size #50

Closed gusevfe closed 8 years ago

gusevfe commented 8 years ago

Hi!

I am trying to merge two jellyfish databases, but it exits with this message: Can't merge hash with different size (4294967296, 2147483648) Is there any way to work it out?

I am using v2.2.4.

Thank you in advance.

kes1smmn commented 8 years ago

You can recount the two libraries using the same hash size with the -s flag. (e.g -s 6G). Make sure you give it enough ram so jellyfish count does not automatically resize.

gmarcais commented 8 years ago

Adding to kes1smm answer, you can use the '--disk' switch to forbid Jellyfish from doing size doubling. This will guarantee you can merge the outputs, provided the '-s' flag was the same.

gusevfe commented 8 years ago

@gmarcais, @kes1smmn Thank you! Indeed, using these two switches fix the error.

phil-grayson commented 4 years ago

Could I suggest that a note about this be included in the "help" calls for count and/or merge? Due to my compute architecture and file structure, it seemed like it would be better for me to parallelize the count calls (computationally intensive) across my libraries and merge afterwards (very cheap). I tested this on a couple files and then ran it across my entire experiment. After running count across 120 files fastq with between 36 and 80 Gb of memory requested per job, I need to completely start again because I can't merge my files. That's a lot of wasted compute.