Alignment of a large genome in 2bit

[ohdongha]

Hello again, Bogdan and Michael,

So far, make_lastz_chains has been working great, and the largest successfully aligned was the wheat genome (14.6 Gb). This time, I was asked to see if the lungfish genome (~40Gb) could be aligned.

First, I had this error (I modified all '.' in the fasta header to '__'):

Error in faToTwoBit, index overflow at NC_056733__1. The 2bit format does not support indexes larger than 4Gb, 
please split up into smaller files, or use -long option.

So I converted the lungfish fasta to 2bit using faToTwoBit -long and supplied it to the make_lastz_chains pipeline. But there seems to be another issue.

The pipeline uses twobitreader.TwoBitFile in many places, e.g., checking if the input is 2bit, checking the sequence headers, and creating chrom.sizes files. But this function raises an error with a 2bit file created with faToTwoBit -long:

Python 3.9.18
>>> from twobitreader import TwoBitFile
>>> two_bit_reader = TwoBitFile("large_genome_long.2bit")
...
twobitreader.TwoBitFileError: Invalid 2-bit file. File version in header should be 0.

The USCS 2bit page says the version of 2bit should be "zero for now" but the UCSC kent utils do have a spec to set 'version = 1' for the "long" version to support large genomes.

So I guess the issue is if the twobitreader.TwoBitFile or any other 2bit reader can recognize 2bit files created with the --long option. I googled and found this source code. But I am unsure if this is it, and also the code is hard to understand for me.

Have you looked into this before?

[MichaelHiller]

Thanks for reporting this. Sounds like it is related to the -long flag and new 64 bit index.

Could you confirm that this problem only exists for 64bit 2bit files? e.g. if you make a mini-lungfish genome of say 500 Mb (just one scaffold). Does this run fine?

And do we know that twobitreader has been updated to work with 64bit 2bit files?