search

HadoopGenomics / Hadoop-BAM

Hadoop-BAM is a Java library for the manipulation of files in common bioinformatics formats using the Hadoop MapReduce framework
MIT License
69 stars 52 forks source link

FastaInputFormat splits file into too many partitions #207

Open PatrycjaKarbownik opened 4 years ago

PatrycjaKarbownik commented 4 years ago

I used the FastaInputFormat to read fasta file in Spark. After code analysis, I see that getSplits method is overrided. Okey, but number of returned partitions equals number of fasta records. E.g: fasta file has 661 bytes (13 fasta records) and my mapred.max.split.size conf is set to 500 bytes. I expect to get 2 partitions, but I get 13 partitions. What is purpose for that extra splitting? Why getSplits method from FileInputFormat couldn't be used?

Moreover FastqInputFormat uses only getSplits method from FileInputFormat and it's behaviour is expected by me.