I'm Fatemeh, @rob-p's student.
This pull request is regarding the conversation we had about outputting a cdbg with segments having overlap of k-1 instead of k. The series of commits in this pull request handle this case and can be called by the same set of inputs for graphdump with a new type as pufferized.
All the changes have been pushed to a separate branch pufferize.
It also takes care of cases that were in general problematic:
1) palindrome cases in which you have a sequence that is equal to its reverse-complement. In a palindrome case, we will have repeated kmers in the sequence other than the overlapping kmers at the two ends of the sequence. This specific case is important for us because our index builder relies on this specific property of cdbgs that each kmer appears at most once in all the contigs (In twopaco's output, if it is not a boundary kmer). This is specifically taken care of for the pufferized type of output. You can apply the change to other output types as well if you think this case should generally be taken care of.
2) graph-dump uses a fixed space size for different inputs that can cause unnecessary overhead for small input files. In this version, the memory consumption can be restricted based on actual number of junctions which is determined in previous step, graphconstruction. In the current implementation, we go one extra pass over the binary input file to determine the number of junctions, but this step can be simply discarded with a minor change in graphconstructor step to store total number of junctions on the disk so that graphdump can read the input in and resize the data structures accordingly.
3) There is a special case in segment_id_generator that will create different IDs (in absolute) for a segment and its reverse complement that causes to output two different contigs for the segment and its reverse-complement. This also has been resolved by just adding one extra condition to the list of conditions. (I will post it as an issue, so that it's not lost in the list of all the other changes here)
Dear @ilyaminkin and @pashadag ,
I'm Fatemeh, @rob-p's student. This pull request is regarding the conversation we had about outputting a cdbg with segments having overlap of k-1 instead of k. The series of commits in this pull request handle this case and can be called by the same set of inputs for graphdump with a new type as
pufferized. All the changes have been pushed to a separate branchpufferize.It also takes care of cases that were in general problematic: 1) palindrome cases in which you have a sequence that is equal to its reverse-complement. In a palindrome case, we will have repeated kmers in the sequence other than the overlapping kmers at the two ends of the sequence. This specific case is important for us because our index builder relies on this specific property of cdbgs that each kmer appears at most once in all the contigs (In twopaco's output, if it is not a boundary kmer). This is specifically taken care of for the
pufferizedtype of output. You can apply the change to other output types as well if you think this case should generally be taken care of.2) graph-dump uses a fixed space size for different inputs that can cause unnecessary overhead for small input files. In this version, the memory consumption can be restricted based on actual number of junctions which is determined in previous step, graphconstruction. In the current implementation, we go one extra pass over the binary input file to determine the number of junctions, but this step can be simply discarded with a minor change in graphconstructor step to store total number of junctions on the disk so that graphdump can read the input in and resize the data structures accordingly.
3) There is a special case in segment_id_generator that will create different IDs (in absolute) for a segment and its reverse complement that causes to output two different contigs for the segment and its reverse-complement. This also has been resolved by just adding one extra condition to the list of conditions. (I will post it as an issue, so that it's not lost in the list of all the other changes here)
Best, Fatemeh