search

agshumate / Liftoff

An accurate GFF3/GTF lift over pipeline
GNU General Public License v3.0
427 stars 52 forks source link

running liftoff on cluster #129

Closed johnathanlo closed 2 years ago

johnathanlo commented 2 years ago

Hi! I'm running older liftoff (the one that runs straight from the python file) and have encountered some issues trying to get separate executions to run in parallel on cluster computing services. for reference, the parallel commands are

python /global/liftoff/liftoff.py -t species/species.fna -r ref/ref.fna -db ref/ref.gff_db -o species/species.gff

i have used cluster commands to set maximum memory and disk space usage for each process, but I encounter this error:

[W::sam_hdr_create] Ignored @SQ line with missing SN: tag [E::sam_hrecs_update_hashes] Header includes @SQ line with no SN: tag [E::sam_parse1] failed to parse header [W::sam_read1] Parse error at line 169448 Traceback (most recent call last): File "/global/scratch/users/lojohnathan6/old_liftoff/liftoff.py", line 121, in main() File "/global/scratch/users/lojohnathan6/old_liftoff/liftoff.py", line 72, in main feature_db, parent_features, intermediate_features, children_features, parent_order = liftover_types.lift_original_annotation( File "/global/scratch/users/lojohnathan6/old_liftoff/liftover_types.py", line 15, in lift_original_annotation aligned_segments = align_features.align_features_to_target(ref_chroms, target_chroms, processes, target_fasta, File "/global/scratch/users/lojohnathan6/old_liftoff/align_features.py", line 29, in align_features_to_target aligned_segments = parse_alignment(file, parent_dict, children_dict, unmapped_features, search_type) File "/global/scratch/users/lojohnathan6/old_liftoff/align_features.py", line 95, in parse_alignment for ref_seq in sam_file_iter: File "pysam/libcalignmentfile.pyx", line 2187, in pysam.libcalignmentfile.IteratorRowAll.next OSError: truncated file

among others indicating that the process was killed due to memory use. I have tried to rectify this using all sorts of memory configurations, however, the issue remains. Also, it runs fine when running only one liftoff process at a time, regardless of memory configuration.

agshumate commented 2 years ago

hi please use the latest version of liftoff to fix this error