Issue Running make_lastz_chains-2.0.6 on Test Data

ZhaoHang-bio commented 1 year ago

Hello,

I encountered an issue when running the test data with the following command: make_chains.py target query test_data/test_reference.fa test_data/test_query.fa --pd test_out -f --chaining_memory 16

The error I received is as follows:

### Nextflow process chain_run finished successfully
An error occurred while executing chain_run: Error! No non-empty files found at /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/chain. The failed operation label is: chain_run

Traceback (most recent call last):
  File "/home/zhaohang/soft/make_lastz_chains-2.0.6/modules/step_manager.py", line 70, in execute_steps
    step_result = step_to_function[step](params, project_paths, step_executables)
  File "/home/zhaohang/soft/make_lastz_chains-2.0.6/modules/pipeline_steps.py", line 64, in chain_run_step
    do_chain_run(params, project_paths, executables)
  File "/home/zhaohang/soft/make_lastz_chains-2.0.6/steps_implementations/chain_run_step.py", line 113, in do_chain_run
    has_non_empty_file(project_paths.chain_output_dir, "chain_run")
  File "/home/zhaohang/soft/make_lastz_chains-2.0.6/modules/common.py", line 51, in has_non_empty_file
    raise PipelineFileNotFoundError(err_msg)
modules.error_classes.PipelineFileNotFoundError: Error! No non-empty files found at /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/chain. The failed operation label is: chain_run

Any assistance on this would be greatly appreciated.

Thank you!

kirilenkobm commented 1 year ago

Hi @ZhaoHang-bio

Thanks for the bug report.

pls provide the following files: test_out/run.log and test_out/temp_chain_run/chains_jobslist

ZhaoHang-bio commented 1 year ago

The content of run.log is as follows:

# Make Lastz Chains #
Version 2.0.6
Commit: unknown
Branch: unknown

* found run_lastz.py at /home/zhaohang/soft/make_lastz_chains-2.0.6/standalone_scripts/run_lastz.py
* found run_lastz_intermediate_layer.py at /home/zhaohang/soft/make_lastz_chains-2.0.6/standalone_scripts/run_lastz_intermediate_layer.py
* found chain_gap_filler.py at /home/zhaohang/soft/make_lastz_chains-2.0.6/standalone_scripts/chain_gap_filler.py
* found faToTwoBit at /home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/faToTwoBit
* found twoBitToFa at /home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/twoBitToFa
* found pslSortAcc at /home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/pslSortAcc
* found axtChain at /home/zhaohang/anaconda3/envs/TOGA/bin/axtChain
* found axtToPsl at /home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/axtToPsl
* found chainAntiRepeat at /home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/chainAntiRepeat
* found chainMergeSort at /home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/chainMergeSort
* found chainCleaner at /home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/chainCleaner
* found chainSort at /home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/chainSort
* found chainScore at /home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/chainScore
* found lastz at /home/zhaohang/soft/bin/lastz
* found chainFilter at /home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/chainFilter
found chainNet in /home/zhaohang/soft/make_lastz_chains-2.0.6/chain_clean_micro_env
All necessary executables found.
Making chains for test_data/test_reference.fa and test_data/test_query.fa files, saving results to /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out
Pipeline started at 2023-09-27 20:34:45.194169
* Setting up genome sequences for target
genomeID: target
input sequence file: test_data/test_reference.fa
is 2bit: False
planned genome dir location: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/target.2bit
Initial fasta file test_data/test_reference.fa saved to /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/target.2bit
For target (target) sequence file: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/target.2bit; chrom sizes saved to: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/target.chrom.sizes
* Setting up genome sequences for query
genomeID: query
input sequence file: test_data/test_query.fa
is 2bit: False
planned genome dir location: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/query.2bit
Initial fasta file test_data/test_query.fa saved to /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/query.2bit
For query (query) sequence file: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/query.2bit; chrom sizes saved to: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/query.chrom.sizes

### Partition Step ###

# Partitioning for target
Saving partitions and creating 1 buckets for lastz output
In particular, 0 partitions for bigger chromosomes
And 1 buckets for smaller scaffolds
Saving target partitions to: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/target_partitions.txt
# Partitioning for query
Saving partitions and creating 1 buckets for lastz output
In particular, 0 partitions for bigger chromosomes
And 1 buckets for smaller scaffolds
Saving query partitions to: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/query_partitions.txt
Num. target partitions: 0
Num. query partitions: 0
Num. lastz jobs: 0

### Lastz Alignment Step ###

LASTZ: making jobs
LASTZ: saved 1 jobs to /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_lastz_run/lastz_joblist.txt
Parallel manager: pushing job nextflow /home/zhaohang/soft/make_lastz_chains-2.0.6/parallelization/execute_joblist.nf --joblist /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_lastz_run/lastz_joblist.txt -c /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_lastz_run/lastz_config.nf

### Nextflow process lastz finished successfully
Found 1 output files from the LASTZ step
Please note that lastz_step.py does not produce output in case LASTZ could not find any alignment

### Concatenating Lastz Results (Cat) Step ###

Concatenating LASTZ output from 1 buckets
* concatenated bucket bucket_ref_bulk_1 to /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_concat_lastz_output/concat_0.psl.gz
Concatenated 1 files in total into 1 files

### Build Chains Step ###

Sorting PSL files, saving the results to /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/sorted_psl
/home/zhaohang/soft/make_lastz_chains-2.0.6/HL_kent_binaries/pslSortAcc nohead /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/sorted_psl /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_kent /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_concat_lastz_output/concat_0.psl.gz
Bundling psl files with the following arguments:
* input_dir: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/sorted_psl
* chrom_sizes: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/target.chrom.sizes
* output_dir: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/split_psl
* max_bases: 1000000
* warning_only: False
* verbose: False
Saving results to: /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/split_psl
Bundling 2 psl files in total
Written to /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/split_psl/bundle.0.psl
DONE. Produced 1 files
PSL bundle sub-step done
Building axtChain joblist for 1 bundled psl files
Saving 1 axtChain jobs to /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/chains_joblist
Parallel manager: pushing job nextflow /home/zhaohang/soft/make_lastz_chains-2.0.6/parallelization/execute_joblist.nf --joblist /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/chains_joblist -c /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/chain_run_config.nf

### Nextflow process chain_run finished successfully
An error occurred while executing chain_run: Error! No non-empty files found at /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/chain. The failed operation label is: chain_run

kirilenkobm commented 1 year ago

I have a few more questions:

does the /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_concat_lastz_output/concat_0.psl.gz contain anything?
please open /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/chains_joblist and execute the command written in this file - does it crash with error or not?

ZhaoHang-bio commented 1 year ago

Thank you for your assistance. When I executed the command in /home/zhaohang/soft/make_lastz_chains-2.0.6/test_out/temp_chain_run/chains_joblist, I found that the axtChain command failed. However, after making the necessary corrections, it is now running successfully. Once again, I appreciate your help.

kirilenkobm commented 1 year ago

@ZhaoHang-bio

Sorry, but I didn't understand - is your problem solved? (if yes - I'd close the issue) Please let me know what corrections did you do in order to make it work. Thanks!

ZhaoHang-bio commented 1 year ago

Yes, I've resolved the issue and it's now running successfully.

The prior failure stemmed from not checking if the UCSC Kent software was functioning properly. Upon running the commands in the chains_joblist file, as per your feedback, I identified the error message as axtChain failed to run, with the error: libssl.so.1.0.0: cannot open shared object file: No such file or directory. By redirecting libssl.so to libssl.so.1.0.0 in the lib path, the program now executes without issues.

Thank you again.

kirilenkobm commented 1 year ago

Nice to hear that it worked for you! Thank you for reporting the issue, I will add the respective note in the project's TODO.

hillerlab / make_lastz_chains

Issue Running make_lastz_chains-2.0.6 on Test Data #34