search

nservant / HiC-Pro

HiC-Pro: An optimized and flexible pipeline for Hi-C data processing
Other
382 stars 183 forks source link

ice_norm Error 126 #408

Open zhanwen-cheng opened 3 years ago

zhanwen-cheng commented 3 years ago

Hello, still some ice normalization issues. I have installed your latest HiC-Por(3.0.0) and run hicpro with my test data, the log file was like below:

(HiC-Pro_v3.0.0) chengzw@profxia-SUSTC-309:~/software/hic/test$ time ../HiC-Pro/bin/HiC-Pro -c config-hicpro.txt -i example -o result
/home/chengzw/software/hic/test/result folder alreads exists. Do you want to overwrite it ? (y/n) [n] :
y

Run HiC-Pro 3.0.0
--------------------------------------------
Mon Feb 22 21:09:53 CST 2021
Bowtie2 alignment step1 ...
Logs: logs/SRR824846.2/mapping_step1.log

--------------------------------------------
Mon Feb 22 23:54:23 CST 2021
Bowtie2 alignment step2 ...
Logs: logs/SRR824846.2/mapping_step2.log

--------------------------------------------
Tue Feb 23 00:24:06 CST 2021
Combine R1/R2 alignment files ...
Logs: logs/SRR824846.2/mapping_combine.log

--------------------------------------------
Tue Feb 23 00:26:03 CST 2021
Mapping statistics for R1 and R2 tags ...
Logs: logs/SRR824846.2/mapping_stats.log

--------------------------------------------
Tue Feb 23 00:27:02 CST 2021
Pairing of R1 and R2 tags ...
Logs: logs/SRR824846.2/mergeSAM.log

--------------------------------------------
Tue Feb 23 00:39:48 CST 2021
Assign alignments to restriction fragments ...
Logs: logs/SRR824846.2/mapped_2hic_fragments.log

--------------------------------------------
Tue Feb 23 01:11:23 CST 2021
Merge chunks from the same sample ...
Logs: logs/SRR824846.2/merge_valid_interactions.log

--------------------------------------------
Tue Feb 23 01:12:50 CST 2021
Merge stat files per sample ...
Logs: logs/SRR824846.2/merge_stats.log

--------------------------------------------
Tue Feb 23 01:12:51 CST 2021
Run quality checks for all samples ...
Logs: logs/SRR824846.2/make_Rplots.log

--------------------------------------------
Tue Feb 23 01:13:05 CST 2021
Generate binned matrix files ...
Logs: logs/SRR824846.2/build_raw_maps.log

--------------------------------------------
Tue Feb 23 01:13:56 CST 2021
Run ICE Normalization ...
Logs: logs/SRR824846.2/ice_10000.log
make: *** [/home/chengzw/software/hic/HiC-Pro/bin/../scripts//Makefile:194: ice_norm] Error 126

real    244m10.573s
user    1161m5.442s
sys     3666m16.041s

It seemed that ice_norm failed. I checked ice_10000.log file, it showed

ice --results_filename hic_results/matrix/SRR824846.2/iced/10000/SRR824846.2_10000_iced.matrix --filter_low_counts_perc 0.02 --filter_high_counts_perc 0 --max_iter 100 --eps 0.1 --remove-all-zeros-loci --output-bias 1 hic_results/matrix//SRR824846.2/raw/10000/SRR824846.2_10000.matrix
/home/chengzw/software/hic/HiC-Pro/scripts/hic.inc.sh: /home/chengzw/.local/bin/ice: /home/chengzw/anaconda3/envs/hic/bin/python: bad interpreter: No such file or directory

and ran the command 

ice --results_filename hic_results/matrix/SRR824846.2/iced/10000/SRR824846.2_10000_iced.matrix --filter_low_counts_perc 0.02 --filter_high_counts_perc 0 --max_iter 100 --eps 0.1 --remove-all-zeros-loci --output-bias 1 hic_results/matrix//SRR824846.2/raw/10000/SRR824846.2_10000.matrix

it showed 

-bash: /home/chengzw/.local/bin/ice: /home/chengzw/anaconda3/envs/hic/bin/python: bad interpreter: No such file or directory

Actually I noticed that in your scripts/Makefile line 194, the program used the ice under scripts folder

189 ## Apply ICE normalization
    190 ice_norm:
    191         @echo "--------------------------------------------"
    192         @date
    193         @echo "Run ICE Normalization ..."
    194         @$(SCRIPTS)/ice_norm.sh -c $(CONFIG_FILE)

and I test the ice_norm.sh under scripts, it could not be called.

(HiC-Pro_v3.0.0) chengzw@profxia-SUSTC-309:~/software/hic/test$ ~/software/hic/HiC-Pro/scripts/ice_norm.sh -h
/home/chengzw/software/hic/HiC-Pro/scripts/ice_norm.sh: line 23: usage: command not found
Exit: Error in input type.'.fastq|.fq|.bam|.validPairs|.allValidPairs|.matrix' files are expected.
(HiC-Pro_v3.0.0) chengzw@profxia-SUSTC-309:~/software/hic/test$ ~/software/hic/HiC-Pro/scripts/ice_norm.sh --help
/home/chengzw/software/hic/HiC-Pro/scripts/ice_norm.sh: error - unrecognized option --help
[config-hicpro.txt](https://github.com/nservant/HiC-Pro/files/6026300/config-hicpro.txt)

However, the ice under the scripts could.

(HiC-Pro_v3.0.0) chengzw@profxia-SUSTC-309:~/software/hic/test$ ~/software/hic/HiC-Pro/scripts/ice -h
/home/chengzw/anaconda3/envs/HiC-Pro_v3.0.0/lib/python3.7/site-packages/iced/normalization/_ca_utils.py:9: UserWarning: The API of this module is likely to change. Use only for testing purposes
  "The API of this module is likely to change. "
usage: ICE normalization [-h] [--results_filename RESULTS_FILENAME]
                         [--filtering_perc FILTERING_PERC]
                         [--filter_low_counts_perc FILTER_LOW_COUNTS_PERC]
                         [--filter_high_counts_perc FILTER_HIGH_COUNTS_PERC]
                         [--remove-all-zeros-loci] [--max_iter MAX_ITER]
                         [--eps EPS] [--dense] [--output-bias OUTPUT_BIAS]
                         [--verbose VERBOSE]
                         File to load

positional arguments:
  File to load          Path to file of contact counts to load

optional arguments:
  -h, --help            show this help message and exit
  --results_filename RESULTS_FILENAME, -r RESULTS_FILENAME
                        results_filename
  --filtering_perc FILTERING_PERC, -f FILTERING_PERC
                        Percentage of reads to filter out
  --filter_low_counts_perc FILTER_LOW_COUNTS_PERC
                        Percentage of reads to filter out
  --filter_high_counts_perc FILTER_HIGH_COUNTS_PERC
                        Percentage of reads to filter out
  --remove-all-zeros-loci
                        If provided, all non-interacting loci will be removed
                        prior to the filtering strategy chosen.
  --max_iter MAX_ITER, -m MAX_ITER
                        Maximum number of iterations
  --eps EPS, -e EPS     Precision
  --dense, -d
  --output-bias OUTPUT_BIAS, -b OUTPUT_BIAS
                        Output the bias vector
  --verbose VERBOSE, -v VERBOSE

The attached is my config-hicpro.txt file. Could you help me to fix this issue? Thanks.

mdozmorov commented 3 years ago

I encountered exactly the same error at the second step of the HiC-Pro run. Tried editing shebang in ~/.local/bin/ice, it didn't help. The following helped, but it is the unstable solution:

  1. cp /home/user/.local/HiC-Pro_3.0.0/scripts/ice /home/user/.local/bin/
  2. Ran the second step, ice_norm only, in command line. Exactly the same command as in the cluster submission script, within the activated environment. It runs from the command line but fails when the job is submitted to the PBS cluster. 
    make --file /home/user/.local/HiC-Pro_3.0.0/scripts/Makefile CONFIG_FILE=/home/sequencing/config_hicpro_hg38.txt CONFIG_SYS=/home/user/.local/HiC-Pro_3.0.0/config-system.txt ice_norm 2>&1