search

ay-lab / dcHiC

dcHiC: Differential compartment analysis for Hi-C datasets
MIT License
57 stars 10 forks source link

FileNotFoundError for chr1 but not others #21

Closed esebesty closed 3 years ago

esebesty commented 3 years ago

I'm trying to run dcHiC chromosome-by-chromosome (after removing chrY), and most of the chromosomes seem to be ok, all of the differential compartment, etc files are there, but it fails on chr1, with the following:

FileNotFoundError ``` [1] "Hierarchy" [[1]] [1] 5602 5602 5602 5602 5602 5602 [[2]] [1] 1 1 1 1 1 1 DEBUG:Blank space skipped. No worries. DEBUG:Blank space skipped. No worries. DEBUG:Blank space skipped. No worries. DEBUG:Blank space skipped. No worries. DEBUG:Blank space skipped. No worries. DEBUG:Blank space skipped. No worries. Positions Not Shared Across All Data Sets, Chromosome 1: None 5602 Rscript --vanilla /home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/Hier.R BalancedChrMatrix_exp_N2P.D0.txt BalancedChrMatrix_exp_N2P.D14.txt BalancedChrMatrix_exp_N2P.D7. R OUTPUT: Traceback (most recent call last): File "/home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/run.py", line 323, in dmfa_file = open(name, "r") FileNotFoundError: [Errno 2] No such file or directory: 'hmfa_chrRAW_N2P.D0_exp_1.txt' Traceback (most recent call last): File "/home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py", line 40, in with open(results.eigfile, "r") as file: FileNotFoundError: [Errno 2] No such file or directory: 'chr_1/hmfa_N2P.D0_exp_1.txt' Traceback (most recent call last): File "/home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py", line 40, in with open(results.eigfile, "r") as file: FileNotFoundError: [Errno 2] No such file or directory: 'chr_1/hmfa_N2P.D14_exp_2.txt' Traceback (most recent call last): File "/home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py", line 40, in with open(results.eigfile, "r") as file: FileNotFoundError: [Errno 2] No such file or directory: 'chr_1/hmfa_N2P.D7_exp_3.txt' Traceback (most recent call last): File "/home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py", line 40, in with open(results.eigfile, "r") as file: FileNotFoundError: [Errno 2] No such file or directory: 'chr_1/hmfa_NGN2.D0_exp_4.txt' Traceback (most recent call last): File "/home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py", line 40, in with open(results.eigfile, "r") as file: FileNotFoundError: [Errno 2] No such file or directory: 'chr_1/hmfa_NGN2.D14_exp_5.txt' Traceback (most recent call last): File "/home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py", line 40, in with open(results.eigfile, "r") as file: FileNotFoundError: [Errno 2] No such file or directory: 'chr_1/hmfa_NGN2.D7_exp_6.txt' ['1'] ['N2P.D0', 'N2P.D14', 'N2P.D7', 'NGN2.D0', 'NGN2.D14', 'NGN2.D7'] ['N2P.D0', 'N2P.D14', 'N2P.D7', 'NGN2.D0', 'NGN2.D14', 'NGN2.D7'] ################### Chromosome 1 Output ################### /home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic 6 6 python /home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/run.py -nExp 6 -chrNum 1 -res 40000 -numGroups 6 -grouping 1 -ncp 2 -group 1 -group 1 -group 1 -group 1 -group 1 -g python /home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py -eigfile chr_1/hmfa_N2P.D0_exp_1.txt -chr 1 -exp N2P.D0 python /home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py -eigfile chr_1/hmfa_N2P.D14_exp_2.txt -chr 1 -exp N2P.D14 python /home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py -eigfile chr_1/hmfa_N2P.D7_exp_3.txt -chr 1 -exp N2P.D7 python /home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py -eigfile chr_1/hmfa_NGN2.D0_exp_4.txt -chr 1 -exp NGN2.D0 python /home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py -eigfile chr_1/hmfa_NGN2.D14_exp_5.txt -chr 1 -exp NGN2.D14 python /home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/makeBedGraph.py -eigfile chr_1/hmfa_NGN2.D7_exp_6.txt -chr 1 -exp NGN2.D7 Traceback (most recent call last): File "/home/user2031/.conda/envs/hicpro/lib/python3.7/shutil.py", line 566, in move os.rename(src, real_dst) FileNotFoundError: [Errno 2] No such file or directory: 'pc_N2P.D0_exp_1.txt' -> 'pcFiles/pc_N2P.D0_exp_1.txt' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/user2031/work/repos/bcro/bit-bio/dcHiC/dchic/dchic.py", line 392, in shutil.move(pcaFileLocation, "pcFiles") File "/home/user2031/.conda/envs/hicpro/lib/python3.7/shutil.py", line 580, in move copy_function(src, real_dst) File "/home/user2031/.conda/envs/hicpro/lib/python3.7/shutil.py", line 266, in copy2 copyfile(src, dst, follow_symlinks=follow_symlinks) File "/home/user2031/.conda/envs/hicpro/lib/python3.7/shutil.py", line 120, in copyfile with open(src, 'rb') as fsrc: FileNotFoundError: [Errno 2] No such file or directory: 'pc_N2P.D0_exp_1.txt' ```

This is a 40k matrix from HiC-Pro. Is it possible that the R script failed due to some error (memory limitation, etc)?

ay-lab commented 3 years ago

Hi There,

This certainly seems like it is caused by a memory error, especially as a 40kb matrix is the pretty large. Can you run the R script (using the printed command) standalone? You could also try adding memory to the chr1 job.

esebesty commented 3 years ago

Looks like everything is ok on a larger memory node! Memory usage peaked around 37Gb, and this failed on a 32Gb node.