Error in make_viewpoints.py

[mmilevskiy]

Hi Nicolas @nservant ,

I am having trouble generating the viewpoints data. I am just using a simple bed file to test it out and am coming back with a syntax error. I have tried to fun the script without a restriction fragment (DNase-HiC) and then with the _abs.bed file as the restriction fragments and no luck. Any help is much appreciated.

Michael

The bed file is attached. CapC_Promoter_Oligos_ReF.bed.gz

vc7-shared 558 % HiC-Pro-master/bin/utils/make_viewpoints.py -i pro_merged/hic_results/data/sample1/sample1.allValidPairs -f pro_maps/hic_results/matrix/sample1/raw/20000/sample1_20000_abs.bed -t CapC_Promoter_Oligos_ReF.bed -c -e 1000 -d -v > CapC_Promoter_Oligos.bedgraph File "HiC-Pro-master/bin/utils/make_viewpoints.py", line 68 print("## Loading BED file {} ...".format(in_file), file=sys.stderr) ^ SyntaxError: invalid syntax

[nservant]

Hi, Your bed looks ok. Are you using the last HiC-pro version (3.0.0) ? and are you sure that python3 is used in your environment ? Best

[mmilevskiy]

Hi Nicolas,

No, using 2.11.4 but using Python 3.7.6. I can see if IT can put an updated version on the server if you think its a HiC-Pro version error?

Michael

[nservant]

Hi Michael Yes the version 3.0.0 fixes a few bugs in make_viewpoints. Note that this is just an utility, meaning that you can just download the script and use it by yourself. Best

[mmilevskiy]

Hi Nicolas,

Ok tried that and with python 3.7.6:

vc7-shared 515 % HiC-Pro-master/HiC-Pro-master_3.0.0/bin/utils/make_viewpoints.py -i pro_merged/hic_results/data/sample1/sample1.allValidPairs -f pro_maps/hic_results/matrix/sample1/raw/20000/sample1_20000_abs.bed -t CapC_Promoter_Oligos_ReF.bed -c -e 1000 -d -v > CapC_Promoters_Oligos.bedgraph Traceback (most recent call last): File "HiC-Pro-master/HiC-Pro-master_3.0.0/bin/utils/make_viewpoints.py", line 19, in from bx.intervals.intersection import Intersecter, Interval ModuleNotFoundError: No module named 'bx'

What do you think? Before I was loading the HiC-Pro singularity, here just the python module. I assume that something is missing from the import?

[nservant]

You need to install the bx python module in your python 3.7.6

[mmilevskiy]

Hi Nicolas,

So have gotten things going, but am getting this error when I use the -e option. I removed the hashes to make it easier to read.

vc7-shared 512 % singularity exec /stornext/System/data/apps/HiC-Pro/HiC-Pro-3.0.0/HiC-Pro.sif make_viewpoints.py -i pro_merged/hic_results/data/sample1/sample1.allValidPairs -f pro_maps/hic_results/matrix/sample1/raw/20000/sample1_20000_abs.bed -t CapC_Promoter_Oligos_ReF.bed -c -e 20000 -v > CapC_Promoters_Oligos.bedgraph make_viewpoints.py validPairsFile=pro_merged/hic_results/data/sample1/sample1.allValidPairs fragmentFile=pro_maps/hic_results/matrix/sample1/raw/20000/sample1_20000_abs.bed targetFile=CapC_Promoter_Oligos_ReF.bed cisOnly=True exclusionSize=20000 verbose=True Loading data ... Loading BED file CapC_Promoter_Oligos_ReF.bed ... Traceback (most recent call last): File "/opt/HiC-Pro/bin/utils/make_viewpoints.py", line 201, in (target, exclu) = load_BED(targetFile, exclusionSize=exclusionSize, verbose=verbose) File "/opt/HiC-Pro/bin/utils/make_viewpoints.py", line 104, in load_BED if exclusionSize > 0: TypeError: '>' not supported between instances of 'str' and 'int'

Then when I remove the -e option, things get going but the output file is empty.

vc7-shared 515 % singularity exec /stornext/System/data/apps/HiC-Pro/HiC-Pro-3.0.0/HiC-Pro.sif make_viewpoints.py -i pro_merged/hic_results/data/sample1/sample1.allValidPairs -f pro_maps/hic_results/matrix/sample1/raw/20000/sample1_20000_abs.bed -t CapC_Promoter_Oligos_ReF.bed -c -v make_viewpoints.py validPairsFile=pro_merged/hic_results/data/sample1/sample1.allValidPairs fragmentFile=pro_maps/hic_results/matrix/sample1/raw/20000/sample1_20000_abs.bed targetFile=CapC_Promoter_Oligos_ReF.bed cisOnly=True exclusionSize=0 verbose=True Loading data ... Loading BED file CapC_Promoter_Oligos_ReF.bed ... Loading BED file pro_maps/hic_results/matrix/sample1/raw/20000/sample1_20000_abs.bed ... WARNINGS for all excluded chromosomes.... Opening file pro_merged/hic_results/data/sample1/sample1.allValidPairs ... Writing in Snai2_viewpoint.bedgraph [Snai2] CAP-REP read pairs = 437 [Snai2] CAP-CAP read pairs = 159 [Snai2] Excluded reads = 0 REP-REP read pairs = 1636396 UA reads = 65296806

File: track type=bedGraph name='hicpro Snai2' description='hicpro Snai2' visibility=full color=200,100,0 altColor=0,100,200 priority=20

Michael

UPDATE: Just fixed the -e option and the output files are still empty.

[mmilevskiy]

With the updated files, this is what I am now getting. The output files are still empty, except for the file headers.

singularity exec /stornext/System/data/apps/HiC-Pro/HiC-Pro-3.0.0/HiC-Pro.sif make_viewpoints.py -i pro_merged/hic_results/data/sample1/sample1.allValidPairs -f pro_maps/hic_results/matrix/sample1/raw/20000/sample1_20000_abs.bed -t CapC_Promoter_Oligos_ReF.bed -c -e 10000 -v -o CapC_Promoters_Oligos.bedgraph make_viewpoints.py validPairsFile=pro_merged/hic_results/data/sample1/sample1.allValidPairs fragmentFile=pro_maps/hic_results/matrix/sample1/raw/20000/sample1_20000_abs.bed targetFile=CapC_Promoter_Oligos_ReF.bed cisOnly=True exclusionSize=10000 verbose=True Loading data ... Loading BED file CapC_Promoter_Oligos_ReF.bed ... Loading BED file pro_maps/hic_results/matrix/sample1/raw/20000/sample1_20000_abs.bed ... Warning : Restrict to cis interactions - chr19 skipped Warning : Restrict to cis interactions - chrUn_GL456381 skipped Warning : Restrict to cis interactions - chrUn_GL456359 skipped Warning : Restrict to cis interactions - chr3 skipped Warning : Restrict to cis interactions - chrUn_GL456396 skipped Warning : Restrict to cis interactions - chr2 skipped Warning : Restrict to cis interactions - chr4_JH584292_random skipped Warning : Restrict to cis interactions - chrUn_GL456360 skipped Warning : Restrict to cis interactions - chr4_JH584293_random skipped Warning : Restrict to cis interactions - chr18 skipped Warning : Restrict to cis interactions - chr4_GL456350_random skipped Warning : Restrict to cis interactions - chr1_GL456213_random skipped Warning : Restrict to cis interactions - chrUn_GL456382 skipped Warning : Restrict to cis interactions - chr4_JH584294_random skipped Warning : Restrict to cis interactions - chr1_GL456212_random skipped Warning : Restrict to cis interactions - chr4_JH584295_random skipped Warning : Restrict to cis interactions - chr1_GL456211_random skipped Warning : Restrict to cis interactions - chrUn_GL456394 skipped Warning : Restrict to cis interactions - chr1 skipped Warning : Restrict to cis interactions - chrUn_GL456383 skipped Warning : Restrict to cis interactions - chr1_GL456210_random skipped Warning : Restrict to cis interactions - chrUn_GL456239 skipped Warning : Restrict to cis interactions - chr5 skipped Warning : Restrict to cis interactions - chrUn_GL456367 skipped Warning : Restrict to cis interactions - chrUn_GL456390 skipped Warning : Restrict to cis interactions - chrUn_GL456387 skipped Warning : Restrict to cis interactions - chrUn_GL456370 skipped Warning : Restrict to cis interactions - chr4 skipped Warning : Restrict to cis interactions - chrUn_GL456366 skipped Warning : Restrict to cis interactions - chr4_GL456216_random skipped Warning : Restrict to cis interactions - chrUn_GL456393 skipped Warning : Restrict to cis interactions - chr6 skipped Warning : Restrict to cis interactions - chrUn_GL456372 skipped Warning : Restrict to cis interactions - chrUn_GL456385 skipped Warning : Restrict to cis interactions - chrUn_GL456392 skipped Warning : Restrict to cis interactions - chr7 skipped Warning : Restrict to cis interactions - chr14 skipped Warning : Restrict to cis interactions - chr1_GL456221_random skipped Warning : Restrict to cis interactions - chr15 skipped Warning : Restrict to cis interactions - chrUn_GL456378 skipped Warning : Restrict to cis interactions - chrY_JH584301_random skipped Warning : Restrict to cis interactions - chr17 skipped Warning : Restrict to cis interactions - chr5_JH584298_random skipped Warning : Restrict to cis interactions - chr5_JH584299_random skipped Warning : Restrict to cis interactions - chr7_GL456219_random skipped Warning : Restrict to cis interactions - chrUn_GL456379 skipped Warning : Restrict to cis interactions - chr8 skipped Warning : Restrict to cis interactions - chr12 skipped Warning : Restrict to cis interactions - chrX_GL456233_random skipped Warning : Restrict to cis interactions - chr5_GL456354_random skipped Warning : Restrict to cis interactions - chr5_JH584297_random skipped Warning : Restrict to cis interactions - chr13 skipped Warning : Restrict to cis interactions - chrM skipped Warning : Restrict to cis interactions - chr9 skipped Warning : Restrict to cis interactions - chr10 skipped Warning : Restrict to cis interactions - chr5_JH584296_random skipped Warning : Restrict to cis interactions - chrY skipped Warning : Restrict to cis interactions - chrUn_GL456389 skipped Warning : Restrict to cis interactions - chr11 skipped Warning : Restrict to cis interactions - chrUn_GL456368 skipped Warning : Restrict to cis interactions - chrUn_JH584304 skipped Warning : Restrict to cis interactions - chrX skipped Warning : Restrict to cis interactions - chrY_JH584300_random skipped Warning : Restrict to cis interactions - chrY_JH584302_random skipped Warning : Restrict to cis interactions - chrY_JH584303_random skipped Opening file pro_merged/hic_results/data/sample1/sample1.allValidPairs ... Writing in Snai2.4_up_viewpoint.bedgraph [Snai2.4_up] CAP-REP read pairs = 0 [Snai2.4_up] CAP-CAP read pairs = 0 [Snai2.4_up] Excluded reads = 190 Writing in Snai2.4_viewpoint.bedgraph [Snai2.4] CAP-REP read pairs = 1185 [Snai2.4] CAP-CAP read pairs = 506 [Snai2.4] Excluded reads = 0 Writing in Snai2.4_dwn_viewpoint.bedgraph [Snai2.4_dwn] CAP-REP read pairs = 0 [Snai2.4_dwn] CAP-CAP read pairs = 0 [Snai2.4_dwn] Excluded reads = 457 REP-REP read pairs = 3055501 UA reads = 122463112

[nservant]

Sorry for the delay. Still in trouble ?

[mmilevskiy]

Hi Nicolas,

Yep, not luck figuring this out. I went through the code and couldn't find anything obvious as to why it wont write the file.

[nservant]

You should have some interaction at list for your Snai2.4 viewpoint ? For the two others, indeed, it does not find any interaction overlapping these regions. Would you agree to share with me your input files please ? So that I can try to reproduce it myself ?

[mmilevskiy]

Hi Nicolas,

Yes, it does appear to find interaction, but the output files are completely empty. I could send one file over to checkout, do you have an easy transfer system for large files?

[nservant]

would you have a small file to start with ;) ?

[mmilevskiy]

I've got fastq at 2.5gb for read 1 and the same for read 2 or the allValidPairs is at 5.8gb.