YaqiangCao / cLoops

Accurate and flexible loops calling tool for 3D genomic data.

https://yaqiangcao.github.io/cLoops/

MIT License

109 stars 17 forks source link

hicpro2bedpe bug #23

Closed KunFang93 closed 4 years ago

KunFang93 commented 4 years ago

Hi, Yaqiang

I found that hicpro2bedpe is not working properly as the code misunderstands the meaning of 11th and 12th col. They are not read length but the mapping quality. Please check the this website:https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY.

Best, Kun

YaqiangCao commented 4 years ago

Hi, Yaqiang

I found that hicpro2bedpe is not working properly as the code misunderstands the meaning of 11th and 12th col. They are not read length but the mapping quality. Please check the this website:https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY.

Best, Kun

Hi Kun, Thank you very much for the reminding. Sorry for the carelessness. I have fix the problem, please refer to https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe for updated code (line 17-18). Please let me kown if you have found other problems. Have a nice day. Thank you again. Best, Yaqiang

KunFang93 commented 4 years ago

Hi, Yaqiang

Thanks for your promptly reply and fixation!

Best, Kun

Sent from my iPhone

On Sep 14, 2020, at 10:00, YaqiangCao notifications@github.com wrote:

Hi, Yaqiang

I found that hicpro2bedpe is not working properly as the code misunderstands the meaning of 11th and 12th col. They are not read length but the mapping quality. Please check the this website:https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY.

Best, Kun

Hi Kun, Thank you very much for the reminding. Sorry for the carelessness. I have fix the problem, please refer to https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe for updated code (line 17-18). Please let me kown if you have found other problems. Have a nice day. Thank you again. Best, Yaqiang

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

KunFang93 commented 4 years ago

Hi, Yaqiang

Sorry for bothering but when I am trying to call hic loops of gsm1631184, I only get 2 loops. The hic data is processed by hicpro with the attached config file and then use the new hicpropairs2bedpe. I was wondering if there is any way to debug where goes wrong? Thanks!

Best, Kun

On Sep 14, 2020, at 10:00 AM, YaqiangCao notifications@github.com wrote:

Hi, Yaqiang

I found that hicpro2bedpe is not working properly as the code misunderstands the meaning of 11th and 12th col. They are not read length but the mapping quality. Please check the this website:https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY.

Best, Kun

Hi Kun, Thank you very much for the reminding. Sorry for the carelessness. I have fix the problem, please refer to https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe for updated code (line 17-18). Please let me kown if you have found other problems. Have a nice day. Thank you again. Best, Yaqiang

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692115131, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXLIA3HMMOTIFWXWLMYCWLSFYVZFANCNFSM4RLVVTPA.

Please change the variable settings below if necessary

#########################################################################

Paths and Settings - Do not edit !

#########################################################################

TMP_DIR = tmp LOGS_DIR = log BOWTIE2_OUTPUT_DIR = bowtie_results MAPC_OUTPUT = hic_results RAW_DIR = rawdata

#######################################################################

SYSTEM AND SCHEDULER - Start Editing Here !!

####################################################################### N_CPU = 25 LOGFILE = hicpro.log

JOB_NAME = Jingwei_HiC JOB_MEM = 20gb JOB_WALLTIME = 600:00:00 JOB_QUEUE = JOB_MAIL = fangk@livemail.uthscsa.edu

#########################################################################

Data

#########################################################################

PAIR1_EXT = _R1 PAIR2_EXT = _R2

#######################################################################

Alignment options

#######################################################################

MIN_MAPQ = 10

BOWTIE2_IDX_PATH = /data/kun/Softwares/hiclib/hiclib-legacy/bin/bowtie2/index BOWTIE2_GLOBAL_OPTIONS = --very-sensitive -L 30 --score-min L,-0.6,-0.2 --end-to-end --reorder BOWTIE2_LOCAL_OPTIONS = --very-sensitive -L 20 --score-min L,-0.6,-0.2 --end-to-end --reorder

#######################################################################

Annotation files

#######################################################################

REFERENCE_GENOME = hg19 GENOME_SIZE = /data/kun/Jingwei/annotation/chrom_hg19.sizes

#######################################################################

Allele specific analysis

#######################################################################

ALLELE_SPECIFIC_SNP =

#######################################################################

Capture Hi-C analysis

#######################################################################

CAPTURE_TARGET = REPORT_CAPTURE_REPORTER = 1

#######################################################################

Digestion Hi-C

#######################################################################

GENOME_FRAGMENT = /data/kun/Jingwei/annotation/HindIII_resfrag_hg19.bed LIGATION_SITE = AAGCTAGCTT MIN_FRAG_SIZE = 100 MAX_FRAG_SIZE = 100000 MIN_INSERT_SIZE = 100 MAX_INSERT_SIZE = 800

#######################################################################

Hi-C processing

#######################################################################

MIN_CIS_DIST = GET_ALL_INTERACTION_CLASSES = 1 GET_PROCESS_SAM = 0 RM_SINGLETON = 1 RM_MULTI = 1 RM_DUP = 1

#######################################################################

Contact Maps

#######################################################################

BIN_SIZE = 20000 40000 150000 MATRIX_FORMAT = upper

#######################################################################

Normalization

####################################################################### MAX_ITER = 100 FILTER_LOW_COUNT_PERC = 0.02 FILTER_HIGH_COUNT_PERC = 0 EPS = 0.1

YaqiangCao commented 4 years ago

Hi Kun, Could you please share me a small chromosome data (such as chr21) for HiC-Pro output or converted BEDPE file. I will check the data. Best, Yaqiang

On Mon, Sep 14, 2020 at 4:08 PM KunFang notifications@github.com wrote:

Hi, Yaqiang

Sorry for bothering but when I am trying to call hic loops of gsm1631184, I only get 2 loops. The hic data is processed by hicpro with the attached config file and then use the new hicpropairs2bedpe. I was wondering if there is any way to debug where goes wrong? Thanks!

Best, Kun

On Sep 14, 2020, at 10:00 AM, YaqiangCao notifications@github.com wrote:

Hi, Yaqiang

I found that hicpro2bedpe is not working properly as the code misunderstands the meaning of 11th and 12th col. They are not read length but the mapping quality. Please check the this website: https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY < https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY>.

Best, Kun

Hi Kun, Thank you very much for the reminding. Sorry for the carelessness. I have fix the problem, please refer to https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe < https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe> for updated code (line 17-18). Please let me kown if you have found other problems. Have a nice day. Thank you again. Best, Yaqiang

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692115131>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ADXLIA3HMMOTIFWXWLMYCWLSFYVZFANCNFSM4RLVVTPA .

Please change the variable settings below if necessary

#########################################################################

Paths and Settings - Do not edit !

#########################################################################

TMP_DIR = tmp LOGS_DIR = log BOWTIE2_OUTPUT_DIR = bowtie_results MAPC_OUTPUT = hic_results RAW_DIR = rawdata

#######################################################################

SYSTEM AND SCHEDULER - Start Editing Here !!

####################################################################### N_CPU = 25 LOGFILE = hicpro.log

JOB_NAME = Jingwei_HiC JOB_MEM = 20gb JOB_WALLTIME = 600:00:00 JOB_QUEUE = JOB_MAIL = fangk@livemail.uthscsa.edu

#########################################################################

Data

#########################################################################

PAIR1_EXT = _R1 PAIR2_EXT = _R2

#######################################################################

Alignment options

#######################################################################

MIN_MAPQ = 10

BOWTIE2_IDX_PATH = /data/kun/Softwares/hiclib/hiclib-legacy/bin/bowtie2/index BOWTIE2_GLOBAL_OPTIONS = --very-sensitive -L 30 --score-min L,-0.6,-0.2 --end-to-end --reorder BOWTIE2_LOCAL_OPTIONS = --very-sensitive -L 20 --score-min L,-0.6,-0.2 --end-to-end --reorder

#######################################################################

Annotation files

#######################################################################

REFERENCE_GENOME = hg19 GENOME_SIZE = /data/kun/Jingwei/annotation/chrom_hg19.sizes

#######################################################################

Allele specific analysis

#######################################################################

ALLELE_SPECIFIC_SNP =

#######################################################################

Capture Hi-C analysis

#######################################################################

CAPTURE_TARGET = REPORT_CAPTURE_REPORTER = 1

#######################################################################

Digestion Hi-C

#######################################################################

GENOME_FRAGMENT = /data/kun/Jingwei/annotation/HindIII_resfrag_hg19.bed LIGATION_SITE = AAGCTAGCTT MIN_FRAG_SIZE = 100 MAX_FRAG_SIZE = 100000 MIN_INSERT_SIZE = 100 MAX_INSERT_SIZE = 800

#######################################################################

Hi-C processing

#######################################################################

MIN_CIS_DIST = GET_ALL_INTERACTION_CLASSES = 1 GET_PROCESS_SAM = 0 RM_SINGLETON = 1 RM_MULTI = 1 RM_DUP = 1

#######################################################################

Contact Maps

#######################################################################

BIN_SIZE = 20000 40000 150000 MATRIX_FORMAT = upper

#######################################################################

Normalization

####################################################################### MAX_ITER = 100 FILTER_LOW_COUNT_PERC = 0.02 FILTER_HIGH_COUNT_PERC = 0 EPS = 0.1

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692285321, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOPQKOJAKIDM6TE7VWCHF3SFZZ2NANCNFSM4RLVVTPA .

KunFang93 commented 4 years ago

Hi, Yaqiang

I just got the reply from the author of HiC-Pro. The position of allValidPairs now changed back to 5’ end in the latest version (https://github.com/nservant/HiC-Pro/issues/362).

Best, Kun

On Sep 14, 2020, at 3:53 PM, YaqiangCao notifications@github.com wrote:

Hi Kun, Could you please share me a small chromosome data (such as chr21) for HiC-Pro output or converted BEDPE file. I will check the data. Best, Yaqiang

On Mon, Sep 14, 2020 at 4:08 PM KunFang notifications@github.com wrote:

Hi, Yaqiang

Sorry for bothering but when I am trying to call hic loops of gsm1631184, I only get 2 loops. The hic data is processed by hicpro with the attached config file and then use the new hicpropairs2bedpe. I was wondering if there is any way to debug where goes wrong? Thanks!

Best, Kun

On Sep 14, 2020, at 10:00 AM, YaqiangCao notifications@github.com wrote:

Hi, Yaqiang

I found that hicpro2bedpe is not working properly as the code misunderstands the meaning of 11th and 12th col. They are not read length but the mapping quality. Please check the this website: https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY < https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY>.

Best, Kun

Hi Kun, Thank you very much for the reminding. Sorry for the carelessness. I have fix the problem, please refer to https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe < https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe> for updated code (line 17-18). Please let me kown if you have found other problems. Have a nice day. Thank you again. Best, Yaqiang

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692115131>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ADXLIA3HMMOTIFWXWLMYCWLSFYVZFANCNFSM4RLVVTPA .

Please change the variable settings below if necessary

#########################################################################

Paths and Settings - Do not edit !

#########################################################################

TMP_DIR = tmp LOGS_DIR = log BOWTIE2_OUTPUT_DIR = bowtie_results MAPC_OUTPUT = hic_results RAW_DIR = rawdata

#######################################################################

SYSTEM AND SCHEDULER - Start Editing Here !!

####################################################################### N_CPU = 25 LOGFILE = hicpro.log

JOB_NAME = Jingwei_HiC JOB_MEM = 20gb JOB_WALLTIME = 600:00:00 JOB_QUEUE = JOB_MAIL = fangk@livemail.uthscsa.edu

#########################################################################

Data

#########################################################################

PAIR1_EXT = _R1 PAIR2_EXT = _R2

#######################################################################

Alignment options

#######################################################################

MIN_MAPQ = 10

BOWTIE2_IDX_PATH = /data/kun/Softwares/hiclib/hiclib-legacy/bin/bowtie2/index BOWTIE2_GLOBAL_OPTIONS = --very-sensitive -L 30 --score-min L,-0.6,-0.2 --end-to-end --reorder BOWTIE2_LOCAL_OPTIONS = --very-sensitive -L 20 --score-min L,-0.6,-0.2 --end-to-end --reorder

#######################################################################

Annotation files

#######################################################################

REFERENCE_GENOME = hg19 GENOME_SIZE = /data/kun/Jingwei/annotation/chrom_hg19.sizes

#######################################################################

Allele specific analysis

#######################################################################

ALLELE_SPECIFIC_SNP =

#######################################################################

Capture Hi-C analysis

#######################################################################

CAPTURE_TARGET = REPORT_CAPTURE_REPORTER = 1

#######################################################################

Digestion Hi-C

#######################################################################

GENOME_FRAGMENT = /data/kun/Jingwei/annotation/HindIII_resfrag_hg19.bed LIGATION_SITE = AAGCTAGCTT MIN_FRAG_SIZE = 100 MAX_FRAG_SIZE = 100000 MIN_INSERT_SIZE = 100 MAX_INSERT_SIZE = 800

#######################################################################

Hi-C processing

#######################################################################

MIN_CIS_DIST = GET_ALL_INTERACTION_CLASSES = 1 GET_PROCESS_SAM = 0 RM_SINGLETON = 1 RM_MULTI = 1 RM_DUP = 1

#######################################################################

Contact Maps

#######################################################################

BIN_SIZE = 20000 40000 150000 MATRIX_FORMAT = upper

#######################################################################

Normalization

####################################################################### MAX_ITER = 100 FILTER_LOW_COUNT_PERC = 0.02 FILTER_HIGH_COUNT_PERC = 0 EPS = 0.1

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692285321, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOPQKOJAKIDM6TE7VWCHF3SFZZ2NANCNFSM4RLVVTPA .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692308254, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXLIA2CRZDF3A5GDWG65JTSFZ7GDANCNFSM4RLVVTPA.

YaqiangCao commented 4 years ago

Hi Kun, Thank you for the reminder. Sorry for the late reply, a little busy for other projects these days. I may update the script later. May I ask if you loop-calling goes well for your data with cLoops? Best, Yaqiang

On Wed, Sep 16, 2020 at 11:35 AM KunFang notifications@github.com wrote:

Hi, Yaqiang

I just got the reply from the author of HiC-Pro. The position of allValidPairs now changed back to 5’ end in the latest version ( https://github.com/nservant/HiC-Pro/issues/362).

Best, Kun

On Sep 14, 2020, at 3:53 PM, YaqiangCao notifications@github.com wrote:

Hi Kun, Could you please share me a small chromosome data (such as chr21) for HiC-Pro output or converted BEDPE file. I will check the data. Best, Yaqiang

On Mon, Sep 14, 2020 at 4:08 PM KunFang notifications@github.com wrote:

Hi, Yaqiang

Sorry for bothering but when I am trying to call hic loops of gsm1631184, I only get 2 loops. The hic data is processed by hicpro with the attached config file and then use the new hicpropairs2bedpe. I was wondering if there is any way to debug where goes wrong? Thanks!

Best, Kun

On Sep 14, 2020, at 10:00 AM, YaqiangCao notifications@github.com wrote:

Hi, Yaqiang

I found that hicpro2bedpe is not working properly as the code misunderstands the meaning of 11th and 12th col. They are not read length but the mapping quality. Please check the this website: https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY < https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY>.

Best, Kun

Hi Kun, Thank you very much for the reminding. Sorry for the carelessness. I have fix the problem, please refer to

https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe <

https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe

for updated code (line 17-18). Please let me kown if you have found other problems. Have a nice day. Thank you again.

Best, Yaqiang

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692115131 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ADXLIA3HMMOTIFWXWLMYCWLSFYVZFANCNFSM4RLVVTPA

.

Please change the variable settings below if necessary

#########################################################################

Paths and Settings - Do not edit !

#########################################################################

TMP_DIR = tmp LOGS_DIR = log BOWTIE2_OUTPUT_DIR = bowtie_results MAPC_OUTPUT = hic_results RAW_DIR = rawdata

#######################################################################

SYSTEM AND SCHEDULER - Start Editing Here !!

####################################################################### N_CPU = 25 LOGFILE = hicpro.log

JOB_NAME = Jingwei_HiC JOB_MEM = 20gb JOB_WALLTIME = 600:00:00 JOB_QUEUE = JOB_MAIL = fangk@livemail.uthscsa.edu

#########################################################################

Data

#########################################################################

PAIR1_EXT = _R1 PAIR2_EXT = _R2

#######################################################################

Alignment options

#######################################################################

MIN_MAPQ = 10

BOWTIE2_IDX_PATH = /data/kun/Softwares/hiclib/hiclib-legacy/bin/bowtie2/index BOWTIE2_GLOBAL_OPTIONS = --very-sensitive -L 30 --score-min L,-0.6,-0.2 --end-to-end --reorder BOWTIE2_LOCAL_OPTIONS = --very-sensitive -L 20 --score-min L,-0.6,-0.2 --end-to-end --reorder

#######################################################################

Annotation files

#######################################################################

REFERENCE_GENOME = hg19 GENOME_SIZE = /data/kun/Jingwei/annotation/chrom_hg19.sizes

#######################################################################

Allele specific analysis

#######################################################################

ALLELE_SPECIFIC_SNP =

#######################################################################

Capture Hi-C analysis

#######################################################################

CAPTURE_TARGET = REPORT_CAPTURE_REPORTER = 1

#######################################################################

Digestion Hi-C

#######################################################################

GENOME_FRAGMENT = /data/kun/Jingwei/annotation/HindIII_resfrag_hg19.bed LIGATION_SITE = AAGCTAGCTT MIN_FRAG_SIZE = 100 MAX_FRAG_SIZE = 100000 MIN_INSERT_SIZE = 100 MAX_INSERT_SIZE = 800

#######################################################################

Hi-C processing

#######################################################################

MIN_CIS_DIST = GET_ALL_INTERACTION_CLASSES = 1 GET_PROCESS_SAM = 0 RM_SINGLETON = 1 RM_MULTI = 1 RM_DUP = 1

#######################################################################

Contact Maps

#######################################################################

BIN_SIZE = 20000 40000 150000 MATRIX_FORMAT = upper

#######################################################################

Normalization

####################################################################### MAX_ITER = 100 FILTER_LOW_COUNT_PERC = 0.02 FILTER_HIGH_COUNT_PERC = 0 EPS = 0.1

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692285321 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAOPQKOJAKIDM6TE7VWCHF3SFZZ2NANCNFSM4RLVVTPA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692308254>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ADXLIA2CRZDF3A5GDWG65JTSFZ7GDANCNFSM4RLVVTPA .

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-693486897, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOPQKMMB245RUMWQGLCG5DSGDLMPANCNFSM4RLVVTPA .

KunFang93 commented 4 years ago

Hi, Yaqiang

No problem at all. I am still get small number of loops of the sample I mentioned. But I checked the quality of the sample, the quality is not very good, I guess this is why we don’t get enough loop from it. Besides, I was wondering if there is anyway to speed up the program? Some of my samples have already processed 30h and not finished yet. Thanks!

Best, Kun

On Sep 17, 2020, at 10:54 AM, YaqiangCao notifications@github.com wrote:

Hi Kun, Thank you for the reminder. Sorry for the late reply, a little busy for other projects these days. I may update the script later. May I ask if you loop-calling goes well for your data with cLoops? Best, Yaqiang

On Wed, Sep 16, 2020 at 11:35 AM KunFang notifications@github.com wrote:

Hi, Yaqiang

I just got the reply from the author of HiC-Pro. The position of allValidPairs now changed back to 5’ end in the latest version ( https://github.com/nservant/HiC-Pro/issues/362).

Best, Kun

On Sep 14, 2020, at 3:53 PM, YaqiangCao notifications@github.com wrote:

Hi Kun, Could you please share me a small chromosome data (such as chr21) for HiC-Pro output or converted BEDPE file. I will check the data. Best, Yaqiang

On Mon, Sep 14, 2020 at 4:08 PM KunFang notifications@github.com wrote:

Hi, Yaqiang

Sorry for bothering but when I am trying to call hic loops of gsm1631184, I only get 2 loops. The hic data is processed by hicpro with the attached config file and then use the new hicpropairs2bedpe. I was wondering if there is any way to debug where goes wrong? Thanks!

Best, Kun

On Sep 14, 2020, at 10:00 AM, YaqiangCao notifications@github.com wrote:

Hi, Yaqiang

I found that hicpro2bedpe is not working properly as the code misunderstands the meaning of 11th and 12th col. They are not read length but the mapping quality. Please check the this website: https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY < https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY>.

Best, Kun

Hi Kun, Thank you very much for the reminding. Sorry for the carelessness. I have fix the problem, please refer to

https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe <

https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe

for updated code (line 17-18). Please let me kown if you have found other problems. Have a nice day. Thank you again.

Best, Yaqiang

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692115131 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ADXLIA3HMMOTIFWXWLMYCWLSFYVZFANCNFSM4RLVVTPA

.

Please change the variable settings below if necessary

#########################################################################

Paths and Settings - Do not edit !

#########################################################################

TMP_DIR = tmp LOGS_DIR = log BOWTIE2_OUTPUT_DIR = bowtie_results MAPC_OUTPUT = hic_results RAW_DIR = rawdata

#######################################################################

SYSTEM AND SCHEDULER - Start Editing Here !!

####################################################################### N_CPU = 25 LOGFILE = hicpro.log

JOB_NAME = Jingwei_HiC JOB_MEM = 20gb JOB_WALLTIME = 600:00:00 JOB_QUEUE = JOB_MAIL = fangk@livemail.uthscsa.edu

#########################################################################

Data

#########################################################################

PAIR1_EXT = _R1 PAIR2_EXT = _R2

#######################################################################

Alignment options

#######################################################################

MIN_MAPQ = 10

BOWTIE2_IDX_PATH = /data/kun/Softwares/hiclib/hiclib-legacy/bin/bowtie2/index BOWTIE2_GLOBAL_OPTIONS = --very-sensitive -L 30 --score-min L,-0.6,-0.2 --end-to-end --reorder BOWTIE2_LOCAL_OPTIONS = --very-sensitive -L 20 --score-min L,-0.6,-0.2 --end-to-end --reorder

#######################################################################

Annotation files

#######################################################################

REFERENCE_GENOME = hg19 GENOME_SIZE = /data/kun/Jingwei/annotation/chrom_hg19.sizes

#######################################################################

Allele specific analysis

#######################################################################

ALLELE_SPECIFIC_SNP =

#######################################################################

Capture Hi-C analysis

#######################################################################

CAPTURE_TARGET = REPORT_CAPTURE_REPORTER = 1

#######################################################################

Digestion Hi-C

#######################################################################

GENOME_FRAGMENT = /data/kun/Jingwei/annotation/HindIII_resfrag_hg19.bed LIGATION_SITE = AAGCTAGCTT MIN_FRAG_SIZE = 100 MAX_FRAG_SIZE = 100000 MIN_INSERT_SIZE = 100 MAX_INSERT_SIZE = 800

#######################################################################

Hi-C processing

#######################################################################

MIN_CIS_DIST = GET_ALL_INTERACTION_CLASSES = 1 GET_PROCESS_SAM = 0 RM_SINGLETON = 1 RM_MULTI = 1 RM_DUP = 1

#######################################################################

Contact Maps

#######################################################################

BIN_SIZE = 20000 40000 150000 MATRIX_FORMAT = upper

#######################################################################

Normalization

####################################################################### MAX_ITER = 100 FILTER_LOW_COUNT_PERC = 0.02 FILTER_HIGH_COUNT_PERC = 0 EPS = 0.1

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692285321 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAOPQKOJAKIDM6TE7VWCHF3SFZZ2NANCNFSM4RLVVTPA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692308254>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ADXLIA2CRZDF3A5GDWG65JTSFZ7GDANCNFSM4RLVVTPA .

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-693486897, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOPQKMMB245RUMWQGLCG5DSGDLMPANCNFSM4RLVVTPA .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-694328463, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXLIAZWHKLGGZGMDKWELHLSGIWMNANCNFSM4RLVVTPA.

YaqiangCao commented 4 years ago

Hi Kun, Thank you for the updates. Regarding the speed, I may not have better solution because from the bottom, cLoops is built up on Python. Following two tricks may work: 1) If you have enough CPUs and memories, -p set to chromosome numbers. 2) with -cut option, filter some close PETs, for Hi-C, for example, 5kb or 10kb. In my experience, run with -p 1 and for about 200 million PETs, it will take about 30-60 hours. Best, Yaqiang

On Thu, Sep 17, 2020 at 12:05 PM KunFang notifications@github.com wrote:

Hi, Yaqiang

No problem at all. I am still get small number of loops of the sample I mentioned. But I checked the quality of the sample, the quality is not very good, I guess this is why we don’t get enough loop from it. Besides, I was wondering if there is anyway to speed up the program? Some of my samples have already processed 30h and not finished yet. Thanks!

Best, Kun

On Sep 17, 2020, at 10:54 AM, YaqiangCao notifications@github.com wrote:

Hi Kun, Thank you for the reminder. Sorry for the late reply, a little busy for other projects these days. I may update the script later. May I ask if you loop-calling goes well for your data with cLoops? Best, Yaqiang

On Wed, Sep 16, 2020 at 11:35 AM KunFang notifications@github.com wrote:

Hi, Yaqiang

I just got the reply from the author of HiC-Pro. The position of allValidPairs now changed back to 5’ end in the latest version ( https://github.com/nservant/HiC-Pro/issues/362).

Best, Kun

On Sep 14, 2020, at 3:53 PM, YaqiangCao notifications@github.com wrote:

Hi Kun, Could you please share me a small chromosome data (such as chr21) for HiC-Pro output or converted BEDPE file. I will check the data. Best, Yaqiang

On Mon, Sep 14, 2020 at 4:08 PM KunFang notifications@github.com wrote:

Hi, Yaqiang

Sorry for bothering but when I am trying to call hic loops of gsm1631184, I only get 2 loops. The hic data is processed by hicpro with the attached config file and then use the new hicpropairs2bedpe. I was wondering if there is any way to debug where goes wrong? Thanks!

Best, Kun

On Sep 14, 2020, at 10:00 AM, YaqiangCao < notifications@github.com> wrote:

Hi, Yaqiang

I found that hicpro2bedpe is not working properly as the code misunderstands the meaning of 11th and 12th col. They are not read length but the mapping quality. Please check the this website: https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY < https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY>.

Best, Kun

Hi Kun, Thank you very much for the reminding. Sorry for the carelessness. I have fix the problem, please refer to

https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe

<

https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe

for updated code (line 17-18). Please let me kown if you have found other problems. Have a nice day. Thank you again.

Best, Yaqiang

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692115131 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ADXLIA3HMMOTIFWXWLMYCWLSFYVZFANCNFSM4RLVVTPA

.

Please change the variable settings below if necessary

#########################################################################

Paths and Settings - Do not edit !

#########################################################################

TMP_DIR = tmp LOGS_DIR = log BOWTIE2_OUTPUT_DIR = bowtie_results MAPC_OUTPUT = hic_results RAW_DIR = rawdata

#######################################################################

SYSTEM AND SCHEDULER - Start Editing Here !!

####################################################################### N_CPU = 25 LOGFILE = hicpro.log

JOB_NAME = Jingwei_HiC JOB_MEM = 20gb JOB_WALLTIME = 600:00:00 JOB_QUEUE = JOB_MAIL = fangk@livemail.uthscsa.edu

#########################################################################

Data

#########################################################################

PAIR1_EXT = _R1 PAIR2_EXT = _R2

#######################################################################

Alignment options

#######################################################################

MIN_MAPQ = 10

BOWTIE2_IDX_PATH = /data/kun/Softwares/hiclib/hiclib-legacy/bin/bowtie2/index BOWTIE2_GLOBAL_OPTIONS = --very-sensitive -L 30 --score-min L,-0.6,-0.2 --end-to-end --reorder BOWTIE2_LOCAL_OPTIONS = --very-sensitive -L 20 --score-min L,-0.6,-0.2 --end-to-end --reorder

#######################################################################

Annotation files

#######################################################################

REFERENCE_GENOME = hg19 GENOME_SIZE = /data/kun/Jingwei/annotation/chrom_hg19.sizes

#######################################################################

Allele specific analysis

#######################################################################

ALLELE_SPECIFIC_SNP =

#######################################################################

Capture Hi-C analysis

#######################################################################

CAPTURE_TARGET = REPORT_CAPTURE_REPORTER = 1

#######################################################################

Digestion Hi-C

#######################################################################

GENOME_FRAGMENT = /data/kun/Jingwei/annotation/HindIII_resfrag_hg19.bed LIGATION_SITE = AAGCTAGCTT MIN_FRAG_SIZE = 100 MAX_FRAG_SIZE = 100000 MIN_INSERT_SIZE = 100 MAX_INSERT_SIZE = 800

#######################################################################

Hi-C processing

#######################################################################

MIN_CIS_DIST = GET_ALL_INTERACTION_CLASSES = 1 GET_PROCESS_SAM = 0 RM_SINGLETON = 1 RM_MULTI = 1 RM_DUP = 1

#######################################################################

Contact Maps

#######################################################################

BIN_SIZE = 20000 40000 150000 MATRIX_FORMAT = upper

#######################################################################

Normalization

####################################################################### MAX_ITER = 100 FILTER_LOW_COUNT_PERC = 0.02 FILTER_HIGH_COUNT_PERC = 0 EPS = 0.1

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692285321 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAOPQKOJAKIDM6TE7VWCHF3SFZZ2NANCNFSM4RLVVTPA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692308254 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ADXLIA2CRZDF3A5GDWG65JTSFZ7GDANCNFSM4RLVVTPA

.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-693486897 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAOPQKMMB245RUMWQGLCG5DSGDLMPANCNFSM4RLVVTPA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-694328463>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ADXLIAZWHKLGGZGMDKWELHLSGIWMNANCNFSM4RLVVTPA .

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-694334867, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOPQKKZYIQRIT7B36BMCNDSGIXVRANCNFSM4RLVVTPA .

KunFang93 commented 4 years ago

Hi, Yaqiang

Thanks for your info! Understand, I just a little bit worried the program was stuck before as I am not sure how long the processing will take. Since your experience is within 30-60h, I might just wait for it.

Best, Kun

On Sep 17, 2020, at 11:11 AM, YaqiangCao notifications@github.com wrote:

Hi Kun, Thank you for the updates. Regarding the speed, I may not have better solution because from the bottom, cLoops is built up on Python. Following two tricks may work: 1) If you have enough CPUs and memories, -p set to chromosome numbers. 2) with -cut option, filter some close PETs, for Hi-C, for example, 5kb or 10kb. In my experience, run with -p 1 and for about 200 million PETs, it will take about 30-60 hours. Best, Yaqiang

On Thu, Sep 17, 2020 at 12:05 PM KunFang notifications@github.com wrote:

Hi, Yaqiang

No problem at all. I am still get small number of loops of the sample I mentioned. But I checked the quality of the sample, the quality is not very good, I guess this is why we don’t get enough loop from it. Besides, I was wondering if there is anyway to speed up the program? Some of my samples have already processed 30h and not finished yet. Thanks!

Best, Kun

On Sep 17, 2020, at 10:54 AM, YaqiangCao notifications@github.com wrote:

Hi Kun, Thank you for the reminder. Sorry for the late reply, a little busy for other projects these days. I may update the script later. May I ask if you loop-calling goes well for your data with cLoops? Best, Yaqiang

On Wed, Sep 16, 2020 at 11:35 AM KunFang notifications@github.com wrote:

Hi, Yaqiang

I just got the reply from the author of HiC-Pro. The position of allValidPairs now changed back to 5’ end in the latest version ( https://github.com/nservant/HiC-Pro/issues/362).

Best, Kun

On Sep 14, 2020, at 3:53 PM, YaqiangCao notifications@github.com wrote:

Hi Kun, Could you please share me a small chromosome data (such as chr21) for HiC-Pro output or converted BEDPE file. I will check the data. Best, Yaqiang

On Mon, Sep 14, 2020 at 4:08 PM KunFang notifications@github.com wrote:

Hi, Yaqiang

Sorry for bothering but when I am trying to call hic loops of gsm1631184, I only get 2 loops. The hic data is processed by hicpro with the attached config file and then use the new hicpropairs2bedpe. I was wondering if there is any way to debug where goes wrong? Thanks!

Best, Kun

On Sep 14, 2020, at 10:00 AM, YaqiangCao < notifications@github.com> wrote:

Hi, Yaqiang

I found that hicpro2bedpe is not working properly as the code misunderstands the meaning of 11th and 12th col. They are not read length but the mapping quality. Please check the this website: https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY < https://groups.google.com/g/hic-pro/c/qhHNbQVJfYY>.

Best, Kun

Hi Kun, Thank you very much for the reminding. Sorry for the carelessness. I have fix the problem, please refer to

https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe

<

https://github.com/YaqiangCao/cLoops/blob/master/scripts/hicpropairs2bedpe

for updated code (line 17-18). Please let me kown if you have found other problems. Have a nice day. Thank you again.

Best, Yaqiang

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <

https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692115131 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ADXLIA3HMMOTIFWXWLMYCWLSFYVZFANCNFSM4RLVVTPA

.

Please change the variable settings below if necessary

#########################################################################

Paths and Settings - Do not edit !

#########################################################################

TMP_DIR = tmp LOGS_DIR = log BOWTIE2_OUTPUT_DIR = bowtie_results MAPC_OUTPUT = hic_results RAW_DIR = rawdata

#######################################################################

SYSTEM AND SCHEDULER - Start Editing Here !!

####################################################################### N_CPU = 25 LOGFILE = hicpro.log

JOB_NAME = Jingwei_HiC JOB_MEM = 20gb JOB_WALLTIME = 600:00:00 JOB_QUEUE = JOB_MAIL = fangk@livemail.uthscsa.edu

#########################################################################

Data

#########################################################################

PAIR1_EXT = _R1 PAIR2_EXT = _R2

#######################################################################

Alignment options

#######################################################################

MIN_MAPQ = 10

BOWTIE2_IDX_PATH = /data/kun/Softwares/hiclib/hiclib-legacy/bin/bowtie2/index BOWTIE2_GLOBAL_OPTIONS = --very-sensitive -L 30 --score-min L,-0.6,-0.2 --end-to-end --reorder BOWTIE2_LOCAL_OPTIONS = --very-sensitive -L 20 --score-min L,-0.6,-0.2 --end-to-end --reorder

#######################################################################

Annotation files

#######################################################################

REFERENCE_GENOME = hg19 GENOME_SIZE = /data/kun/Jingwei/annotation/chrom_hg19.sizes

#######################################################################

Allele specific analysis

#######################################################################

ALLELE_SPECIFIC_SNP =

#######################################################################

Capture Hi-C analysis

#######################################################################

CAPTURE_TARGET = REPORT_CAPTURE_REPORTER = 1

#######################################################################

Digestion Hi-C

#######################################################################

GENOME_FRAGMENT = /data/kun/Jingwei/annotation/HindIII_resfrag_hg19.bed LIGATION_SITE = AAGCTAGCTT MIN_FRAG_SIZE = 100 MAX_FRAG_SIZE = 100000 MIN_INSERT_SIZE = 100 MAX_INSERT_SIZE = 800

#######################################################################

Hi-C processing

#######################################################################

MIN_CIS_DIST = GET_ALL_INTERACTION_CLASSES = 1 GET_PROCESS_SAM = 0 RM_SINGLETON = 1 RM_MULTI = 1 RM_DUP = 1

#######################################################################

Contact Maps

#######################################################################

BIN_SIZE = 20000 40000 150000 MATRIX_FORMAT = upper

#######################################################################

Normalization

####################################################################### MAX_ITER = 100 FILTER_LOW_COUNT_PERC = 0.02 FILTER_HIGH_COUNT_PERC = 0 EPS = 0.1

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692285321 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAOPQKOJAKIDM6TE7VWCHF3SFZZ2NANCNFSM4RLVVTPA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-692308254 , or unsubscribe <

https://github.com/notifications/unsubscribe-auth/ADXLIA2CRZDF3A5GDWG65JTSFZ7GDANCNFSM4RLVVTPA

.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-693486897 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAOPQKMMB245RUMWQGLCG5DSGDLMPANCNFSM4RLVVTPA

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-694328463>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ADXLIAZWHKLGGZGMDKWELHLSGIWMNANCNFSM4RLVVTPA .

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-694334867, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOPQKKZYIQRIT7B36BMCNDSGIXVRANCNFSM4RLVVTPA .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/YaqiangCao/cLoops/issues/23#issuecomment-694338304, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADXLIA5Y3NQ6ANJLV3GTKCTSGIYM3ANCNFSM4RLVVTPA.