search

pinellolab / dictys

Context specific and dynamic gene regulatory network reconstruction and analysis
GNU Affero General Public License v3.0
108 stars 14 forks source link

Run bulk ATAC-seq data failed!!! #37

Closed bioinformaticspcj closed 11 months ago

bioinformaticspcj commented 11 months ago

Dear Lingfei,

I have tried to add the number to distinguish each motif name as you suggested in https://github.com/pinellolab/dictys/issues/28. However the same error still occured. I also tried dictys_helper makefile_check.py -c command, the results are as follows: Joint profile: True Found 7047 cells with RNA profile Found 55095 genes with RNA profile ERROR:root:2 WARNING:root:Using RNA cell names for ATAC cell names for validations below. Found 7047 cells with ATAC profile Found 356 motifs Found 356 TFs Found 311 TFs in current dataset Missing 45 TFs in current dataset: ANDR,AP2A,AP2C,ARI5B,BHA15,BHE40,BMAL1,BRAC,COE1,COT1,COT2,DMRTB,EVI1,GCR,HEN1,HNF6,HTF4,ITF2,KAISO,NDF1,NDF2,NGN2,NKX2-8,PEBB,PKNX1,PRD14,PRD16,PRGR,RORG,SUH,TF2L1,TF65,TF7L1,TF7L2,TFE2,THA,THA11,ZBT17,ZBT7A,ZKSC1,ZN143,ZN281,ZN322,ZN335,ZN431 Found 268 genes with TSS information WARNING:root:Cannot find dynamic.mk or traj_node.h5. Skipping dynamic network inference checks. Traceback (most recent call last): File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/scripts/helper/makefile_check.py", line 354, in raise RuntimeError(f'Found {nerr} error(s) in total.') RuntimeError: Found 1 error(s) in total.

The motif file that I used has been uploaded to https://figshare.com/articles/dataset/Mouse_TF_motif/24333844 in figshare.

Could you spare sometime to help me?

Thanks a lot.

Best, Bob

lingfeiwang commented 11 months ago

Dear Bob,

That seems alright for bulk ATAC-seq data assuming you followed #23, because

Expect failures in "Validate input data" step because of that and stop before "Network inference" step.

Btw if you update Dictys to the latest version you should see the error message showing up correctly.

Please let us know if you have further questions.

Lingfei

bioinformaticspcj commented 11 months ago

Dear Lingfei, Thank you very much for your timely reply. I am sorry if I did not express myself clearly.

I have disccused this issue with you in Issue #28. In short, I have updated the dictys program from v1.0.0 to dev branch as you suggested and rerun the command. You have seen the out include error that I provide to you in https://figshare.com/articles/online_resource/Single_Cell_Log/24278206 in Figshare. Then I have followed your suggestions and added the number (_0,_1...) in the motif file (has been uploaded to https://figshare.com/articles/dataset/Mouse_TF_motif/24333844). However the sam error that was reported in https://figshare.com/articles/online_resource/Single_Cell_Log/24278206 occured again.

Now I hope you could help me check the motif file (in https://figshare.com/articles/dataset/Mouse_TF_motif/24333844) to see if there are any other issues.

Btw, I have reran the makefile_check.py program and the output is as follows: Joint profile: True Found 7047 cells with RNA profile Found 55095 genes with RNA profile ERROR:root:2 WARNING:root:Using RNA cell names for ATAC cell names for validations below. Found 7047 cells with ATAC profile Found 356 motifs Found 356 TFs Found 311 TFs in current dataset Missing 45 TFs in current dataset: ANDR,AP2A,AP2C,ARI5B,BHA15,BHE40,BMAL1,BRAC,COE1,COT1,COT2,DMRTB,EVI1,GCR,HEN1,HNF6,HTF4,ITF2,KAISO,NDF1,NDF2,NGN2,NKX2-8,PEBB,PKNX1,PRD14,PRD16,PRGR,RORG,SUH,TF2L1,TF65,TF7L1,TF7L2,TFE2,THA,THA11,ZBT17,ZBT7A,ZKSC1,ZN143,ZN281,ZN322,ZN335,ZN431 Found 268 genes with TSS information WARNING:root:Cannot find dynamic.mk or traj_node.h5. Skipping dynamic network inference checks. Traceback (most recent call last):   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/scripts/helper/makefile_check.py", line 354, in <module>     raise RuntimeError(f'Found {nerr} error(s) in total.') RuntimeError: Found 1 error(s) in total.

I hope I made myself clear this time and you could help me.

Thanks again.

Best, Bob

------------------ 原始邮件 ------------------ 发件人: "pinellolab/dictys" @.>; 发送时间: 2023年11月2日(星期四) 晚上8:00 @.>; @.**@.>; 主题: Re: [pinellolab/dictys] Run bulk ATAC-seq data failed!!! (Issue #37)

Dear Bob,

That seems alright for bulk ATAC-seq data assuming you followed #23, because

Expect failures in "Validate input data" step because of that and stop before "Network inference" step.

Btw if you update Dictys to the latest version you should see the error message showing up correctly.

Please let us know if you have further questions.

Lingfei

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

lingfeiwang commented 11 months ago

Dear Bob,

Thank you for the clarifications. Could you update Dictys to the latest dev branch and run the same line for GRN inference again? It should provide expanded error message about which motif names are duplicates. From this we can troubleshoot further.

Lingfei

bioinformaticspcj commented 11 months ago

Dear Lingfei,

Thanks for your valuable advice. I have updated the Dictys and run the same line for inference again. The error output is as follows:

/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in <module>     docstringrunner(package)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner     run_args(pkgname,funcs,args)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args     return func(a,ka)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 250, in homer     return _motif_postproc(d2,fi_exp,fo_bed,fo_wellington,fo_homer)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 194, in _motif_postproc     raise ValueError('Found non-unique motif name suffices. Each motif name is recommended to contain a unique suffix. First three non-unique motif names: {}'.format(', '.join(t1))) ValueError: Found non-unique motif name suffices. Each motif name is recommended to contain a unique suffix. First three non-unique motif names:  make: [makefiles/common.mk:144: tmp_static/Subset4/motifs.bed] Error 1 Traceback (most recent call last):   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in <module>     docstringrunner(package)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner     run_args(pkgname,funcs,args)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args     return func(*a,ka)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 250, in homer     return _motif_postproc(d2,fi_exp,fo_bed,fo_wellington,fo_homer)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 194, in _motif_postproc     raise ValueError('Found non-unique motif name suffices. Each motif name is recommended to contain a unique suffix. First three non-unique motif names: {}'.format(', '.join(t1))) ValueError: Found non-unique motif name suffices. Each motif name is recommended to contain a unique suffix. First three non-unique motif names:  Traceback (most recent call last):   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in <module>     docstringrunner(package)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner     run_args(pkgname,funcs,args)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args     return func(*a,*ka)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 250, in homer     return _motif_postproc(d2,fi_exp,fo_bed,fo_wellington,fo_homer)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 194, in _motif_postproc     raise ValueError('Found non-unique motif name suffices. Each motif name is recommended to contain a unique suffix. First three non-unique motif names: {}'.format(', '.join(t1))) ValueError: Found non-unique motif name suffices. Each motif name is recommended to contain a unique suffix. First three non-unique motif names:  make: [makefiles/common.mk:144: tmp_static/Subset2/motifs.bed] Error 1 Traceback (most recent call last):   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in <module>     docstringrunner(package)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner     run_args(pkgname,funcs,args)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args     return func(a,ka)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 250, in homer     return _motif_postproc(d2,fi_exp,fo_bed,fo_wellington,fo_homer)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 194, in _motif_postproc     raise ValueError('Found non-unique motif name suffices. Each motif name is recommended to contain a unique suffix. First three non-unique motif names: {}'.format(', '.join(t1))) ValueError: Found non-unique motif name suffices. Each motif name is recommended to contain a unique suffix. First three non-unique motif names:  make: [makefiles/common.mk:144: tmp_static/Subset6/motifs.bed] Error 1 Traceback (most recent call last):   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in <module>     docstringrunner(package)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner     run_args(pkgname,funcs,args)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args     return func(a,ka)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 250, in homer     return _motif_postproc(d2,fi_exp,fo_bed,fo_wellington,fo_homer)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/chromatin.py", line 194, in _motif_postproc     raise ValueError('Found non-unique motif name suffices. Each motif name is recommended to contain a unique suffix. First three non-unique motif names: {}'.format(', '.join(t1))) ValueError: Found non-unique motif name suffices. Each motif name is recommended to contain a unique suffix. First three non-unique motif names:  make: [makefiles/common.mk:144: tmp_static/Subset1/motifs.bed] Error 1 make: [makefiles/common.mk:144: tmp_static/Subset3/motifs.bed] Error 1 make: Target 'cpu' not remade because of errors. mkdir -p output/ OPENBLAS_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 MKL_NUM_THREADS=1 OPENBLAS_MAX_THREADS=1 NUMEXPR_MAX_THREADS=1 MKL_MAX_THREADS=1 python3 -m dictys  network tofile  data tmp_static data/subsets.txt output/static.h5 WARNING:26283:2023-11-06 17:42:30,639:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset1 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,641:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset1 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,641:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset10 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,642:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset10 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,643:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset2 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,643:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset2 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,644:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset3 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,644:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset3 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,645:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset4 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,646:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset4 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,646:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset5 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,647:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset5 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,647:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset6 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,648:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset6 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,649:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset7 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,649:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset7 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,650:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset8 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,650:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset8 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,651:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset9 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:26283:2023-11-06 17:42:30,652:/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:403:Skipping cell subset Subset9 due to error: FileNotFoundError(2, 'No such file or directory') Traceback (most recent call last):   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in <module>     docstringrunner(package)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner     run_args(pkgname,funcs,args)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args     return func(a,ka)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 1074, in tofile     n=network.from_folders(diri_data,diri_work,fi_subsets,dynamic=dynamic,nettype=nettype,optional=optional,fi_c=fi_c)   File "/data/nfs/OriginTools/pcj/python3/miniconda3/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py", line 407, in from_folders     raise RuntimeError('No network found.') RuntimeError: No network found. make: *** [makefiles/common.mk:178: output/static.h5] Error 1

Even though the expanded error was printed, the duplicated motif names were not output.  I do not know why.

Do you have any more suggestions?

Best, Bob

------------------ 原始邮件 ------------------ 发件人: "pinellolab/dictys" @.>; 发送时间: 2023年11月3日(星期五) 上午10:40 @.>; @.**@.>; 主题: Re: [pinellolab/dictys] Run bulk ATAC-seq data failed!!! (Issue #37)

Dear Bob,

Thank you for the clarifications. Could you update Dictys to the latest dev branch and run the same line for GRN inference again? It should provide expanded error message about which motif names are duplicates. From this we can troubleshoot further. 

Lingfei

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

lingfeiwang commented 11 months ago

It appears that a motif with empty name '' was identified. Is it because you have an empty line in the end of your motif file?

I also suggest to try the tutorial first. If the tutorial doesn't work for you, please raise an issue.

If the tutorial works, you can try changing the motif file to your own one. You can also try many things like using only one motif or splitting the motif file by half to see where the problem is. You can try your motif file with homer too. But generally, motif files are an established format and you can search for ways to diagnose any problem with it.