Running Dictys makefile static.mk

[sureshsatpati]

Checks before submitting the issue

• [x] I installed the latest Dictys version successfully
• [x] I downloaded the whole folder for the tutorial
• [x] I ran the tutorial cells and notebooks (if more than one) in the correct order and only once
• [x] The preceeding cells and notebooks finished successfully (i.e. the reported error is the first encountered)

Describe the error

• How did you install Dictys (which option)?
• conda environment
• Which tutorial?
• https://github.com/pinellolab/dictys/blob/master/doc/tutorials/full-multiome/notebooks/3-static-inference.ipynb
• What and where was the error?
• make: No rule to make target 'tmp_static/Subset1/net_weight.tsv.gz', needed by 'tmp_static/Subset1/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset1/net_meanvar.tsv.gz', needed by 'tmp_static/Subset1/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset1/net_covfactor.tsv.gz', needed by 'tmp_static/Subset1/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset10/net_weight.tsv.gz', needed by 'tmp_static/Subset10/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset10/net_meanvar.tsv.gz', needed by 'tmp_static/Subset10/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset10/net_covfactor.tsv.gz', needed by 'tmp_static/Subset10/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset11/net_weight.tsv.gz', needed by 'tmp_static/Subset11/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset11/net_meanvar.tsv.gz', needed by 'tmp_static/Subset11/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset11/net_covfactor.tsv.gz', needed by 'tmp_static/Subset11/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset12/net_weight.tsv.gz', needed by 'tmp_static/Subset12/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset12/net_meanvar.tsv.gz', needed by 'tmp_static/Subset12/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset12/net_covfactor.tsv.gz', needed by 'tmp_static/Subset12/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset2/net_weight.tsv.gz', needed by 'tmp_static/Subset2/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset2/net_meanvar.tsv.gz', needed by 'tmp_static/Subset2/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset2/net_covfactor.tsv.gz', needed by 'tmp_static/Subset2/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset3/net_weight.tsv.gz', needed by 'tmp_static/Subset3/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset3/net_meanvar.tsv.gz', needed by 'tmp_static/Subset3/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset3/net_covfactor.tsv.gz', needed by 'tmp_static/Subset3/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset4/net_weight.tsv.gz', needed by 'tmp_static/Subset4/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset4/net_meanvar.tsv.gz', needed by 'tmp_static/Subset4/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset4/net_covfactor.tsv.gz', needed by 'tmp_static/Subset4/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset5/net_weight.tsv.gz', needed by 'tmp_static/Subset5/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset5/net_meanvar.tsv.gz', needed by 'tmp_static/Subset5/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset5/net_covfactor.tsv.gz', needed by 'tmp_static/Subset5/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset6/net_weight.tsv.gz', needed by 'tmp_static/Subset6/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset6/net_meanvar.tsv.gz', needed by 'tmp_static/Subset6/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset6/net_covfactor.tsv.gz', needed by 'tmp_static/Subset6/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset7/net_weight.tsv.gz', needed by 'tmp_static/Subset7/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset7/net_meanvar.tsv.gz', needed by 'tmp_static/Subset7/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset7/net_covfactor.tsv.gz', needed by 'tmp_static/Subset7/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset8/net_weight.tsv.gz', needed by 'tmp_static/Subset8/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset8/net_meanvar.tsv.gz', needed by 'tmp_static/Subset8/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset8/net_covfactor.tsv.gz', needed by 'tmp_static/Subset8/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset9/net_weight.tsv.gz', needed by 'tmp_static/Subset9/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset9/net_meanvar.tsv.gz', needed by 'tmp_static/Subset9/net_nweight.tsv.gz'. make: No rule to make target 'tmp_static/Subset9/net_covfactor.tsv.gz', needed by 'tmp_static/Subset9/net_nweight.tsv.gz'. Traceback (most recent call last): File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in docstringrunner(package) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner run_args(pkgname,funcs,args) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args return func(a,ka) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/preproc.py", line 140, in qc_reads raise RuntimeError('All genes removed in QC.') RuntimeError: All genes removed in QC. make: [makefiles/common.mk:124: tmp_static/Subset12/expression.tsv.gz] Error 1 make: Target 'cpu' not remade because of errors.
• Did you deviate from any of the tutorial instructions?

Optional steps (may accelerate troubleshooting)

• Attach the error notebook and all preceeding notebooks with run output

[lingfeiwang]

Hi sureshsatpati,

Thank you for reporting this! I understand these messages look daunting. However they are actually fine and do not indicate any problem on their own.

The "make: *** No rule to make target..." error arises because the GPU step was not yet performed. The majority of them should be gone in CPU part 2 as long as no other failures happen for the particular cell subset, e.g. from QC below. I will suppress these messages in the next release because they do cause confusion.

The "RuntimeError: All genes removed in QC." error does occur for certain cell subsets if they are too small so no gene pass QC. It does not mean all cell subsets failed.

If the notebook did stop running because of any error, or did not produce the final network file upon finishing, please do post any error you may received. The above ones look benign to me.

Lingfei

[sureshsatpati]

Hi Lingfei, thank you for the prompt response!

I have another question about the input files for the next step, "python3 -m dictys preproc selects_rna -h" from the tutorial.

Looking at the positional arguments, it looks like I needs 1 "reads" file and 1 "names" file as input. What files are those? Could you please possibly share demo files for those files? Or point me to where I should find them if they were produced in previous steps?

Could you also please provide the same information for the input files for the following steps?

Thank you, Dr. Suresh Satpati

[lingfeiwang]

Hi Suresh,

Sure. The GRN inference pipeline is based on make, so you can see the exact commands executed there. You can simply search for python3 -m dictys preproc selects_rna in the tutorial notebook above to check its parameters and look into the files. To understand how these files were produced, you can further search for these file names upwards in the same notebook. You can also inspect the relevant .mk Makefiles to understand each rule that created these files.

I hope that answers your question.

Lingfei

[sureshsatpati]

Thanks Lingfei! When I ran GPU part for network reconstruct I see below errors. Can you please suggest what could be the possible reason for this.

AssertionError: Torch not compiled with CUDA enabled make: [makefiles/common.mk:169: tmp_static/Subset2/net_weight.tsv.gz] Error 1 WARNING:31373:2023-07-31 13:01:51,898:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/utils/parallel.py:63:Environmental variables have different values. Traceback (most recent call last): File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in docstringrunner(package) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner run_args(pkgname,funcs,args) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args return func(a,ka) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 787, in reconstruct model=model('pyromodel',np.array(np.nonzero(mask)),mask.shape,npc,dc,floattype=dtype,device=device,scale_lyapunov=scale_lyapunov) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 626, in init super().init(name,npc,dcs,ka) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 499, in init self.dcs=self.tensor(dcs.T) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 211, in tensor return torch.tensor(v,device=device,dtype=dtype,ka) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/torch/cuda/init.py", line 239, in _lazy_init raise AssertionError("Torch not compiled with CUDA enabled")

[lingfeiwang]

Sure. Here: https://github.com/pinellolab/dictys#faq

[lingfeiwang]

Hi Suresh,

We just updated our support for pytorch 2. You may try a fresh reinstall to see if the problem goes away.

Lingfei

[sureshsatpati]

Hi Lingfei, Thanks for the update and congratulations for the paper published in nature methods.

As you suggested I reinstalled Dictys and I started running from the aggregate network command once again where I gets the following error:

WARNING:74140:2023-07-31 12:03:51,345:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset1 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,346:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset1 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,346:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset10 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,347:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset10 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,347:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset11 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,347:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset11 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,348:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,348:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,349:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset2 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,349:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset2 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,349:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset3 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,350:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset3 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,350:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset4 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,351:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset4 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,351:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset5 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,351:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset5 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,352:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset6 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,352:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset6 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,353:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset7 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,355:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset7 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,355:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset8 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,356:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset8 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,356:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset9 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:74140:2023-07-31 12:03:51,356:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset9 due to error: FileNotFoundError(2, 'No such file or directory') Traceback (most recent call last): File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in docstringrunner(package) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner run_args(pkgname,funcs,args) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args return func(a,ka) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 1071, in tofile n=network.from_folders(diri_data,diri_work,fi_subsets,dynamic=dynamic,nettype=nettype,optional=optional,fi_c=fi_c) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py", line 398, in from_folders raise RuntimeError('No network found.') RuntimeError: No network found. make: [makefiles/common.mk:175: output/static.h5] Error 1

Please suggest how to proceed?

Thanks, Suresh

[lingfeiwang]

Hi Suresh,

That appears a consequence of an earlier error. Can you confirm all four checkboxes on the top of the issue? If so, could you upload the full notebook?

Also, if you simply want to run Dictys successfully, I recommend the short-multiome tutorial as a start.

Thanks, Lingfei

[sureshsatpati]

Hi Lingfei, Yes, I installed dictys using conda environment in my HPCC account and also another installation in the same HPCC with the help of an IT guy. I still see the same error in both of them. As you suggested I am doing the short-multiome tutorial and will come back to you If I see the same issue. How can I upload my full notebook? Did you mean uploading the steps I followed till now in the HPCC?

Thanks, Suresh

[sureshsatpati]

Hi Linfei, When I run the GRN inference with scRNA-seq on GPU with below commands I get below error. I assume its with the GPU installation with CUDA.

module purge module load miniconda3 eval "$(/risapps/rhel7/miniconda3/38_4.9.2/bin/conda shell.bash hook)" . activate dictys-new

set -eo pipefail

GRN inference with scRNA-seq on GPU

make -f makefiles/static.mk -j 2 -k gpu || true make: [makefiles/common.mk:169: tmp_static/Subset7/net_weight.tsv.gz] Error 1 make: [makefiles/common.mk:169: tmp_static/Subset6/net_weight.tsv.gz] Error 1 Traceback (most recent call last): File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/main.py", line 13, in docstringrunner(package) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner run_args(pkgname,funcs,args) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args return func(*a,ka) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 790, in reconstruct model=model('pyromodel',np.array(np.nonzero(mask)),mask.shape,npc,dc,floattype=dtype,device=device,scale_lyapunov=scale_lyapunov) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 629, in init super().init(name,npc,dcs,ka) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 499, in init self.dcs=self.tensor(dcs) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 211, in tensor return torch.tensor(v,device=device,dtype=dtype,ka) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/torch/cuda/init.py", line 239, in _lazy_init raise AssertionError("Torch not compiled with CUDA enabled") AssertionError: Torch not compiled with CUDA enabled Traceback (most recent call last): File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/main.py", line 13, in docstringrunner(package) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner run_args(pkgname,funcs,args) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args return func(*a,ka) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 790, in reconstruct model=model('pyromodel',np.array(np.nonzero(mask)),mask.shape,npc,dc,floattype=dtype,device=device,scale_lyapunov=scale_lyapunov) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 629, in init super().init(name,npc,dcs,ka) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 499, in init self.dcs=self.tensor(dcs) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/dictys/network.py", line 211, in tensor return torch.tensor(v,device=device,dtype=dtype,*ka) File "/home/ssatpati/.conda/envs/dictys-new/lib/python3.9/site-packages/torch/cuda/init.py", line 239, in _lazy_init raise AssertionError("Torch not compiled with CUDA enabled") AssertionError: Torch not compiled with CUDA enabled make: [makefiles/common.mk:169: tmp_static/Subset9/net_weight.tsv.gz] Error 1 make: *** [makefiles/common.mk:169: tmp_static/Subset8/net_weight.tsv.gz] Error 1 make: Target 'gpu' not remade because of errors.

Please suggest!

Thanks Suresh

[lingfeiwang]

Hi Suresh,

This is the same issue you reported earlier. You should follow the FAQ at https://github.com/pinellolab/dictys#faq to address the issue.

Lingfei

[sureshsatpati]

Hi Lingfei, Because our GPU nodes on cluster don't have internet access. Therefore, when we install conda DICTYS environment on a CPU node with pytorch in a CPU version. Is DICTYS conda environment working for CPU node?

Thanks, Suresh & Rong (IT Dept)

[lingfeiwang]

Hi Suresh & Rong,

Yes. You can install and use the CPU version for pytorch. You need to set pytorch to CPU mode according to the tutorial. But it can be slow.

Alternatively, you can install the GPU version of pytorch on a CPU node and then use it on a GPU node. You need to complete the data downloading step on a CPU node in advance as well.

Dictys has a standard anaconda installation option. Most IT should support this.

Lingfei

[ryao-mdanderson]

Hi Lingfei,

Thank you for your suggestion. I am an HPC system administor, for a conda environment, if install pytorch on a cpu node, it is a cpu version of pytorch; if install pytorch on a gpu node, then it is a gpu version. On our cluster, pytorch for conda environment is cpu version.

Therefore, Dr. Satpati should be able to use cpu version of the software, as he uses command make -f makefiles/static.mk -j 32 -k cpu || true to do make file , the output is without gpu error as he reported earlier.

I have no problem to install dictys conda environment, However, not being a scientist, my running of the command in an active mode $ make -f makefiles/static.mk -j 32 -k cpu || true def is_extension_array_dtype(arr_or_dtype) -> bool: KeyboardInterrupt make: [makefiles/common.mk:124: tmp_static/Subset8/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset2/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset4/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset10/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset6/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset3/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset12/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset11/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset9/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset5/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset1/expression0.tsv.gz] Interrupt make: [makefiles/common.mk:124: tmp_static/Subset7/expression0.tsv.gz] Error 1

It ends with the abovve message, I am not sure of the last sentense Error 1 does mean anything? However, I can see the tmp_static directory filled with subdirectories Subset[1-12], each directories has two files names_rna.txt and names_atac0.txt. I wonder if this completes the command. I believe Scientifically, Suresh and you know much better than I do.

Besides that, I am curious the possibility you mentioned " install the GPU version of pytorch on a CPU node" for a conda environment, any reference for that?

Thank you, Rong Yao

[lingfeiwang]

Hi Rong,

The pytorch installation method worked for us. You should contact pytorch to install it properly on your cluster.

Your error message on KeyboardInterrupt appears a cluster issue too.

Lingfei

[sureshsatpati]

Hi Lingfei, I was able to successfully run GRN inference using CPU mode. But when I run aggregate command I see network files are missing in the Subset12 cluster.

Subset10/binding.tsv.gz Subset11/net_meanvar.tsv.gz Subset2/expression.tsv.gz Subset3/net_weight.tsv.gz Subset5/names_atac.txt Subset7/binding.tsv.gz Subset8/net_meanvar.tsv.gz Subset10/binlinking.tsv.gz Subset11/net_nweight.tsv.gz Subset2/footprints.bed Subset3/peaks.bed Subset5/names_rna.txt Subset7/binlinking.tsv.gz Subset8/net_nweight.tsv.gz Subset10/expression0.tsv.gz Subset11/net_stats.tsv.gz Subset2/homer.tsv.gz Subset3/reads.bai Subset5/net_covfactor.tsv.gz Subset7/expression0.tsv.gz Subset8/net_stats.tsv.gz Subset10/expression.tsv.gz Subset11/net_weight.tsv.gz Subset2/linking.tsv.gz Subset3/reads.bam Subset5/net_inweight.tsv.gz Subset7/expression.tsv.gz Subset8/net_weight.tsv.gz Subset10/footprints.bed Subset11/peaks.bed Subset2/motifs.bed Subset3/tssdist.tsv.gz Subset5/net_iweight.tsv.gz Subset7/footprints.bed Subset8/peaks.bed Subset10/homer.tsv.gz Subset11/reads.bai Subset2/names_atac0.txt Subset3/wellington.tsv.gz Subset5/net_loss.tsv.gz Subset7/homer.tsv.gz Subset8/reads.bai Subset10/linking.tsv.gz Subset11/reads.bam Subset2/names_atac.txt Subset4/binding.tsv.gz Subset5/net_meanvar.tsv.gz Subset7/linking.tsv.gz Subset8/reads.bam Subset10/motifs.bed Subset11/tssdist.tsv.gz Subset2/names_rna.txt Subset4/binlinking.tsv.gz Subset5/net_nweight.tsv.gz Subset7/motifs.bed Subset8/tssdist.tsv.gz Subset10/names_atac0.txt Subset11/wellington.tsv.gz Subset2/net_covfactor.tsv.gz Subset4/expression0.tsv.gz Subset5/net_stats.tsv.gz Subset7/names_atac0.txt Subset8/wellington.tsv.gz Subset10/names_atac.txt Subset12/expression0.tsv.gz Subset2/net_inweight.tsv.gz Subset4/expression.tsv.gz Subset5/net_weight.tsv.gz Subset7/names_atac.txt Subset9/binding.tsv.gz Subset10/names_rna.txt Subset12/names_atac0.txt Subset2/net_iweight.tsv.gz Subset4/footprints.bed Subset5/peaks.bed Subset7/names_rna.txt Subset9/binlinking.tsv.gz Subset10/net_covfactor.tsv.gz Subset12/names_rna.txt Subset2/net_loss.tsv.gz Subset4/homer.tsv.gz Subset5/reads.bai Subset7/net_covfactor.tsv.gz Subset9/expression0.tsv.gz Subset10/net_inweight.tsv.gz Subset1/binding.tsv.gz Subset2/net_meanvar.tsv.gz Subset4/linking.tsv.gz Subset5/reads.bam Subset7/net_inweight.tsv.gz Subset9/expression.tsv.gz Subset10/net_iweight.tsv.gz Subset1/binlinking.tsv.gz Subset2/net_nweight.tsv.gz Subset4/motifs.bed Subset5/tssdist.tsv.gz Subset7/net_iweight.tsv.gz Subset9/footprints.bed Subset10/net_loss.tsv.gz Subset1/expression0.tsv.gz Subset2/net_stats.tsv.gz Subset4/names_atac0.txt Subset5/wellington.tsv.gz Subset7/net_loss.tsv.gz Subset9/homer.tsv.gz Subset10/net_meanvar.tsv.gz Subset1/expression.tsv.gz Subset2/net_weight.tsv.gz Subset4/names_atac.txt Subset6/binding.tsv.gz Subset7/net_meanvar.tsv.gz Subset9/linking.tsv.gz Subset10/net_nweight.tsv.gz Subset1/footprints.bed Subset2/peaks.bed Subset4/names_rna.txt Subset6/binlinking.tsv.gz Subset7/net_nweight.tsv.gz Subset9/motifs.bed Subset10/net_stats.tsv.gz Subset1/homer.tsv.gz Subset2/reads.bai Subset4/net_covfactor.tsv.gz Subset6/expression0.tsv.gz Subset7/net_stats.tsv.gz Subset9/names_atac0.txt Subset10/net_weight.tsv.gz Subset1/linking.tsv.gz Subset2/reads.bam Subset4/net_inweight.tsv.gz Subset6/expression.tsv.gz Subset7/net_weight.tsv.gz Subset9/names_atac.txt Subset10/peaks.bed Subset1/motifs.bed Subset2/tssdist.tsv.gz Subset4/net_iweight.tsv.gz Subset6/footprints.bed Subset7/peaks.bed Subset9/names_rna.txt Subset10/reads.bai Subset1/names_atac0.txt Subset2/wellington.tsv.gz Subset4/net_loss.tsv.gz Subset6/homer.tsv.gz Subset7/reads.bai Subset9/net_covfactor.tsv.gz Subset10/reads.bam Subset1/names_atac.txt Subset3/binding.tsv.gz Subset4/net_meanvar.tsv.gz Subset6/linking.tsv.gz Subset7/reads.bam Subset9/net_inweight.tsv.gz Subset10/tssdist.tsv.gz Subset1/names_rna.txt Subset3/binlinking.tsv.gz Subset4/net_nweight.tsv.gz Subset6/motifs.bed Subset7/tssdist.tsv.gz Subset9/net_iweight.tsv.gz Subset10/wellington.tsv.gz Subset1/net_covfactor.tsv.gz Subset3/expression0.tsv.gz Subset4/net_stats.tsv.gz Subset6/names_atac0.txt Subset7/wellington.tsv.gz Subset9/net_loss.tsv.gz Subset11/binding.tsv.gz Subset1/net_inweight.tsv.gz Subset3/expression.tsv.gz Subset4/net_weight.tsv.gz Subset6/names_atac.txt Subset8/binding.tsv.gz Subset9/net_meanvar.tsv.gz Subset11/binlinking.tsv.gz Subset1/net_iweight.tsv.gz Subset3/footprints.bed Subset4/peaks.bed Subset6/names_rna.txt Subset8/binlinking.tsv.gz Subset9/net_nweight.tsv.gz Subset11/expression0.tsv.gz Subset1/net_loss.tsv.gz Subset3/homer.tsv.gz Subset4/reads.bai Subset6/net_covfactor.tsv.gz Subset8/expression0.tsv.gz Subset9/net_stats.tsv.gz Subset11/expression.tsv.gz Subset1/net_meanvar.tsv.gz Subset3/linking.tsv.gz Subset4/reads.bam Subset6/net_inweight.tsv.gz Subset8/expression.tsv.gz Subset9/net_weight.tsv.gz Subset11/footprints.bed Subset1/net_nweight.tsv.gz Subset3/motifs.bed Subset4/tssdist.tsv.gz Subset6/net_iweight.tsv.gz Subset8/footprints.bed Subset9/peaks.bed Subset11/homer.tsv.gz Subset1/net_stats.tsv.gz Subset3/names_atac0.txt Subset4/wellington.tsv.gz Subset6/net_loss.tsv.gz Subset8/homer.tsv.gz Subset9/reads.bai Subset11/linking.tsv.gz Subset1/net_weight.tsv.gz Subset3/names_atac.txt Subset5/binding.tsv.gz Subset6/net_meanvar.tsv.gz Subset8/linking.tsv.gz Subset9/reads.bam Subset11/motifs.bed Subset1/peaks.bed Subset3/names_rna.txt Subset5/binlinking.tsv.gz Subset6/net_nweight.tsv.gz Subset8/motifs.bed Subset9/tssdist.tsv.gz Subset11/names_atac0.txt Subset1/reads.bai Subset3/net_covfactor.tsv.gz Subset5/expression0.tsv.gz Subset6/net_stats.tsv.gz Subset8/names_atac0.txt Subset9/wellington.tsv.gz Subset11/names_atac.txt Subset1/reads.bam Subset3/net_inweight.tsv.gz Subset5/expression.tsv.gz Subset6/net_weight.tsv.gz Subset8/names_atac.txt Subset11/names_rna.txt Subset1/tssdist.tsv.gz Subset3/net_iweight.tsv.gz Subset5/footprints.bed Subset6/peaks.bed Subset8/names_rna.txt Subset11/net_covfactor.tsv.gz Subset1/wellington.tsv.gz Subset3/net_loss.tsv.gz Subset5/homer.tsv.gz Subset6/reads.bai Subset8/net_covfactor.tsv.gz Subset11/net_inweight.tsv.gz Subset2/binding.tsv.gz Subset3/net_meanvar.tsv.gz Subset5/linking.tsv.gz Subset6/reads.bam Subset8/net_inweight.tsv.gz Subset11/net_iweight.tsv.gz Subset2/binlinking.tsv.gz Subset3/net_nweight.tsv.gz Subset5/motifs.bed Subset6/tssdist.tsv.gz Subset8/net_iweight.tsv.gz Subset11/net_loss.tsv.gz Subset2/expression0.tsv.gz Subset3/net_stats.tsv.gz Subset5/names_atac0.txt Subset6/wellington.tsv.gz Subset8/net_loss.tsv.gz

What could be the reason and how to proceed further? Please suggest

I get the below error while running aggregate command:

WARNING:52151:2023-09-01 14:24:10,982:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory') WARNING:52151:2023-09-01 14:24:10,987:/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py:394:Skipping cell subset Subset12 due to error: FileNotFoundError(2, 'No such file or directory') Traceback (most recent call last): File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/main.py", line 13, in docstringrunner(package) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 340, in docstringrunner run_args(pkgname,funcs,args) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/docstring2argparse/init.py", line 330, in run_args return func(*a,ka) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/network.py", line 1071, in tofile n=network.from_folders(diri_data,diri_work,fi_subsets,dynamic=dynamic,nettype=nettype,optional=optional,fi_c=fi_c) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py", line 511, in from_folders return cls(params) File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py", line 99, in init self.check() File "/home/ssatpati/.conda/envs/dictys/lib/python3.9/site-packages/dictys/net/init.py", line 107, in check raise ValueError('Cell/state/node names must be non-empty, unique, and matching their counts exactly.') ValueError: Cell/state/node names must be non-empty, unique, and matching their counts exactly. make: *** [makefiles/common.mk:175: output/static.h5] Error 1

Thanks, Suresh

[lingfeiwang]

Hi Suresh,

Great that the CPU version is working at least. Missing subset 12 is fine. Actually this is the error:

ValueError: Cell/state/node names must be non-empty, unique, and matching their counts exactly.

Could you update Dictys to the latest version for a more detailed error message? You can find the instructions at https://github.com/pinellolab/dictys#updating-dictys. You can also try a different tutorial and see if the error disappears there.

Lingfei

[sureshsatpati]

Hi Lingfei, Thanks for the information!

Actually I removed the subset 12 from the subset.txt file and it worked for remaining subset. I was able to generate the dotplot for this. But the issue I see that when I generate heatmap for the TFs I can only see the color key legend and not any image. I used the command shown in tutorial (fig,fig_colorbar,net=fig_heatmap_top(d0,tf,ntop=ntop,direction=direction,aspect=1,gann='all'). Please suggest!

[lingfeiwang]

Hi Suresh,

Good to hear your feedbacks!

This seems like a jupyter-matplotlib compatibility issue. Can you try creating a notebook cell right after and run with the content:

fig

This should show the heatmap figure which is stored in variable fig. You should be able to use this trick to show any figure afterwards that is similarly hidden.

Lingfei

[sureshsatpati]

Hi Lingfei, Thanks for the response! Actually, I am running the code in the HPC cluster and I don't think I can see the image right away. But for earlier images I don't see any compatibility issue, as all images generated properly. I don't have any idea on creating notebook. Can you help with the notebook generation.

Suresh

[lingfeiwang]

Hi Suresh,

After GRN inference, you can download the h5 file to your local computer for visualization using this notebook.

Lingfei