Closed koenvandenberge closed 4 years ago
I have not used pySCENIC on slurm, but this must be something specific to your HPC, I would guess. It's clearly exceeding some memory limitation (a per process limit perhaps?). For this dataset, I'd expect maybe 2 or 3GB used per process, which isn't much. Have you tried lowering the number of processes?
You could also try the command line version instead of running it interactively (from a shell, check out pyscenic ctx -h
).
If you just want to look at the TF-gene modules from the GRN without running the pruning step, you should be able to get it from the previous command:
modules = list(modules_from_adjacencies(adjacencies, exprMat))
but this won't provide any of the benefit of module pruning to generate regulons, of course.
Thanks for the swift reply. I deliberately did not use any parallelization here in an attempt to try and find out what was going wrong, and since the memory tracking is showing me that memory consumption is actually rather low, I was thinking the error may originate from something else, rather than a true memory issue. I will look into the command line version.
I am trying the CLI right now, where it starts running, but I seem to be getting an error related to auc. Any ideas?
(scenic2) gandalf.koenvdberge$ pyscenic ctx -o "/accounts/campus/koenvdberge/Documents/CRISPRi/pyScenic_cli/output2_prune" --annotations_fname "/accounts/campus/koenvdberge/Documents/SCENIC/motif2tf/mouse/motifs-v9-nr.mgi-m0.001-o0.0.tbl" "/accounts/campus/koenvdberge/Documents/CRISPRi/pyScenicResults/modules.dat" "/accounts/campus/koenvdberge/Documents/SCENIC/cisTarget_databases/mm9-tss-centered-10kb-7species.mc9nr.feather" "/accounts/campus/koenvdberge/Documents/SCENIC/cisTarget_databases/mm9-500bp-upstream-7species.mc9nr.feather"
/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/dask/config.py:161: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
data = yaml.load(f.read()) or {}
2020-04-20 20:48:59,451 - pyscenic.cli.pyscenic - INFO - Loading modules.
2020-04-20 20:49:01,867 - pyscenic.cli.pyscenic - INFO - Loading databases.
2020-04-20 20:49:01,867 - pyscenic.cli.pyscenic - INFO - Calculating regulons.
[ ] | 0% Completed | 25.1s
2020-04-20 20:49:29,803 - pyscenic.transform - WARNING - Less than 80% of the genes in Regulon for EEF1D could be mapped to mm9-tss-centered-10kb-7species.mc9nr. Skipping this module.
[ ] | 0% Completed | 27.3s
2020-04-20 20:49:31,903 - pyscenic.transform - WARNING - Less than 80% of the genes in Regulon for EGR1 could be mapped to mm9-tss-centered-10kb-7species.mc9nr. Skipping this module.
[ ] | 0% Completed | 39.8s
2020-04-20 20:49:44,471 - pyscenic.transform - WARNING - Less than 80% of the genes in Regulon for HNRNPC could be mapped to mm9-tss-centered-10kb-7species.mc9nr. Skipping this module.
[ ] | 0% Completed | 42.5s
2020-04-20 20:49:47,100 - pyscenic.transform - WARNING - Less than 80% of the genes in Regulon for HNRNPH3 could be mapped to mm9-tss-centered-10kb-7species.mc9nr. Skipping this module.
[ ] | 0% Completed | 47.4s
2020-04-20 20:49:52,063 - pyscenic.transform - WARNING - Less than 80% of the genes in Regulon for LARP1 could be mapped to mm9-tss-centered-10kb-7species.mc9nr. Skipping this module.
[ ] | 0% Completed | 51.8s
Traceback (most recent call last):
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/bin/pyscenic", line 8, in <module>
sys.exit(main())
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 408, in main
args.func(args)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 159, in prune_targets_command
num_workers=args.num_workers)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/prune.py", line 351, in prune2df
num_workers, module_chunksize)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/prune.py", line 300, in _distributed_calc
return create_graph().compute(scheduler='processes', num_workers=num_workers if num_workers else cpu_count())
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/dask/base.py", line 156, in compute
(result,) = compute(self, traverse=False, **kwargs)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/dask/base.py", line 397, in compute
results = schedule(dsk, keys, **kwargs)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/dask/multiprocessing.py", line 192, in get
raise_exception=reraise, **kwargs)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/dask/local.py", line 501, in get_async
raise_exception(exc, tb)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/dask/compatibility.py", line 112, in reraise
raise exc
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/dask/local.py", line 272, in execute_task
result = _execute_task(task, data)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/dask/local.py", line 252, in _execute_task
args2 = [_execute_task(a, cache) for a in args]
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/dask/local.py", line 252, in <listcomp>
args2 = [_execute_task(a, cache) for a in args]
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/dask/local.py", line 253, in _execute_task
return func(*args2)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/transform.py", line 231, in modules2df
for module in modules])
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/transform.py", line 231, in <listcomp>
for module in modules])
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/transform.py", line 185, in module2df
weighted_recovery=weighted_recovery)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/transform.py", line 129, in module2features_auc1st_impl
aucs = calc_aucs(df, db.total_genes, weights, auc_threshold)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/recovery.py", line 284, in aucs
assert maxauc > 0
AssertionError
It looks like your gene matrix contains human (HGNC gene symbols) gene names instead of mouse (MGI gene symbols), so in that case use the human databases.
Hence the Less than 80% of the genes in Regulon for EEF1D could be mapped to mm9-tss-centered-10kb-7species.mc9nr
warnings.
Thanks, working with the command line interface instead worked for me in avoiding the errors.
You were right that I had selected the mouse database while I should have selected the human one. However, a lot of these Less than 80% of the genes
messages keep popping up which results in a low number of TFs after pruning. Is there a way TFs must be defined, eg uppercase vs lowercase or something I could check?
Gene names for Human should be in HGNC gene symbols (uppercase):
Thanks, that's indeed what I am using, but I am still getting abundant warnings. Is that expected?
Here is my code
pyscenic grn -t -m grnboost2 -o "/accounts/campus/koenvdberge/Documents/CRISPRi/pyScenic_cli/output1_grn" "/accounts/campus/koenvdberge/Documents/CRISPRi/data/counts_nogRNA.csv" "/accounts/campus/koenvdberge/Documents/CRISPRi/tf_names.csv"
pyscenic ctx -o "/accounts/campus/koenvdberge/Documents/CRISPRi/pyScenic_cli/output2_prune.dat" --annotations_fname "/accounts/campus/koenvdberge/Documents/SCENIC/motif2tf/human/motifs-v9-nr.hgnc-m0.001-o0.0.tbl" "/accounts/campus/koenvdberge/Documents/CRISPRi/pyScenic_cli/output1_grn.csv" "/accounts/campus/koenvdberge/Documents/SCENIC/cisTarget_databases/human/hg19-500bp-upstream-7species.mc9nr.feather" "/accounts/campus/koenvdberge/Documents/SCENIC/cisTarget_databases/human/hg19-tss-centered-10kb-7species.mc9nr.feather"
pyscenic aucell -t -o "/accounts/campus/koenvdberge/Documents/CRISPRi/pyScenic_cli/output3_aucell" "/accounts/campus/koenvdberge/Documents/CRISPRi/data/counts_nogRNA.csv" "/accounts/campus/koenvdberge/Documents/CRISPRi/pyScenic_cli/output2_prune.dat"
Could you post a section of your gene names?
Sure, here's some gene names:
"TTTY17B" "TTTY17C" "TTTY17A" "TTTY4C" "TTTY4B" "TTTY4" "BPY2C" "BPY2B" "BPY2" "DAZ4" "DAZ1" "DAZ3" "DAZ2" "TTTY3B" "TTTY3" "CDY1B" "CDY1" "CSPG4P1Y" "GOLGA2P3Y" "GOLGA2P2Y"
and TF names:
HOXA9
ZFP128
ZFP853
NR1H2
NR1H3
NR1H4
NR1H5
NR1I2
I think the issue stems from the fact that you have long non-coding RNAs in your expression matrix. Our databases only contain normal genes (mostly protein coding, there might be some pseudogenes in there too), but not lincRNAs, microRNAs, ...RNA. https://www.genenames.org/tools/search/#!/all?query=TTTY4C
Thanks, so I suspect it would be better to remove these prior to running pySCENIC
such that the regulons aren't 'contaminated' with non-protein coding genes.
Yes remove them before running the first step.
Hi, thanks for all the help so far.
Restricting the analysis to protein-coding genes only indeed reduces the Less than 80% of the genes
messages. I am interested in an analysis without pruning, which seems to run successfully to some point until I get the following error I couldn't diagnose the origin from.
Here's the command I've used:
pyscenic ctx -t -n -o "/accounts/campus/koenvdberge/Documents/CRISPRi/pyScenic_cli/output2_noPrune.csv" --annotations_fname "/accounts/campus/koenvdberge/Documents/SCENIC/motif2tf/human/motifs-v9-nr.hgnc-m0.001-o0.0.tbl" --num_workers 8 --expression_mtx_fname "/accounts/campus/koenvdberge/Documents/CRISPRi/data/counts_codingGenes_noQuotes.csv" "/accounts/campus/koenvdberge/Documents/CRISPRi/pyScenic_cli/output1_grn.csv" "/accounts/campus/koenvdberge/Documents/SCENIC/cisTarget_databases/human/hg19-500bp-upstream-7species.mc9nr.feather" "/accounts/campus/koenvdberge/Documents/SCENIC/cisTarget_databases/human/hg19-tss-centered-10kb-7species.mc9nr.feather"
And the error:
[####################################### ] | 99% Completed | 20hr 14min 8.3s
[########################################] | 100% Completed | 20hr 14min 8.4s
Traceback (most recent call last):
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/bin/pyscenic", line 8, in <module>
sys.exit(main())
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 408, in main
args.func(args)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 159, in prune_targets_command
num_workers=args.num_workers)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/prune.py", line 380, in find_features
filter_for_annotation=False, **kwargs), base_url=motif_base_url)
File "/accounts/campus/koenvdberge/.conda/envs/scenic2/lib/python3.7/site-packages/pyscenic/utils.py", line 316, in add_motif_url
df[("Enrichment", COLUMN_NAME_MOTIF_URL)] = list(map(partial(urljoin, base=base_url), df.index.get_level_values(COLUMN_NAME_MOTIF_ID)))
TypeError: urljoin() got multiple values for argument 'base'
Hi all, Thanks for looking into this; seeing the recent commits mentioning this issue, would you consider it safe to proceed or is work on a fix still ongoing?
Thank you for the upates.
Hi @koenvandenberge ,
The dev branch should now have a fix to skip modules with no genes overlapping the ranking database (this also would be solved when you removed non protein coding genes). But, sorry for not following up on your last issue (with urljoin
). It seems like it was a bug in the calling function for the no pruning method (which we very rarely use here). I've pushed another fix to the dev branch, which you could install via:
pip install git+https://github.com/aertslab/pySCENIC.git@dev
I have been using pySCENIC on a few datasets locally but have been looking into larger datasets that required me to move to an HPC infrastructure. My dataset is not huge, around 10k cells and 17k genes. I have been able to successfully run the
grnboost2
step, and have saved the output for that. However, I keep running into issues when I try pruning the modules. I have anOSError: [Errno 12] Cannot allocate memory
error, as I show below, even though the job does not exceed 8% of the available memory usage.