demo fails with "Error occurred while starting new kernel client for kernel 0287ef32-45b2-4f5d-8470-97ab4a40626a: Kernel didn't respond in 60 seconds"

[taylorreiter]

[Wed Jul 13 14:32:15 2022]
rule make_notebook_alignment:
    input: output.demo/stage2/LoombaR_2017__SID1050_bax__bin.11.fa.gz.x.GCF_002159845.mashmap.align, output.demo/stage2/LoombaR_2017__SID1050_bax__bin.11.fa.gz.x.GCF_
900199515.mashmap.align, output.demo/stage2/LoombaR_2017__SID1050_bax__bin.11.fa.gz.x.GCF_002160955.mashmap.align, output.demo/stage2/LoombaR_2017__SID1050_bax__bin.1
1.fa.gz.x.GCF_900104675.mashmap.align, output.demo/stage2/LoombaR_2017__SID1050_bax__bin.11.fa.gz.x.GCA_900066645.mashmap.align, output.demo/stage2/LoombaR_2017__SID1
050_bax__bin.11.fa.gz.x.GCF_002160025.mashmap.align, output.demo/stage2/LoombaR_2017__SID1050_bax__bin.11.fa.gz.x.GCF_900142265.mashmap.align, output.demo/stage2/Loom
baR_2017__SID1050_bax__bin.11.fa.gz.x.GCF_002159455.mashmap.align, charcoal/notebooks/report-alignment.ipynb, output.demo/stage2/LoombaR_2017__SID1050_bax__bin.11.fa.
gz.matches.json, output.demo/stage2/hitlist-accessions.info.csv, output.demo/.kernel.set
    output: output.demo/report/LoombaR_2017__SID1050_bax__bin.11.fa.gz.align.ipynb
    jobid: 92
    wildcards: g=LoombaR_2017__SID1050_bax__bin.11.fa.gz

Activating conda environment: /home/tereiter/github/test_tmp/charcoal/.snakemake/conda/c206c211943f949dafc61630804a06c4
Error occurred while starting new kernel client for kernel 0287ef32-45b2-4f5d-8470-97ab4a40626a: Kernel didn't respond in 60 seconds
Traceback (most recent call last):
  File "/home/tereiter/github/test_tmp/charcoal/.snakemake/conda/496585d5bca58acb629892cb82646aa0/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/tereiter/github/test_tmp/charcoal/.snakemake/conda/496585d5bca58acb629892cb82646aa0/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/tereiter/github/test_tmp/charcoal/.snakemake/conda/496585d5bca58acb629892cb82646aa0/lib/python3.7/site-packages/ipykernel_launcher.py", line 15, in <mod
ule>
    from ipykernel import kernelapp as app
  File "/home/tereiter/github/test_tmp/charcoal/.snakemake/conda/496585d5bca58acb629892cb82646aa0/lib/python3.7/site-packages/ipykernel/kernelapp.py", line 18, in <mo
dule>
    from IPython.core.application import (
  File "/home/tereiter/github/test_tmp/charcoal/.snakemake/conda/496585d5bca58acb629892cb82646aa0/lib/python3.7/site-packages/IPython/__init__.py", line 56, in <modul
e>
    from .terminal.embed import embed
  File "/home/tereiter/github/test_tmp/charcoal/.snakemake/conda/496585d5bca58acb629892cb82646aa0/lib/python3.7/site-packages/IPython/terminal/embed.py", line 17, in
<module>
    from IPython.terminal.ipapp import load_default_config
  File "/home/tereiter/github/test_tmp/charcoal/.snakemake/conda/496585d5bca58acb629892cb82646aa0/lib/python3.7/site-packages/IPython/terminal/ipapp.py", line 29, in
<module>
    from IPython.core.magics import (
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 963, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 906, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1280, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1252, in _get_spec

[taylorreiter]

https://github.com/nteract/papermill/issues/239

[bluegenes]

Since this is a papermill issue, one potential solution is to use the notebook directive in snakemake instead. It allows you to access wildcards, specific input files, etc. I think it works well with conda and just starts a kernel within the specified conda env (so no need to set the kernel specifically).

[taylorreiter]

IMHO +1, but would be curious what @ctb thinks!

[taylorreiter]

also FYI -- clean still works, so that's good. so this error only impacts python -m charcoal run demo/demo.conf -j 4 or report

[taylorreiter]

lol it works on mac...only failing on farm.