Error when running `run.sh`

[Shashankti]

Hey, I tried to follow the steps to reproduce the results but I get an error when running the script -2_10x_processing.py which causes the next stesp to fail as well. Here is the error which I get when trying to convert to notebook.:

Running:  02_10x_processing
[jupytext] Reading 02_10x_processing.py in format py
[jupytext] Writing 02_10x_processing.ipynb (destination file replaced [use --update to preserve cell outputs and ids])
[jupytext] Updating the timestamp of 02_10x_processing.py
[NbConvertApp] Converting notebook 02_10x_processing.ipynb to notebook
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/nbclient/client.py", line 841, in async_execute_cell
    exec_reply = await self.task_poll_for_reply
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/jupyter-nbconvert", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/traitlets/config/application.py", line 845, in launch_instance
    app.start()
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/nbconvertapp.py", line 346, in start
    self.convert_notebooks()
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/nbconvertapp.py", line 518, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/nbconvertapp.py", line 483, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/nbconvertapp.py", line 412, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/exporter.py", line 199, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/exporters/exporter.py", line 318, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/preprocessors/execute.py", line 84, in preprocess
    self.preprocess_cell(cell, resources, index)
  File "/usr/local/lib/python3.8/dist-packages/nbconvert/preprocessors/execute.py", line 105, in preprocess_cell
    cell = self.execute_cell(cell, index, store_history=True)
  File "/usr/local/lib/python3.8/dist-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/usr/local/lib/python3.8/dist-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/nbclient/client.py", line 845, in async_execute_cell
    raise DeadKernelError("Kernel died")
nbclient.exceptions.DeadKernelError: Kernel died

I am also attaching the complete output of the run here. Thanks for the help

error.txt

[Iwo-K]

Hi @Shashankti, Thanks for following our work! It looks like kernel dies during execution of the 02_10x_processing.py code. This output has the nbconvert traceback but no errors from the actual notebook contents so it is difficult to diagnose. My guess would be that your machine may be running out of memory, which would cause the kernel to die. 02_10x_processing.py is quite memory intensive, I don't have a specific minimal number but I would typically use machines with >80GB of RAM. Could this be the issue here?