nbconvert with execute gives error

[dschneiderch]

i'm trying to execute and save my notebook but i keep getting an error. I tried --execute and then the more verbose --ExecutePreprocessor.enabled=True but neither worked. in the included example I also tried --ExecutePreprocessor.allow_errors=True. The conversion works if the notebook is not executed (both with nbconvert and from the gui).

i'm running homebrew jupyter 4.0.6 on OSX 10.9.5 any suggestions?

iMac:~/Documents/snotel-regression_project/src $ jupyter nbconvert --debug --ExecutePreprocessor.enabled=True --ExecutePreprocessor.allow_errors=True --to html compare_maxswe.ipynb
[NbConvertApp] Config changed:
[NbConvertApp] {'NbConvertApp': {'export_format': u'html', 'log_level': 10}, 'ExecutePreprocessor': {'allow_errors': True, 'enabled': True}}
[NbConvertApp] Searching [u'/Volumes/Dominik/Documents/snotel-regression_project/src', '/Users/dosc3612/.jupyter', '/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[NbConvertApp] Attempting to load config file jupyter_config.py in path /etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.json in path /etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.py in path /usr/local/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.json in path /usr/local/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.py in path /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.json in path /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.py in path /Users/dosc3612/.jupyter
[NbConvertApp] Attempting to load config file jupyter_config.json in path /Users/dosc3612/.jupyter
[NbConvertApp] Attempting to load config file jupyter_config.py in path /Volumes/Dominik/Documents/snotel-regression_project/src
[NbConvertApp] Attempting to load config file jupyter_config.json in path /Volumes/Dominik/Documents/snotel-regression_project/src
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.py in path /etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.json in path /etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.py in path /usr/local/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.json in path /usr/local/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.py in path /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.json in path /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.py in path /Users/dosc3612/.jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.json in path /Users/dosc3612/.jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.py in path /Volumes/Dominik/Documents/snotel-regression_project/src
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.json in path /Volumes/Dominik/Documents/snotel-regression_project/src
[NbConvertApp] Converting notebook compare_maxswe.ipynb to html
[NbConvertApp] Notebook name is 'compare_maxswe'
[NbConvertApp] Applying preprocessor: ExecutePreprocessor
[NbConvertApp] Executing notebook with kernel: ir
[NbConvertApp] Found kernel ir in /Users/dosc3612/Library/Jupyter/kernels
[NbConvertApp] Native kernel (python2) available from /usr/local/lib/python2.7/site-packages/ipykernel/resources
[NbConvertApp] Connecting to: tcp://127.0.0.1:64942
[NbConvertApp] connecting shell channel to tcp://127.0.0.1:64939
[NbConvertApp] Connecting to: tcp://127.0.0.1:64939
[NbConvertApp] connecting iopub channel to tcp://127.0.0.1:64940
[NbConvertApp] Connecting to: tcp://127.0.0.1:64940
[NbConvertApp] connecting stdin channel to tcp://127.0.0.1:64941
[NbConvertApp] Connecting to: tcp://127.0.0.1:64941
[NbConvertApp] connecting heartbeat channel to tcp://127.0.0.1:64943
> IRkernel::main()
[NbConvertApp] Executing cell:
setwd('~/Documents/snotel-regression_project')
basepath='reports/sweinterp_compare'
recon.version='v3.1'
config=''
covrange='idp1'
snotelscale='scale'
dateflag='B'
cost='r2'
fscaMatch='wofsca'
resid=''
if(resid==''){
    residblending='unblended'
} else {
    residblending='blended'
}
product='phv'
inpath=file.path(basepath,paste(product,fscaMatch,snotelscale,dateflag,cost,residblending,sep='-'))
inpath
[NbConvertApp] ERROR | failed to run cell: KeyError('status',)
[NbConvertApp] ERROR | setwd('~/Documents/snotel-regression_project')
basepath='reports/sweinterp_compare'
recon.version='v3.1'
config=''
covrange='idp1'
snotelscale='scale'
dateflag='B'
cost='r2'
fscaMatch='wofsca'
resid=''
if(resid==''){
    residblending='unblended'
} else {
    residblending='blended'
}
product='phv'
inpath=file.path(basepath,paste(product,fscaMatch,snotelscale,dateflag,cost,residblending,sep='-'))
inpath
Traceback (most recent call last):
  File "/usr/local/bin/jupyter-nbconvert", line 6, in <module>
    main()
  File "/usr/local/lib/python2.7/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/traitlets/config/application.py", line 592, in launch_instance
    app.start()
  File "/usr/local/lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 286, in start
    self.convert_notebooks()
  File "/usr/local/lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 409, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 380, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 332, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/exporters/exporter.py", line 166, in from_filename
    return self.from_notebook_node(nbformat.read(f, as_version=4), resources=resources, **kw)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/exporters/html.py", line 65, in from_notebook_node
    return super(HTMLExporter, self).from_notebook_node(nb, resources, **kw)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/exporters/templateexporter.py", line 198, in from_notebook_node
    nb_copy, resources = super(TemplateExporter, self).from_notebook_node(nb, resources, **kw)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/exporters/exporter.py", line 131, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/exporters/exporter.py", line 303, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb,resources)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/preprocessors/execute.py", line 83, in preprocess
    nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/preprocessors/base.py", line 70, in preprocess
    nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/preprocessors/execute.py", line 97, in preprocess_cell
    outputs = self.run_cell(cell)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/preprocessors/execute.py", line 128, in run_cell
    if msg['metadata']['status'] == 'error' and not self.allow_errors:
KeyError: 'status'

[minrk]

This is a bug in 4.0, and fixed in master. It shouldn't be looking at msg['metadata'] at all.

[dschneiderch]

Great, Thanks! is there a target date for 4.1?

[dschneiderch]

so I installed the git versions of nbconvert and nbformat but it still didn't work (although things went farther). not sure if there are other dependencies that need to be updated though.

 pip install git+https://github.com/jupyter/nbconvert.git

iMac:~/Documents/snotel-regression_project/src $ jupyter nbconvert --debug --execute  --to html compare_maxswe.ipynb
[NbConvertApp] Config changed:
[NbConvertApp] {'NbConvertApp': {'export_format': u'html', 'log_level': 10}, 'ExecutePreprocessor': {'enabled': True}}
[NbConvertApp] Searching [u'/Volumes/Dominik/Documents/snotel-regression_project/src', '/Users/dosc3612/.jupyter', '/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[NbConvertApp] Attempting to load config file jupyter_config.py in path /etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.json in path /etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.py in path /usr/local/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.json in path /usr/local/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.py in path /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.json in path /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_config.py in path /Users/dosc3612/.jupyter
[NbConvertApp] Attempting to load config file jupyter_config.json in path /Users/dosc3612/.jupyter
[NbConvertApp] Attempting to load config file jupyter_config.py in path /Volumes/Dominik/Documents/snotel-regression_project/src
[NbConvertApp] Attempting to load config file jupyter_config.json in path /Volumes/Dominik/Documents/snotel-regression_project/src
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.py in path /etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.json in path /etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.py in path /usr/local/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.json in path /usr/local/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.py in path /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.json in path /usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/etc/jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.py in path /Users/dosc3612/.jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.json in path /Users/dosc3612/.jupyter
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.py in path /Volumes/Dominik/Documents/snotel-regression_project/src
[NbConvertApp] Attempting to load config file jupyter_nbconvert_config.json in path /Volumes/Dominik/Documents/snotel-regression_project/src
[NbConvertApp] Converting notebook compare_maxswe.ipynb to html
[NbConvertApp] Notebook name is 'compare_maxswe'
[NbConvertApp] Applying preprocessor: ExecutePreprocessor
[NbConvertApp] Executing notebook with kernel: ir
[NbConvertApp] Found kernel ir in /Users/dosc3612/Library/Jupyter/kernels
[NbConvertApp] Native kernel (python2) available from /usr/local/lib/python2.7/site-packages/ipykernel/resources
[NbConvertApp] Connecting to: tcp://127.0.0.1:57955
[NbConvertApp] connecting shell channel to tcp://127.0.0.1:57952
[NbConvertApp] Connecting to: tcp://127.0.0.1:57952
[NbConvertApp] connecting iopub channel to tcp://127.0.0.1:57953
[NbConvertApp] Connecting to: tcp://127.0.0.1:57953
[NbConvertApp] connecting stdin channel to tcp://127.0.0.1:57954
[NbConvertApp] Connecting to: tcp://127.0.0.1:57954
[NbConvertApp] connecting heartbeat channel to tcp://127.0.0.1:57956
> IRkernel::main()
[NbConvertApp] Executing cell:
setwd('~/Documents/snotel-regression_project')
basepath='reports/sweinterp_compare'
recon.version='v3.1'
config=''
covrange='idp1'
snotelscale=Sys.getenv('snotelscale')#'scale'
print(snotelscale)
dateflag='B'
cost='r2'
fscaMatch='wofsca'
resid=''
if(resid==''){
    residblending='unblended'
} else {
    residblending='blended'
}
product='phv'
inpath=file.path(basepath,paste(product,fscaMatch,snotelscale,dateflag,cost,residblending,sep='-'))
inpath
[NbConvertApp] output: status
[NbConvertApp] output: execute_input
[NbConvertApp] output: stream
[NbConvertApp] output: execute_result
[NbConvertApp] output: status
[NbConvertApp] Executing cell:
library(raster)
library(rasterVis)
library(ggplot2)
library(gridExtra)
library(dplyr)
library(tidyr)
[NbConvertApp] output: status
[NbConvertApp] output: execute_input
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: stream
[NbConvertApp] output: status
[NbConvertApp] Executing cell:
rafiles=list.files(pattern='reanalysis.tif',path=inpath,full.names=T)
rtfiles=list.files(pattern='real-time.tif',path=inpath,full.names=T)
[NbConvertApp] output: status
[NbConvertApp] output: execute_input
[NbConvertApp] output: status
[NbConvertApp] Executing cell:
rastack=stack(rafiles)[[-1]]
rtstack=stack(rtfiles)

# str(as.data.frame(stack(rastack,rtstack),xy=T))
swedF=as.data.frame(stack(rastack,rtstack),xy=T) %>%
    gather(raster,swe,-x,-y) %>%
    separate(raster,into=c('yr','style'),sep='\\.') %>%
# head(swedF)
    mutate(
        yr=extract_numeric(yr))
[NbConvertApp] output: status
[NbConvertApp] output: execute_input
[NbConvertApp] output: error
[NbConvertApp] output: error
[NbConvertApp] output: error
[NbConvertApp] output: status
[NbConvertApp] ERROR | Error while converting 'compare_maxswe.ipynb'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/nbconvert/nbconvertapp.py", line 335, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/exporters/exporter.py", line 165, in from_filename
    return self.from_notebook_node(nbformat.read(f, as_version=4), resources=resources, **kw)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/exporters/html.py", line 65, in from_notebook_node
    return super(HTMLExporter, self).from_notebook_node(nb, resources, **kw)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/exporters/templateexporter.py", line 197, in from_notebook_node
    nb_copy, resources = super(TemplateExporter, self).from_notebook_node(nb, resources, **kw)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/exporters/exporter.py", line 130, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/exporters/exporter.py", line 302, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb,resources)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/preprocessors/execute.py", line 83, in preprocess
    nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/preprocessors/base.py", line 70, in preprocess
    nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
  File "/usr/local/lib/python2.7/site-packages/nbconvert/preprocessors/execute.py", line 112, in preprocess_cell
    raise CellExecutionError(msg)
CellExecutionError
[NbConvertApp] Exiting application: jupyter-nbconvert

[minrk]

@dschneiderch yes, I think it should be ready pretty soon.