search

fastai / nbdev

Create delightful software with Jupyter Notebooks
https://nbdev.fast.ai/
Apache License 2.0
4.93k stars 488 forks source link

nbdev_build_lib seems to convert more notebooks than expected #423

Closed MarcoGorelli closed 3 years ago

MarcoGorelli commented 3 years ago

I have cloned the nbdev template to https://github.com/MarcoGorelli/learning_nbdev

Then, in the terminal, I did:

$ git clone git@github.com:MarcoGorelli/learning_nbdev.git
$ cd learning_nbdev/
$ virtualenv venv
$ . venv/bin/activate
$ pip install ipykernel nbdev
$ python -m ipykernel install --user --name learning_nbdev
$ nbdev_install_git_hooks 
$ nbdev_build_lib
Converted 00_core.ipynb.
Converted index.ipynb.
Converted attachment.ipynb.
Converted notebook2.ipynb.
Converted pngmetadata.ipynb.
Converted prompt_numbers.ipynb.
Converted rawtest.ipynb.
Converted svg.ipynb.
Converted Check History in Memory.ipynb.
Converted Clear Output.ipynb.
Converted Disable Stdin.ipynb.
Converted Empty Cell.ipynb.
Converted Factorials.ipynb.
Converted HelloWorld.ipynb.
Converted Inline Image.ipynb.
Converted Interrupt-IPY6.ipynb.
Converted Interrupt.ipynb.
Converted JupyterWidgets.ipynb.
Converted Parallel Execute A.ipynb.
Converted Parallel Execute B.ipynb.
Converted SVG.ipynb.
Converted Skip Exceptions with Cell Tags-IPY6.ipynb.
Converted Skip Exceptions with Cell Tags.ipynb.
Converted Skip Exceptions-IPY6.ipynb.
Converted Skip Exceptions.ipynb.
Converted Sleep One.ipynb.
Converted Unicode.ipynb.
Converted UnicodePy3.ipynb.
Converted update-display-id.ipynb.
Converted Widget_List.ipynb.
Converted latex-linked-image.ipynb.
Converted markdown_display_priority.ipynb.
Converted notebook1.ipynb.
Converted notebook2.ipynb.
Converted notebook3_with_errors.ipynb.
Converted notebook4_jpeg.ipynb.
Converted notebook_jl.ipynb.
Converted notebook_tags.ipynb.
Converted empty.ipynb.

However, the tutorial shows that the output should just be

$ nbdev_build_lib
Converted 00_core.ipynb.
Converted index.ipynb.

Any idea what went wrong in my setup?

MarcoGorelli commented 3 years ago

The issue is that then, nbdev_build_lib will also try building all of these (irrelevant) notebooks:

$ nbdev_build_docs
converting: /home/marco/learning_nbdev/index.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/exporters/tests/files/prompt_numbers.ipynb
converting: /home/marco/learning_nbdev/00_core.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/exporters/tests/files/attachment.ipynb
[Errno 2] No such file or directory: '/home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/exporters/tests/files/attachment:image.png'
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/exporters/tests/files/rawtest.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/exporters/tests/files/svg.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/exporters/tests/files/pngmetadata.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/exporters/tests/files/notebook2.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/tests/files/notebook_jl.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/tests/files/notebook1.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/tests/files/notebook4_jpeg.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/tests/files/latex-linked-image.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/tests/files/markdown_display_priority.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/tests/files/notebook2.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/tests/files/notebook_tags.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/tests/files/Widget_List.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/tests/files/notebook3_with_errors.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/SVG.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Disable Stdin.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Skip Exceptions with Cell Tags-IPY6.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/UnicodePy3.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Inline Image.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Skip Exceptions-IPY6.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Unicode.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Check History in Memory.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/JupyterWidgets.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Sleep One.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Interrupt.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Clear Output.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/update-display-id.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/HelloWorld.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Factorials.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Skip Exceptions with Cell Tags.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Parallel Execute A.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Parallel Execute B.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Interrupt-IPY6.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Skip Exceptions.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbconvert/preprocessors/tests/files/Empty Cell.ipynb
converting: /home/marco/learning_nbdev/venv/lib/python3.8/site-packages/notebook/bundler/tests/resources/empty.ipynb
Conversion failed on the following:
attachment.ipynb
Traceback (most recent call last):
  File "/home/marco/learning_nbdev/venv/bin/nbdev_build_docs", line 8, in <module>
    sys.exit(nbdev_build_docs())
  File "/home/marco/learning_nbdev/venv/lib/python3.8/site-packages/fastcore/script.py", line 105, in _f
    tfunc(**merge(args, args_from_prog(func, xtra)))
  File "/home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbdev/export2html.py", line 665, in nbdev_build_docs
    if fname is None: make_sidebar()
  File "/home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbdev/export2html.py", line 730, in make_sidebar
    create_default_sidebar()
  File "/home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbdev/export2html.py", line 722, in create_default_sidebar
    dic = {Config().lib_name: _create_default_sidebar()}
  File "/home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbdev/export2html.py", line 714, in _create_default_sidebar
    titles = [_get_title(f) for f in fnames if f.stem!='index']
  File "/home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbdev/export2html.py", line 714, in <listcomp>
    titles = [_get_title(f) for f in fnames if f.stem!='index']
  File "/home/marco/learning_nbdev/venv/lib/python3.8/site-packages/nbdev/export2html.py", line 704, in _get_title
    with open(fname, 'r') as f: code = f.read()
FileNotFoundError: [Errno 2] No such file or directory: '/home/marco/learning_nbdev/docs/attachment.html'
MarcoGorelli commented 3 years ago

OK, got it, it's because it's trying to convert the notebooks in the venv folder.

By instead making my virtualenv outside this repo, it works as expected:

$ virtualenv ~/virtualenvs/learning_nbdev

Could/should these be excluded by default?

hamelsmu commented 3 years ago

Are you using the latest version of nbdev? Can you install the current version from github

pip install git+<URL>

hamelsmu commented 3 years ago

This shouldn't be happening if recursive is set to false in settings.ini. I'm not able to reproduce this issue, even with your repo. Or even if I put a notebook into a nested folder, etc.

My only hypothesis is your version of nbdev

MarcoGorelli commented 3 years ago

I was using the latest version of nbdev on PyPI (1.1.12)

hamelsmu commented 3 years ago

Can you try

pip install -U git+https://github.com/fastai/nbdev.git

hamelsmu commented 3 years ago

@MarcoGorelli can you provide a reproducible example that doesn't involve private files in your virtual env? A repo that we can use to reproduce this error would be helpful

MarcoGorelli commented 3 years ago

Hi @hamelsmu - sure, I've force-added the ipynb files to the repo: https://github.com/MarcoGorelli/learning_nbdev , please let me know if you're able to reproduce

I tried installing nbdev via pip install -U git+https://github.com/fastai/nbdev.git but the same thing happens

hamelsmu commented 3 years ago

Thanks for providing the example, that really helped. I fixed this in https://github.com/fastai/nbdev/pull/433

cc: @jph00