search

sphinx-gallery / sphinx-gallery

Sphinx extension for automatic generation of an example gallery
https://sphinx-gallery.github.io
BSD 3-Clause "New" or "Revised" License
415 stars 201 forks source link

[sphinx 1.5] _gdbm.error No such file or directory #186

Closed sciunto closed 7 years ago

sciunto commented 7 years ago

@Titan-C It looks like there is still some issues... I updated sphinx-gallery to 0.1.7.

I copy paste the log file. Travis job is here: https://travis-ci.org/scikit-image/scikit-image/jobs/183569010

# Sphinx version: 1.5
# Python version: 3.5.2 (CPython)
# Docutils version: 0.13.1 release
# Jinja2 version: 2.8
# Last messages:
#   
#   copying static files...
#   done
#   copying extra files...
#   done
#   dumping search index in English (code: en) ...
#   done
#   dumping object inventory...
#   done
#   build succeeded.
# Loaded extensions:
#   sphinx.builders.xml (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/xml.py
#   sphinx.builders.html (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/html.py
#   sphinx.domains.python (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/domains/python.py
#   sphinx.builders.latex (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/latex.py
#   sphinx.builders.epub3 (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/epub3.py
#   sphinx.builders.gettext (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/gettext.py
#   sphinx.builders.texinfo (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/texinfo.py
#   sphinx.builders.htmlhelp (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/htmlhelp.py
#   matplotlib.sphinxext.plot_directive (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/matplotlib/sphinxext/plot_directive.py
#   sphinx.ext.linkcode (1.5) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/ext/linkcode.py
#   sphinx.ext.intersphinx (1.5) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/ext/intersphinx.py
#   sphinx.builders.qthelp (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/qthelp.py
#   alabaster (0.7.9) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/alabaster/__init__.py
#   sphinx.domains.javascript (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/domains/javascript.py
#   sphinx.domains.std (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/domains/std.py
#   sphinx.domains.cpp (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/domains/cpp.py
#   sphinx.builders.dummy (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/dummy.py
#   sphinx.builders.changes (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/changes.py
#   sphinx.builders.manpage (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/manpage.py
#   sphinx.roles (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/roles.py
#   sphinx.directives.code (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/directives/code.py
#   sphinx.domains.rst (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/domains/rst.py
#   sphinx.builders.applehelp (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/applehelp.py
#   numpydoc (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/numpydoc/__init__.py
#   sphinx.directives.other (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/directives/other.py
#   sphinx.builders.websupport (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/websupport.py
#   sphinx.domains.c (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/domains/c.py
#   sphinx.directives.patches (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/directives/patches.py
#   sphinx_gallery.gen_gallery (unknown version) from /home/fr/github/scikit-image/doc/source/../ext/sphinx_gallery/gen_gallery.py
#   sphinx.ext.autosummary (1.5) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/ext/autosummary/__init__.py
#   sphinx.ext.autodoc (1.5) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/ext/autodoc.py
#   sphinx.builders.linkcheck (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/linkcheck.py
#   sphinx.directives (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/directives/__init__.py
#   sphinx.builders.devhelp (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/devhelp.py
#   sphinx.builders.epub (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/epub.py
#   sphinx.builders.text (unknown version) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/builders/text.py
#   sphinx.ext.imgmath (1.5) from /home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/ext/imgmath.py
Traceback (most recent call last):
  File "/home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/cmdline.py", line 296, in main
    app.build(opts.force_all, filenames)
  File "/home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/application.py", line 351, in build
    self.emit('build-finished', None)
  File "/home/fr/.local/share/virtualenvs/sk/lib/python3.5/site-packages/sphinx/application.py", line 589, in emit
    results.append(callback(self, *args))
  File "/home/fr/github/scikit-image/doc/source/../ext/sphinx_gallery/docs_resolv.py", line 456, in embed_code_links
    _embed_code_links(app, gallery_conf, gallery_dir)
  File "/home/fr/github/scikit-image/doc/source/../ext/sphinx_gallery/docs_resolv.py", line 343, in _embed_code_links
    gallery_dir)
  File "/home/fr/github/scikit-image/doc/source/../ext/sphinx_gallery/docs_resolv.py", line 214, in __init__
    sindex = get_data(searchindex_url, gallery_dir)
  File "/home/fr/github/scikit-image/doc/source/../ext/sphinx_gallery/docs_resolv.py", line 62, in get_data
    search_index = shelve.open(cached_file)
  File "/usr/lib64/python3.5/shelve.py", line 243, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/lib64/python3.5/shelve.py", line 227, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/lib64/python3.5/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 2] No such file or directory
lesteve commented 7 years ago

I do get an error as well on your scikit-image PR branch, but with an error which is a bit more informative (if you follow the traceback the stack trace is very similar):

Traceback (most recent call last):
  File "/home/lesteve/miniconda3/envs/scikit-image/lib/python3.5/dbm/dumb.py", line 81, in _create
    f = _io.open(self._datfile, 'r', encoding="Latin-1")
FileNotFoundError: [Errno 2] No such file or directory: 'auto_examples/searchindex.dat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lesteve/miniconda3/envs/scikit-image/lib/python3.5/site-packages/Sphinx-1.5-py3.5.egg/sphinx/cmdline.py", line 296, in 
    app.build(opts.force_all, filenames)
  File "/home/lesteve/miniconda3/envs/scikit-image/lib/python3.5/site-packages/Sphinx-1.5-py3.5.egg/sphinx/application.py", line 351,
    self.emit('build-finished', None)
  File "/home/lesteve/miniconda3/envs/scikit-image/lib/python3.5/site-packages/Sphinx-1.5-py3.5.egg/sphinx/application.py", line 589,
    results.append(callback(self, *args))
  File "/home/lesteve/dev/scikit-image/doc/source/../ext/sphinx_gallery/docs_resolv.py", line 456, in embed_code_links
    _embed_code_links(app, gallery_conf, gallery_dir)
  File "/home/lesteve/dev/scikit-image/doc/source/../ext/sphinx_gallery/docs_resolv.py", line 343, in _embed_code_links
    gallery_dir)
  File "/home/lesteve/dev/scikit-image/doc/source/../ext/sphinx_gallery/docs_resolv.py", line 214, in __init__
    sindex = get_data(searchindex_url, gallery_dir)
  File "/home/lesteve/dev/scikit-image/doc/source/../ext/sphinx_gallery/docs_resolv.py", line 62, in get_data
    search_index = shelve.open(cached_file)
  File "/home/lesteve/miniconda3/envs/scikit-image/lib/python3.5/shelve.py", line 243, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/home/lesteve/miniconda3/envs/scikit-image/lib/python3.5/shelve.py", line 227, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/home/lesteve/miniconda3/envs/scikit-image/lib/python3.5/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
  File "/home/lesteve/miniconda3/envs/scikit-image/lib/python3.5/dbm/dumb.py", line 303, in open
    return _Database(file, mode, flag=flag)
  File "/home/lesteve/miniconda3/envs/scikit-image/lib/python3.5/dbm/dumb.py", line 69, in __init__
    self._create(flag)
  File "/home/lesteve/miniconda3/envs/scikit-image/lib/python3.5/dbm/dumb.py", line 83, in _create
    with _io.open(self._datfile, 'w', encoding="Latin-1") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'auto_examples/searchindex.dat'
lesteve commented 7 years ago

It could well be related to https://github.com/sphinx-gallery/sphinx-gallery/issues/115. I seem to remember that scikit-image had some specific config that was different than the default.

I am guessing we should try to create searchindex.dat in source/auto_examples/searchindex.data in the case of scikit-image.

sciunto commented 7 years ago

@lesteve Thanks for your quick reply.

Why would you like to create a file in source/* during a build procedure? This would go against the idea of the configuration to generate all the files in a separate directory (namely build), no?

lesteve commented 7 years ago

Why would you like to create a file in source/* during a build procedure? This would go against the idea of the configuration to generate all the files in a separate directory (namely build), no?

I am not sure why but this is definitely what the code is doing. IIRC we are using the default settings in sphinx-start and we don't have this distinction of build and source folders.

This is what #115 was talking about indeed, @emmanuelle was saying:

I'm wondering if some developers/users have already successfully used sphinx-gallery for projects that have a separate source directory (when you answer "yes" to the following question in sphinx-quickstart " > Separate source and build directories (y/n) [n]: y")

Full disclosure this is the kind of sphinx details that I have no clue about, if you have references that explain this a bit I am interested. It seems like we fixed #115 through a kludge and we need to understand this build/source folder more if we want to support it going forward.

Titan-C commented 7 years ago

If related to #115 then I think scikit-image has an extra patch to handle this case that got washed in your update @sciunto. Look at line https://github.com/scikit-image/scikit-image/commit/73ec69c15f7d1d3433cdb53afb457f26c1ea4471#diff-70a0de33ac06adccedd71571a25f2b60L320 You also need to restore that change of directory in https://github.com/scikit-image/scikit-image/commit/73ec69c15f7d1d3433cdb53afb457f26c1ea4471#diff-70a0de33ac06adccedd71571a25f2b60L404 Can you test if this manual patch works for you?

This issue of configuring sphinx gallery for different directory structures or launching the build from a directory different that the default one is not completely supported by the gallery yet. I started a PR #133, to go fix this but got way more complicated that it should have. I'll shall revisit it again.

lesteve commented 7 years ago

@sciunto not sure how you update your sphinx-gallery copy but using the copy_sphinxgallery.sh script rather than doing it manually is probably a good idea (copy_sphinxgallery.sh should be in your PATH if you did pip install sphinx-gallery).

sciunto commented 7 years ago

@lesteve sphinx-gallery is shipped with scikit-image. our copy is supposed to correspond to upstream (we submit patches as needed). From what @Titan-C said, it seems that some of our modifications have not been merged in your branch.

I ping @emmanuelle as she set up sphinx-gallery and I'm pretty sure to be the first one who modified that since then.

Our procedure is not optimal, but I think there was a good reason behind it (like you didn't released modifications that we needed). @emmanuelle Am I right?

lesteve commented 7 years ago

Our procedure is not optimal, but I think there was a good reason behind it (like you didn't released modifications that we needed). @emmanuelle Am I right?

Just to be clear, scikit-image is a first class citizen as far as sphinx-gallery is concerned, so do not hesitate to open issues if you encounter problems or submit your patches as PRs. We'll do our best to tackle them!

sciunto commented 7 years ago

@lesteve Yes, normally, our policy is to submit all our modifications to upstream :) Let's wait for @emmanuelle 's answer, she is the expert.

sciunto commented 7 years ago

After some discussions with @emmanuelle here: https://github.com/scikit-image/scikit-image/pull/2395 It seems that it is a regression on your side. All our modifications have been submitted and merged to master when we integrated the gallery for the first time.

lesteve commented 7 years ago

Indeed I can see that your sphinx-gallery copy is sphinx-gallery 0.1.7 without any modification. Hmmm we really need to have better test coverage for this.

Titan-C commented 7 years ago

Hmmm we really need to have better test coverage for this.

Yes. We don't have testing for building the gallery out of our default structure. From what I see is that scikit-image when propagating their changes to Sphinx-gallery, we are missing their change of directory when resolving links, which is done after building the gallery and all html

lesteve commented 7 years ago

@sciunto this seems to fix (tested on your scikit-image PR):

diff --git a/sphinx_gallery/docs_resolv.py b/sphinx_gallery/docs_resolv.py
index b2757d2..0ee5120 100644
--- a/sphinx_gallery/docs_resolv.py
+++ b/sphinx_gallery/docs_resolv.py
@@ -447,6 +447,9 @@ def embed_code_links(app, exception):
     print('Embedding documentation hyperlinks in examples..')

     gallery_conf = app.config.sphinx_gallery_conf
+    # cd to the appropriate directory regardless of sphinx configuration
+    working_dir = os.getcwd()
+    os.chdir(app.builder.srcdir)

     gallery_dirs = gallery_conf['gallery_dirs']
     if not isinstance(gallery_dirs, list):
@@ -454,3 +457,6 @@ def embed_code_links(app, exception):

     for gallery_dir in gallery_dirs:
         _embed_code_links(app, gallery_conf, gallery_dir)
+
+    # Back to initial directory
+    os.chdir(working_dir)

@Titan-C this is because embed_code_links is done once the build is finished like this: app.connect('build-finished', embed_code_links) so we are not in the right folder anymore (the chdir hack is in generate_gallery_rst. Do you remember why it was done like this?

Titan-C commented 7 years ago

@lesteve as I wrote earlier. scikit-image has this extra patch on their codebase that changes directory in embed_code_links.

I have no good memory why we did not propagate the hack from generate_gallery_rst to embed_code_links as it is required in both steps.(Certainly because we gave no conclusive answer to https://github.com/sphinx-gallery/sphinx-gallery/issues/115#issuecomment-219863074)

As a matter of implementation Sphinx does not change directories during build, but we do. We change once when generating the gallery to the source directory. And while building each example to the example location directory. Finally we return to Sphinx workdir. Then we need (because is not implemented yet) in the embed_code_links to change to the source dir again. In PR #133 I started working to avoid changing the directories while building the gallery and when putting the code links in favor of using only relative paths to the Sphinx workdir, but it did not work as I wanted at that time(I don't remember the problem I had, I shall certainly redo a new PR). There are few issues to solve on building the gallery one is building when the source directory is not the directory of the makefile and another is when sphinx-build is called directly from an arbitrary location. In general we shall also be able to build the docs when calling setup.py(setup tools offers its own sphinx-build call + uploading the docs to pythonhosted.org).

lesteve commented 7 years ago

@lesteve as I wrote earlier. scikit-image has this extra patch on their codebase that changes directory in embed_code_links.

OK that is what you said earlier, sorry about that, it was not very clear to me where this diff was coming from.

I guess the short-term solution is for scikit-image to restore this change in embed_code_links and for sphinx-gallery to integrate it in master.

Titan-C commented 7 years ago

I did a new PR #190 which shall address this. Drop the dir changing during build and link embedding. Dirs will only be changed when running the example. For scikit-image I have propagated #190 on top of their PR https://github.com/scikit-image/scikit-image/pull/2395. I think it shall build but travis is still on it (https://travis-ci.org/Titan-C/scikit-image/builds/185812316) If it builds, then PR #190 is enough and we can do a new release of Sphinx-Gallery

sciunto commented 7 years ago

Looks good so far :) Thanks for being so responsive!

Titan-C commented 7 years ago

Fixed in #190