search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.13k stars 5.47k forks source link

Salt 2014.1.0 docs not building on Debian GNU/Linux Wheezy. #10603

Closed boltronics closed 10 years ago

boltronics commented 10 years ago 
git/doc/ref/configuration/master.rst:8: SEVERE: Duplicate ID: "std:conf_master-fileserver_backend".
git/doc/topics/proxyminion/index.rst:48: ERROR: Unknown target name: "salt modules".

...

Exception occurred:
  File "/usr/lib/pymodules/python2.7/sphinx/cmdline.py", line 189, in main
    app.build(force_all, filenames)
  File "/usr/lib/pymodules/python2.7/sphinx/application.py", line 204, in build
    self.builder.build_update()
  File "/usr/lib/pymodules/python2.7/sphinx/builders/__init__.py", line 196, in build_update
    'out of date' % len(to_build))
  File "/usr/lib/pymodules/python2.7/sphinx/builders/__init__.py", line 252, in build
    self.write(docnames, list(updated_docnames), method)
  File "/usr/lib/pymodules/python2.7/sphinx/builders/__init__.py", line 292, in write
    self.write_doc(docname, doctree)
  File "/usr/lib/pymodules/python2.7/sphinx/builders/html.py", line 424, in write_doc
    ctx = self.get_doc_context(docname, body, metatags)
  File "/usr/lib/pymodules/python2.7/sphinx/builders/html.py", line 407, in get_doc_context
    display_toc = (self.env.toc_num_entries[docname] > 1),
KeyError: 'ref/modules/all/salt.modules.apt'

...

make[1]: *** [html] Error 1
make[1]: Leaving directory `/home/abolte/Downloads/applications/cfg_mgnt/salt/git/doc'
make: *** [clean] Error 2

Wheezy uses python-sphinx 1.1.3+dfsg-4, if that helps.

My quick work-around was to comment out the line make -C doc html in debian/rules.

basepi commented 10 years ago

So just the docs are failing to build, not salt itself, right? I'm going to update the title as such.

boltronics commented 10 years ago

No problem.

cachedout commented 10 years ago

This was broken for a while during the Great Docs Reorg of 2014. It is now fixed. I have verified that the docs are building correctly.

boltronics commented 10 years ago

Just checked out tag v2014.1.5, and this hasn't been fixed.

Maybe it's fixed in develop (haven't looked yet), but this issue is specifically about the 2014.1 branch on Wheezy.

basepi commented 10 years ago

Interesting. Is there a particular reason that you need to build it on that branch? develop is generally the best documentation we have, just have to watch for those versionadded directives. We also have old versions hosted on readthedocs: http://salt.readthedocs.org/en/v2014.1.4/

With the next feature release imminent, I'm just trying to avoid making more work on soon-to-be-abandoned 2014.1 brach. ;)

boltronics commented 10 years ago

I've been burned by using the documentation from develop way too many times. Just last week I was using file.copy as per the docs, only it doesn't seem to exist in v2014.1.5. My states have been littered with {# REPLACE THE BELOW CMD.RUN WITH THIS SECTION WHEN COMMIT xxxxxx MAKES IT TO A STABLE RELEASE #} comments (along with similar commented out sections for various require statements). No big deal, but it gets annoying after a while.

This has been a problem for a long time, so having the problem remain for another few months is not the end of the world.

basepi commented 10 years ago

Since we're targeting the release candidate for the next feature release for early next week, I'm going to say keep using the read the docs archive for a little bit longer. (Unless @whiteinge has a few spare minutes and wants to try to debug this). I totally understand the frustration with the develop docs, and I think that's why we're probably going to start defaulting docs.saltstack.com to the current release instead of the develop branch.

whiteinge commented 10 years ago

I checked out v2014.1.5 and built the docs using Sphinx 1.1.3. I see the Sphinx warnings / errors but not the traceback so the build did complete. If you're interested in getting that working I'm happy to work with you to figure out what the issue is on Wheezy. If you're just interested in having the local files I tar'ed up that build and uploaded it here:

https://github.com/saltstack/salt/releases/tag/v2014.1.5

Linking to #12754.

boltronics commented 10 years ago

@basepi having docs.saltstack.com default to the current release would be a huge improvement. I'm not sure if you mean that it would just point to the current stable branch, or the latest stable tag. I know that the latest tag seems to take some time to appear on the docs.saltstack.com site currently, but I'd much rather not start using something that I can't use yet anyway.

@whiteinge Thanks for the offer. I've got a local git repo that I work from though - seems more efficient than downloading a tarball all the time, and often I need to cherry-pick fixes from develop, or add commits of my own to work around problems with a release. eg. bug #13648 in v2014.1.5, past salt-cloud packaging issues, etc. So my workflow is:

$ git pull
Already up-to-date.
$ git checkout $(git tag | sort -V | tail -n 1)  # Switch to the latest tagged release
Note: checking out 'v2014.1.5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 3ae2c15... Fix pylint for nested outputter
$ head debian/changelog  # check if the changelog needs updating (it usually does)
salt (2014.1.0-1) unstable; urgency=low

  * New upstream version

 -- Bret Palsson <bretep@gmail.com>  Thu, 20 Mar 2014 14:00:54 -0700

salt (0.17.0-1) unstable; urgency=low

  * New upstream version

$ debchange -v 2014.1.5-1 -D stable 'New upstream version'  # changelog update
$ dpkg-buildpackage -rfakeroot -F
dpkg-buildpackage: source package salt
dpkg-buildpackage: source version 2014.1.5-1
dpkg-buildpackage: source changed by Adam Bolte <adam.bolte@sitepoint.com>
dpkg-buildpackage: host architecture amd64
 dpkg-source --before-build git
 fakeroot debian/rules clean
make -C doc html
make[1]: Entering directory `/home/abolte/Downloads/applications/cfg_mgnt/salt/git/doc'
No need to update translations. Skipping...
sphinx-build -b html -d _build/doctrees    . _build/html
Running Sphinx v1.1.3
/usr/lib/python2.7/dist-packages/pygments/plugin.py:39: UserWarning: Module mako was already imported from /dev/null, but /usr/lib/python2.7/dist-packages is being added to sys.path
  import pkg_resources
/usr/lib/python2.7/dist-packages/pygments/plugin.py:39: UserWarning: Module M2Crypto was already imported from /dev/null, but /usr/lib/python2.7/dist-packages is being added to sys.path
  import pkg_resources
/usr/lib/python2.7/dist-packages/pygments/plugin.py:39: UserWarning: Module MySQLdb was already imported from /dev/null, but /usr/lib/python2.7/dist-packages is being added to sys.path
  import pkg_resources
loading translations [en]... locale not available
loading pickled environment... done
[autosummary] generating autosummary for: contents.rst, faq.rst, index.rst, ref/auth/all/index.rst, ref/auth/all/salt.auth.auto.rst, ref/auth/all/salt.auth.keystone.rst, ref/auth/all/salt.auth.ldap.rst, ref/auth/all/salt.auth.pam.rst, ref/auth/all/salt.auth.stormpath_mod.rst, ref/cli/index.rst, ..., topics/tutorials/states_pt2.rst, topics/tutorials/states_pt3.rst, topics/tutorials/states_pt4.rst, topics/tutorials/states_pt5.rst, topics/tutorials/walkthrough.rst, topics/tutorials/walkthrough_macosx.rst, topics/tutorials/writing_tests.rst, topics/virt/disk.rst, topics/virt/index.rst, topics/virt/nic.rst
building [html]: targets for 631 source files that are out of date
updating environment: [config changed] 631 added, 7 changed, 0 removed
reading sources... [100%] topics/virt/nic                                                                                                                                                                         
/home/abolte/Downloads/applications/cfg_mgnt/salt/git/doc/ref/configuration/master.rst:8: SEVERE: Duplicate ID: "std:conf_master-fileserver_backend".
/home/abolte/Downloads/applications/cfg_mgnt/salt/git/doc/topics/proxyminion/index.rst:48: ERROR: Unknown target name: "salt modules".
/home/abolte/Downloads/applications/cfg_mgnt/salt/git/doc/topics/releases/2014.1.5.rst:38: WARNING: Inline literal start-string without end-string.
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 54%] ref/states/all/salt.states.file                                                                                                                                                          
Exception occurred:
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 165, in walkabout
    visitor.dispatch_visit(self)
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 1604, in dispatch_visit
    return method(node)
  File "/usr/lib/pymodules/python2.7/sphinx/writers/html.py", line 249, in visit_literal_block
    **highlight_args)
  File "/usr/lib/pymodules/python2.7/sphinx/highlighting.py", line 188, in highlight_block
    lexer = lexers[lang] = get_lexer_by_name(lang)
  File "/usr/lib/python2.7/dist-packages/pygments/lexers/__init__.py", line 77, in get_lexer_by_name
    for cls in find_plugin_lexers():
  File "/usr/lib/python2.7/dist-packages/pygments/plugin.py", line 53, in find_plugin_lexers
    yield entrypoint.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1989, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
ImportError: No module named ext.pygmentplugin
The full traceback has been saved in /tmp/sphinx-err-d4rjHH.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
Either send bugs to the mailing list at <http://groups.google.com/group/sphinx-dev/>,
or report them in the tracker at <http://bitbucket.org/birkenfeld/sphinx/issues/>. Thanks!
make[1]: *** [html] Error 1
make[1]: Leaving directory `/home/abolte/Downloads/applications/cfg_mgnt/salt/git/doc'
make: *** [clean] Error 2
dpkg-buildpackage: error: fakeroot debian/rules clean gave error exit status 2
$

I don't know what pygmentplugin is. If I create a virtualenv and attempt to pip install it, it's not found. In any case, if a dependency is missing it should fail immediately due to the debian/control Build-Depends section.

As it stands, I have to comment out the make -C doc html line in debian/rules for a successful dpkg-buildpackage execution, and I've had to do this for quite some time. I'd love to get to the bottom of this.

whiteinge commented 10 years ago

The unhelpful No module named ext.pygmentplugin error is caused by a .. code-block:: XXX that has a bad value for XXX. Usually a typo or a trailing space. I didn't see that error when building v2014.1.5 locally so my first instinct is that error is coming from elsewhere.

Does the /home/abolte/Downloads/applications/cfg_mgnt/salt/git/doc/_build directory exist before the dpkg-buildpackage process runs? Is there anything in it? If yes, that dir should be nuked before building. I wonder if you're tripping over an old build.

boltronics commented 10 years ago

Yes it does. Even running git reset --hard ; git clean -fd does not remove it, presumably because the directory is specified in .gitignore. Apparently I need to use git clean -fdx to ignore the gitignore - and that nukes the _build directory. I'll certainly keep that in mind - thanks for pointing it out.

Unfortunately, I continue to get exactly the same build output (running diff on /tmp/sphinx-err-d4rjHH.log and the new log shows no changes).

whiteinge commented 10 years ago

The doc/Makefile has a clean option if you want to re-use that functionality.

Try this: nuke the _build directory again and also nuke any .pyc files in your parent Salt dir to make sure the build is working with a completely fresh start.

If that doesn't get us anywhere will you pastebin the full /tmp/sphinx-err-XXX.log file?

boltronics commented 10 years ago

In the doc directory:

$ make clean
rm -rf _build/*
find locale/ -name *.mo -exec rm {} \;
find: `locale/': No such file or directory
make: *** [clean] Error 1

Also, find . -iname '*.pyc' in the project git root returns no matches. _build is already removed too. All this was taken care of by the git clean -fdx command.

I'm convinced it was clean, but took a fresh clone anyway to remove all doubt.

$ git clone git@github.com:saltstack/salt.git git
...quite some time later...
$ cd git
$ git checkout $(git tag | sort -V | tail -n 1)
Note: checking out 'v2014.1.5'.
...
HEAD is now at 3ae2c15... Fix pylint for nested outputter
$ debchange -v 2014.1.5-1 -D stable 'New upstream version'
$ dpkg-buildpackage -rfakeroot -F
...Exception occurred:.... ImportError... etc.
The full traceback has been saved in /tmp/sphinx-err-vqGN9C.log, if you want to report the issue to the developers.
$ cat /tmp/sphinx-err-vqGN9C.log
# Sphinx version: 1.1.3
# Python version: 2.7.3
# Docutils version: 0.8.1 release
# Jinja2 version: 2.6
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/sphinx/cmdline.py", line 189, in main
    app.build(force_all, filenames)
  File "/usr/lib/pymodules/python2.7/sphinx/application.py", line 204, in build
    self.builder.build_update()
  File "/usr/lib/pymodules/python2.7/sphinx/builders/__init__.py", line 196, in build_update
    'out of date' % len(to_build))
  File "/usr/lib/pymodules/python2.7/sphinx/builders/__init__.py", line 252, in build
    self.write(docnames, list(updated_docnames), method)
  File "/usr/lib/pymodules/python2.7/sphinx/builders/__init__.py", line 292, in write
    self.write_doc(docname, doctree)
  File "/usr/lib/pymodules/python2.7/sphinx/builders/html.py", line 419, in write_doc
    self.docwriter.write(doctree, destination)
  File "/usr/lib/pymodules/python2.7/docutils/writers/__init__.py", line 77, in write
    self.translate()
  File "/usr/lib/pymodules/python2.7/sphinx/writers/html.py", line 38, in translate
    self.document.walkabout(visitor)
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 173, in walkabout
    if child.walkabout(visitor):
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 165, in walkabout
    visitor.dispatch_visit(self)
  File "/usr/lib/pymodules/python2.7/docutils/nodes.py", line 1604, in dispatch_visit
    return method(node)
  File "/usr/lib/pymodules/python2.7/sphinx/writers/html.py", line 249, in visit_literal_block
    **highlight_args)
  File "/usr/lib/pymodules/python2.7/sphinx/highlighting.py", line 188, in highlight_block
    lexer = lexers[lang] = get_lexer_by_name(lang)
  File "/usr/lib/python2.7/dist-packages/pygments/lexers/__init__.py", line 77, in get_lexer_by_name
    for cls in find_plugin_lexers():
  File "/usr/lib/python2.7/dist-packages/pygments/plugin.py", line 53, in find_plugin_lexers
    yield entrypoint.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1989, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
ImportError: No module named ext.pygmentplugin
$
whiteinge commented 10 years ago

I appreciate you triple-checking that.

This is a little frustrating. You're also not the only one seeing this traceback. @terminalmage is seeing it as well. (@terminalmage are you seeing it on the develop branch as well?)

I'm not sure what the difference is between both of your systems and my system or the servers that build the docs for the website. I'm building out of a nearly empty virtualenv so perhaps there some rogue lib version causing the problem. I downgraded both Sphinx and docutils to try to reproduce and no dice. Below is the output of pip freeze on my system.

Jinja2==2.7.3
MarkupSafe==0.23
Pygments==1.6
Sphinx==1.1.3
docutils==0.8.1
six==1.7.2
sphinxcontrib-httpdomain==1.2.1
wsgiref==0.1.2

What version of Pygments are y'all running?

boltronics commented 10 years ago

If I use a minimal virtualenv (docs fail either way):

argparse==1.2.1
distribute==0.6.24
wsgiref==0.1.2

If I don't use a virtualenv (and I usually don't when building deb packages)... get ready for a long list:

Babel==0.9.6 (python-babel)
CDDB==1.4 (python-cddb)
Cheetah==2.4.4 (python-cheetah)
Cython==0.15.1 (cython)
Fabric==1.8.2 (fabric)
GnuPGInterface==0.3.2 (python-gnupginterface)
IPy==0.75 (python-ipy)
Jinja2==2.6  (python-jinja2)
M2Crypto==0.21.1 (python-m2crypto)
Magic-file-extensions==0.2
Mako==0.7.0
Markdown==2.1.1
MarkupSafe==0.15 (python-markupsafe)
MySQL-python==1.2.3
PAM==0.4.2
PIL==1.1.7
PyOpenGL==3.0.1
PyYAML==3.10
Pygments==1.5 (python-pygments)
Pymacs==0.23
SOAPpy==0.12.0
SQLAlchemy==0.7.8
Sphinx==1.1.3 (python-sphinx)
Tempita==0.5.1
Twisted-Conch==12.0.0
Twisted-Core==12.0.0
Twisted-Mail==12.0.0
Twisted-Names==12.0.0
Twisted-Web==12.0.0
Twisted-Words==12.0.0
arandr==0.1.6
argparse==1.2.1
binwalk==1.0
boto==2.3.0
bottle==0.10.11
bsddb3==5.2.0
buildbot==0.8.6p1
buildbot-slave==0.8.6p1
bzr==2.6.0dev2
chardet==2.0.1
configobj==4.7.2
coverage==3.4
cracklib==2.8.19
cups==1.0
decorator==3.3.3
defer==1.0.6
devscripts==2.12.6-deb7u2
distribute==0.6.24dev-r0
dkimpy==0.5.3
docutils==0.8.1 (python-docutils)
ecdsa==0.11
elementtidy==1.0-20050212
euca2ools==2.0.2
eyeD3==0.6.18
flup==1.0.2
fpconst==0.7.2
gdata==2.0.17
gevent==0.13.6
gnome-app-install==0.4.7-nmu1
greenlet==0.3.1
gyp==0.1
httplib2==0.7.4
iotop==0.4.4
ipython==0.13.1
iso8601==0.1.4
jack==3.1.1
jockey==0.9.7
keyring==0.7.1
kinterbasdb==3.3.0
launchpadlib==1.9.12
lazr.restfulclient==0.12.0
lazr.uri==1.0.3
libturpial==1.0-b1
logilab-astng==0.23.1
logilab-common==0.58.0
luma==3.0.6
lxml==2.3.2
mercurial==2.2.2
mini-dinstall==0.0.0
mock==0.8.0
msgpack-python==0.1.10
mysql-connector-python==0.3.2-devel
mysql-utilities==1.0.3
netifaces==0.8
nose==1.1.2
numpy==1.6.2
oauth==1.0.1
oauthlib==0.1.2
offlineimap==6.3.4
os-diskconfig-python-novaclient-ext==0.1.1
os-networksv2-python-novaclient-ext==0.21
paramiko==1.14.0
pexpect==2.4
prettytable==0.6.1
psycopg2==2.4.5
py-sg==0.11
pyOpenSSL==0.13
pyasn1==0.1.3
pychecker==0.8.19
pycrypto==2.6
pycryptopp==0.5.29
pycurl==7.19.0
pydns==2.3.6
pyenchant==1.6.5
pygpgme==0.2
pygtkspellcheck==3.0
pylint==0.25.1
pymssql==1.0.2
pyogg==1.3
pyserial==2.5
pysqlite==2.6.3
python-apt==0.8.8.2
python-dateutil==1.5
python-debian==0.1.21
python-debianbts==1.11
python-gtkmvc==0.0.0
python-ldap==2.4.10
python-novaclient==2.13.0
pytidylib==0.2.1
pytz==2012c
pyvorbis==1.5
pyxdg==0.19
pyzmq==13.1.0
rackspace-auth-openstack==1.0
rackspace-novaclient==1.2
rax-backup-schedule-python-novaclient-ext==0.1.2
rax-default-network-flags-python-novaclient-ext==0.1.3
reportbug==6.4.4
reportlab==2.5
requests==1.2.0
rope==0.9.2
ropemacs==0.6c2
sessioninstaller==0.0.0
simplegeneric==0.8.1
simplejson==2.5.2
six==1.7.2 (python-six)
smbc==1.0
sqlalchemy-migrate==0.7.2
tornado==2.3
turpial==2.9.6-a1
uTidylib==0.2
unattended-upgrades==0.1
unittest2==0.5.1
urlscan==0.5
urwid==1.0.1
vboxapi==1.0
virtualenv==1.7.1.2
wadllib==1.3.0
web.py==0.37
wsgiref==0.1.2 (python2.7)
wxPython==2.8.12.1
wxPython-common==2.8.12.1
xkit==0.0.0
zope.interface==3.6.1

I put some of the Debian package names (from where the python package came from) in brackets, but it was too much to do them all. I believe probably all of the above are from standard Debian packages, as I don't remember ever pip installing anything outside of the package management system or a virtualenv.

You mentioned Pygments specifically, and I see there is a 1.6 in wheezy-backports, so I upgraded to that and ran git clean -fdx and re-ran dpkg-buildpackage. No difference. Then again, I get the failure without Pygments in a virtualenv anyway, so I guess that's not terribly surprising.

whiteinge commented 10 years ago

Sorry, I don't quite follow your virtualenv-based test. If you're building out of a virtualenv you should have Jinja, Sphinx, docutils, and Pygments at a minimum. Could you elaborate?

boltronics commented 10 years ago

I fail to see why I need a virtualenv in the first place for building a Debian package with correct dependencies listed in the control file. Your hint that it may be required makes me suspect something dodgy is going on in the Salt build setup - it should be possible to build this with official packages.

I can build a virtualenv with --system-site-packages (and I'll effectively get the same as not using a virtualenv - see below), or I can use --no-site-packages (which is the default anyway). I suspect that in both cases, the virtualenv may be bypassed when building due to fakeroot? I assume you're using fakeroot too though and not actually building as root?

abolte@myhost:~/Downloads/applications/cfg_mgnt/salt$ virtualenv --no-site-packages virtualenv/
The --no-site-packages flag is deprecated; it is now the default behavior.
New python executable in virtualenv/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
abolte@myhost:~/Downloads/applications/cfg_mgnt/salt$ . virtualenv/bin/activate
(virtualenv)abolte@myhost:~/Downloads/applications/cfg_mgnt/salt$ pip freeze
argparse==1.2.1
distribute==0.6.24
wsgiref==0.1.2
(virtualenv)abolte@myhost:~/Downloads/applications/cfg_mgnt/salt$

So this probably isn't taking effect due to fakeroot I imagine, but it shouldn't matter.

Doing a diff between virtualenv with --system-site-packages and non-virtualenv python package listings (where neither work anyway), the only change is:

< distribute==0.6.24dev-r0
---
> distribute==0.6.24

where distribute 0.6.24dev-r0 is included with python-setuptools, but 0.6.24 seems to come from the python-virtualenv package. These are both official main Debian packages.

abolte@myhost:~$ for i in python-{setuptools,virtualenv} ; do apt-cache policy ${i}; dpkg -L ${i} | grep distribute | head -n 1 ; echo ; done
python-setuptools:
  Installed: 0.6.24-1
  Candidate: 0.6.24-1
  Version table:
 *** 0.6.24-1 0
        990 http://repocache.lan/debian/ wheezy/main amd64 Packages
        100 /var/lib/dpkg/status
/usr/lib/python2.7/dist-packages/distribute-0.6.24dev_r0.egg-info

python-virtualenv:
  Installed: 1.7.1.2-2
  Candidate: 1.7.1.2-2
  Version table:
 *** 1.7.1.2-2 0
        990 http://repocache.lan/debian/ wheezy/main amd64 Packages
        100 /var/lib/dpkg/status
/usr/share/python-virtualenv/distribute-0.6.24.tar.gz

abolte@myhost:~$

Hopefully that clarifies things for you, but I'm still in the dark about how you are invoking dpkg-buildpackage.

whiteinge commented 10 years ago

I fail to see why I need a virtualenv in the first place

You totally don't need one. I only mentioned that in the context of speculating that some strange lib might be causing interference in the build. The only deps required (by design) to build the docs are a copy of the Salt repo and Sphinx (plus Sphinx deps). No other software. Thus I'm doing builds on environments that only have those installed. Namely a virtualenv on my laptop or on a bare CentOS 6 system for the docs servers. As you pointed out, you also have a bare environment via fakeroot -- so this is perplexing to say the least.

We're both on the same Salt/Sphinx/docutils/Pygments versions so it must be something else. (I just tried again with the same Pygments version you replied with. Still no dice.)

how you are invoking dpkg-buildpackage.

I'm not. I hope I didn't give you the impression I was. I'm not (usually) involved in the Debian/Ubuntu packaging. You can see the actual command I use to build the docs here.

Since we can't isolate the differences between your environment and mine I suppose the next step is for me to set up a Debian environment and to try a build using dpkg-buildpackage to see if I can reproduce and spot the trouble directly. Since I don't do this frequently it usually takes me an hour or two to remember/read how and set this up. I definitely won't be able to get to it before the holiday and my schedule all next week is pegged. I'll circle back here when I can. :-P

boltronics commented 10 years ago

Ah. Makes sense. :)

I tried to mimic somewhat your invocation of make with:

make -C doc html PHINXOPTS='-q' SPHINXBUILD=/usr/bin/sphinx-build LATEXOPTS="-interaction=nonstopmode"

but it made no difference.

Even without a Debian GNU/Linux system, you could just setup a chroot:

$ mkdir wheezy-chroot
$ debootstrap wheezy wheezy-chroot http://ftp.debian.org/debian
$ sudo cp -r git wheezy-chroot/root/  # Assuming your salt repo is checked out into the "git" directory
$ sudo chroot wheezy-chroot
# cd /root/git
# apt-get install git dpkg-dev python-all-dev python-crypto python-jinja2 python-m2crypto python-setuptools python-sphinx python-yaml python-zmq msgpack-python debhelper cython
# git clean -fdx
# dpkg-buildpackage -F

and see what happens? :)

In fact, I just did that (to test that the instructions were accurate), and the bloody thing built! Gah!

That's obviously a minimal install with no backports, security updates, etc. Also running as root (as opposed to fakeroot) if that mattered. I'll have to look closer into this to see what the difference is that's causing this breakage.

terminalmage commented 10 years ago

@boltronics Is that a typo, or did you actually use PHINXOPTS instead of SPHINXOPTS?

boltronics commented 10 years ago

@terminalmage Sorry, I did use that, but it was a typo. It doesn't matter either way actually (the build fails regardless on my desktop Debian install), since it appears that the Makefile just uses it as a parameter to the sphinx-build command, which (from the man page): -q Quiet operation, just print warnings and errors on stderr.

joehealy commented 9 years ago

This is very old - and I guess everyone has moved on - but there are some patches to the docs needed to build on debian wheezy. It is best to look at the debian packaging on alioth which contians the patches. It can be view at http://anonscm.debian.org/cgit/pkg-salt/salt.git/tree/debian/patches?h=wheezy

boltronics commented 9 years ago

Thanks. I haven't built Salt on Wheezy in a while, but will probably need to again in the upcoming weeks. Last I tried this, I continued to use a temporary clean chroot to avoid the build issues I was experiencing. I never found the root cause of what is preventing the build from working on my usual desktop install.

joehealy commented 9 years ago

Things like:

http://anonscm.debian.org/cgit/pkg-salt/salt.git/tree/debian/patches/doc_fixes#n5

Have been a real issue over time. I suspect it was something similar - a different set of pygments plugins installed compared to whatever saltstack have on their machines (often rh).

Also this seemed to bite you:

http://anonscm.debian.org/cgit/pkg-salt/salt.git/tree/debian/patches/missing_locale_dir

I would recommend working from the "official" packaging files for your release which are contained at http://anonscm.debian.org/cgit/pkg-salt/salt.git (git clone git:// anonscm.debian.org/pkg-salt/salt.git). The files in the salt repo are quite outdated and really target ubuntu lucid.

I'm currently working through some docs on the best way to build your own packages for salt including various patches. I'll make an announcement on the salt-user mailing list when they are ready.

Cheers,

Joe

On Mon, Feb 16, 2015 at 10:54 PM, Adam Bolte notifications@github.com wrote:

Thanks. I haven't built Salt on Wheezy in a while, but will probably need to again in the upcoming weeks. Last I tried this, I continued to use a temporary clean chroot to avoid the build issues I was experiencing. I never found the root cause of what is preventing the build from working on my usual desktop install.

— Reply to this email directly or view it on GitHub https://github.com/saltstack/salt/issues/10603#issuecomment-74497925.

boltronics commented 9 years ago

Thanks for clarifying. It seems strange that Salt would keep this directory around in its repo at all if it was not expected to be used. All it can do is confuse people. IMO, either the patches from that repository you pointed out should be merged in (which should be acceptable since Salt officially claims to support Debian), or we should remove the /debian directory from the Salt repo entirely.

whiteinge commented 9 years ago

I set up the docs system and I'm the closest thing we have to a "docs manager" for now. I'm more than happy to field questions about building the docs. You are of course welcome to file issues with questions or you can ping me directly here or on IRC. The less patching you have to do for packaging the better.

  1. The only deps necessary to build the docs are Sphinx (which has Pygments as a dep). We do not use any Pygments plugins. The HTTP lexer is available in Pygments 1.5. We do have minimum version requirements for Sphinx and Pygments and those versions may not correspond to the versions that ship with Wheezy or any other of our 15+ supported distributions or supported platforms. Can you imagine if we had to hold the docs back to whatever version shipped with Cent 5 or Lucid? (To say nothing of Windows.) Bleh. :-)
  2. The locale directory is used by our documentation translators. What error are you seeing without the above patch? Do we need an empty dir in the repo? Do we need to update the Makefile?

@joehealy, @boltronics, what changes can we make or what things can we put in place to keep the packaging work you need to do as minimal as possible? Could we put a docs_requirements.txt file in the repo with the full list of software & versions required to build the docs?

Let me know how I can help.

boltronics commented 9 years ago

I agree we shouldn't need to hold progress back and wait for new distribution releases. However I do feel that all dependencies (both build and runtime) not met by the distribution in question should be available as packages if we are supporting packages. It doesn't matter to me if those packages are hosted from debian.saltstack.com or the Debian backports repos (as long as it's documented that backports is required).

Regarding Wheezy's versions, it looks like Wheezy has python-pygments 1.5 in main, and 1.6 in wheezy-backpors. Sphinx isn't in backports and is 1.1.3 in main.

I'm actually on vacation right now, so am away from my usual build environment. I'm not sure what the status is when building the unpatched 2014.7 release is (as I've not looked at 2014.7 yet at all), but it's something I'll likely be investigating when I return.

@whiteinge docs_requirements.txt might be useful to @joehealy when writing debian/control files, but as an end user I would expect that debian/control already has everything in it that was used to build the official packaged releases. So for an end user, a docs_requirements.txt file would probably be redundant.

I'm not 100% certain that we didn't have the correct dependencies met originally. It's possible that this bug was related to something else in my environment which was causing trouble, although I still have no idea what that could be. However the issue did go away when I used a chroot, so it's even possible that extra packages I had installed somehow conflicted with the build process. I haven't reinstalled the system I was building on in years, and I have no Sphinx experience, so identifying the exact cause isn't easy.

whiteinge commented 9 years ago

However I do feel that all dependencies (both build and runtime) not met by the distribution in question should be available as packages.

All the deps to build the docs are in packages already. Making the correct versions of those deps available on whatever distro you'll be building them on is where the work lies.

Let me float an alternate idea, we've long-discussed the idea of including the HTML and PDF builds in Salt's source-dist tarball that is uploaded to PyPI; the idea being that packages can make use of those without having to resort to building themselves. I know Debian has strict guidelines on building things from source, but I don't know details on those guidelines. Is there any flexibility there? The .rst files ship with Salt so the raw documentation is always available no matter what. Do the alternate formats (HTML/PDF/manpages) need to be rebuilt? Could you make use of those format conversions that SaltStack would supply in the sdist tarball?

joehealy commented 9 years ago

On Thu, Feb 19, 2015 at 10:24 AM, Seth House notifications@github.com wrote:

However I do feel that all dependencies (both build and runtime) not met by the distribution in question should be available as packages.

All the deps to build the docs are in packages already. Making the correct versions of those deps available on whatever distro you'll be building them on is where the work lies.

Let me float an alternate idea, we've long-discussed the idea of including the HTML and PDF builds in Salt's source-dist tarball that is uploaded to PyPI; the idea being that packages can make use of those without having to resort to building themselves. I know Debian has strict guidelines on building things from source, but I don't know details on those guidelines. Is there any flexibility there? The .rst files ship with Salt so the raw documentation is always available no matter what. Do the alternate formats (HTML/PDF/manpages) need to be rebuilt? Could you make use of those format conversions that SaltStack would supply in the sdist tarball?

My feeling is that this is a bit of a non issue. The real need is for me to finish of docs for how to build/patch packages for debian/ubuntu and get people basing work off official packages.

I'm currently able to build the docs for all but lucid [1]. I also suspect that anyone building off the official packages will be able to do so without too much trouble.

If the docs are prebuilt, I'm going to have to strip them out for the packages that go in the debian archives.

The "aim" here is that everything in the archive is able to be built by the archive. If we (debian hat on now) rely on building in a particular environment or outside the archive, we can never be sure that the documentation is truly free.

I think there is a distinction here between a sdist and bdist tarball. Ideally the sdist will just contain source - the bdist can contain binary/built stuff...

Cheers,

Joe

[1] I suspect I may be able to build them on lucid, but haven't tried for awhile. - now I've got lucids packaging under control, it may be trivial.

— Reply to this email directly or view it on GitHub https://github.com/saltstack/salt/issues/10603#issuecomment-74970740.

whiteinge commented 9 years ago

for me to finish of docs for how to build/patch packages for debian/ubuntu

Well, I won't argue with you since you're volunteering to do the work. :-) Maybe that's just the price of Freedom, but I do wish there was a middle ground we could find to make your work easier. It feels somewhat at odds if one party is pushing to add whiz-bang to the docs which forces the other party to then have to manually un-do that work with patches.

Out of curiosity, and this is purely a hypothetical, what would the packaging approach be if Salt was Free and the docs were not Free? Would Debian package Salt without the docs or would that preclude Debian packaging? Does just having them in an open format (such as plain text) change the equation or must they be Free all the way from source-to-build-to-final?

joehealy commented 9 years ago

On Thu, Feb 19, 2015 at 11:42 AM, Seth House notifications@github.com wrote:

for me to finish of docs for how to build/patch packages for debian/ubuntu

Well, I won't argue with you since you're volunteering to do the work. :-) Maybe that's just the price of Freedom, but I do wish there was a middle ground we could find to make your work easier.

I suspect here that regular builds for each of the distros/releases would help - if we knew at the time of adding whizbang to the docs which distros worked and didn't and why, we would be in a much better position to figure out the best approach (ie backport packages/patch/different approach/...). Finding out at somewhere fairly late (or after) the RC process is not ideal and just makes things more difficult and frustrating for everyone involved.

It feels somewhat at odds if one party is pushing to add whiz-bang to the

docs which forces the other party to then have to manually un-do that work with patches.

I agree. The http headers example [1] is an odd one. I can't remember the exact details, but I think the pygments part was working, but they were being tripped up in a latter stage - maybe by the content-type header triggering additional/different parsing/colouring.

Out of curiosity, and this is purely a hypothetical, what would the packaging approach be if Salt was Free and the docs were not Free? Would Debian package Salt without the docs or would that preclude Debian packaging? Does just having them in an open format (such as plain text) change the equation or must they be Free all the way from source-to-build-to-final?

Depends on definition of free...

If the docs were not DFSG (simplest case would be copyright owned by saltstack and distribution limited) then the docs package would need to be dropped. The rest would presumably be fine. If the source format was "open" ie most suitable for modification, but something about the destination was closed, then the bits that were "closed" in some way would need to be removed.

Simplest example I could think of for that would be if images/templates were used that were not freely distributable, but the text was. In that case, then the images etc would need to be stripped out and replaced with something that was free. Similarly, if other packages are included - ie jquery, bootstrap etc, then a few things need to happen - documentation of copyright status of files, ensuring there are no conflicts, dealing with any, preferably using versions of things that are in the archive (ie using jquery from the debian archive rather than shipping our own version of jquery etc).

Cheers,

Joe [1] http://anonscm.debian.org/cgit/pkg-salt/salt.git/tree/debian/patches/doc_fixes#n16