Enable intersphinx links to SageMath's documentation

[nthiery]

When configuring Sphinx for documenting a package, it's in principle very easy to enable cross links to the Sage documentation, with:

intersphinx_mapping = {'https://docs.python.org/': None,
                                      'https://doc.sagemath.org/html/en/reference/': None}

That's a nice feature that packages built on top of Sage will want to exploit.

However, upon make html, I get:

WARNING: intersphinx inventory 'https://doc.sagemath.org/html/en/reference/objects.inv' not fetchable due to <class 'urllib2.HTTPError'>: HTTP Error 403: Forbidden

even though the file can be downloaded with e.g. wget.

I assume that the user agent "Python-urlib2" is considered by default as being a robot. Would it be possible to configure robots.txt to accept calls from urllib2 or just enable any user agent to retrieve objects.inv?

An alternative would be to dig in Sphinx's code and set a different user agent. I haven't quite yet managed to do that. But in any cases, this adds a piece of burden to package writers.

cc: @haraldschilly

[haraldschilly]

Hmm, I have to check how this is setup. The documentation, like the website, is hosted on github and (last time I checked) served via fastly.net. I'll check if cloudflare is setup, too.

However, downloading the file directly via "urllib" works. I.e. can you try a path like https://github.com/sagemath/documentation/raw/gh-pages/html/en/reference/objects.inv ?

Also, you can try again with the settings you already have, e.g. this works for me in python 3:

note: there is no https, I got a certificate error from the urllib library

>>> import urllib.request
>>> x = urllib.request.urlopen('http://doc.sagemath.org/html/en/reference/objects.inv')
>>> x.read()[:1000]
b'# Sphinx inventory version 2\n# Project: Sage Reference Manual\n# Version: 7.6\n# The remainder of this file is compressed using zlib.\nx\xda\xbc\xbd\xdb\x92\xec\xb6\x956x\xaf\xa7\xc8\x [...]

[nthiery]

On Sat, Apr 29, 2017 at 03:31:59AM -0700, Harald Schilly wrote:

Hmm, I have to check how this is setup. The documentation, like the website, is hosted on github and (last time I checked) served via fastly.net. I'll check if cloudflare is set, too.

However, downloading the file directly via "urllib" works. I.e. can you try a path like http://github.com/sagemath/documentation/raw/gh-pages/html/en/reference /objects.inv ?

Also, you can try again with the settings you already have, e.g. this works for me in python 3:

Ah ah, that's in fact sphinx's version that matters (or maybe the underlying urllib). I was running sphinx 1.3.6 in python2. With sphinx 1.5.5 running in python3 this works fine. Same with the sphinx 1.5.3 bundled with Sage.

Thanks for the insight. I guess the issue can be closed.

[haraldschilly]

With sphinx 1.5.5 running in python3 this works fine.

just FYI: there was also cloudflare in the game, so when you check 1.3.6 in py2 it could work right now, too. it's all kind of outsourced by setting it up once and then forgotten.

[nthiery]

On Sat, Apr 29, 2017 at 11:51:54AM -0700, Harald Schilly wrote:

just FYI: there was also cloudflare in the game, so when you check 1.3.6 in py2 it could work right now, too. it's all kind of outsourced by setting it up once and then forgotten.

Oh, I see :-) Issue closed anyway. Thanks!