dman support - Githubissues

anarcat commented 7 years ago

one of the great uses of manpages.d.o is for installs that do not have man installed. for this, the ubuntu people wrote this clever little script called dman which fetches the raw pages to display locally. it's pretty neat!

but out of the box, it doesn't work with the debiman layout. i had to modify the script so that it works with manpages.debian.org - here's the diff:

--- dman    2016-04-13 10:47:42.000000000 -0400
+++ dman.debian 2017-01-31 13:31:49.110251075 -0500
@@ -26,7 +26,11 @@
 ###############################################################################

-. /etc/lsb-release
+if [ -r /etc/lsb-release ] ; then
+    . /etc/lsb-release
+else
+    DISTRIB_CODENAME=$(lsb_release -c -s)
+fi
 while true; do
    case "$1" in
        --release)
@@ -44,21 +48,27 @@
 # Mirror support of man's languages
 if [ ! -z "$LANG" ]; then
    LOCALE="$LANG"
+    LOCPATH="-$LANG"
+    LOCDOT=".$LANG"
 fi
 if [ ! -z "$LC_MESSAGES" ]; then
    LOCALE="$LC_MESSAGES"
+    LOCPATH="-$LOCALE"
+    LOCDOT=".$LOCALE"
 fi
-if echo $LOCALE | grep -q "^en"; then
+if echo $LOCALE | grep -E -q "^(C|en)"; then
    LOCALE=""
+    LOCPATH=""
+    LOCDOT=".en"
 fi

-URL="http://manpages.ubuntu.com/manpages.gz/"
+URL="http://manpages.debian.org"

 mandir=`mktemp -d dman.XXXXXX`
 trap "rm -rf $mandir" EXIT HUP INT QUIT TERM
 for i in `seq 1 9`; do
    man="$mandir/$i"
-   if wget -O "$man" "$URL/$DISTRIB_CODENAME/$LOCALE/man$i/$PAGE.$i.gz" 2>/dev/null; then
+   if wget -O "$man" "$URL/$DISTRIB_CODENAME/manpages$LOCPATH/$PAGE.$i$LOCDOT.gz" 2>/dev/null; then
        man $MAN_ARGS -l "$man" || true
    fi
    rm -f "$man"

notice how I had to dance around the locale stuff... i posted the full script in this paste bin: http://paste.debian.net/911957/

also, the above doesn't actually work in Debian stretch because it looks for https://manpages.debian.org/stretch/manpages/intro.1.en.gz - which is a 302 into the redirector, which redirects to the HTML page:

anarcat@curie:~$ curl -I https://manpages.debian.org/stable/manpages/intro.1.en.gz     
HTTP/1.1 302 Found
Date: Tue, 31 Jan 2017 18:42:00 GMT
Server: Apache
Strict-Transport-Security: max-age=15552000
Public-Key-Pins: pin-sha256="m2r9mfIa+ot6bIIC0bCt/7KZ1ych/f8QY3gk9cqUWqs="; pin-sha256="35f/cSfa9he3sUJgp1wZT9gzbI7/zH10hlT/utpEziU="; max-age=5184000
Location: https://dyn.manpages.debian.org/stable/manpages/intro.1.en.gz?
Cache-Control: max-age=3600
Expires: Tue, 31 Jan 2017 19:42:00 GMT
Content-Type: text/html; charset=iso-8859-1

anarcat@curie:~$ curl -I https://dyn.manpages.debian.org/stable/manpages/intro.1.en.gz?     
HTTP/1.1 307 Temporary Redirect
Date: Tue, 31 Jan 2017 18:41:54 GMT
Server: Apache
Strict-Transport-Security: max-age=15552000
Public-Key-Pins: pin-sha256="3T9ypiCPJdEeUOpKSooGJ1IpFbKsl/ktH0dV/wygJMk="; pin-sha256="xV7KmbTUH6WeUjOC5Tv7gsOpie45AvOH8/vjaIBsBxk="; max-age=5184000
Content-Length: 68
Content-Type: text/html; charset=utf-8
X-Clacks-Overhead: GNU Terry Pratchett
Location: https://manpages.debian.org/jessie/manpages/intro.1.en.html
Content-Language: en

so to fix this, we'd need to:

[x] ship the dman script along with other assets for people's easy deployment and joy
[x] fix the redirector so it supports codenames (e.g. stretch) along with symbolic names (e.g. testing), maybe related to #54?
[x] ~~ideally, make this part of the builtin man binary and/or package this as a separate dman binary package that can be installed for real (see also #1 for debian packaging)~~ was made part of the debian-goodies package
[x] ~~maybe rewrite dman in go while we're at it? :p~~ - moved to #1

anarcat commented 7 years ago

redirectory fixed in ae44c70, thanks!

anarcat commented 7 years ago

i just noticed there's a debman command in the debian-goodies package. That command can show the manpage within a .deb or it will download the .deb with debget if missing.

Sounds like another thing we could hijack. :) debman is a shellscript as well.

anarcat commented 7 years ago

in the meantime, could we ship the dman script in the assets and link it somewhere? maybe in the about section?

i'd make a PR if that's okay with you.

stapelberg commented 7 years ago

I’d prefer it if we could package up the dman script in a debian package (probably to experimental as we’re currently frozen). I think it’d be more confusing than useful to have different distribution channels for software, and Debian packages are definitely the best way to distribute software to Debian users :)

anarcat commented 7 years ago

Okay well, the problem is this will take forever to trickle down into a stable release... How about this - we figure out a debian package for our little shell script refugee and link to its source code in the FAQ?

In fact, I have created a collab-maint branch for dman in debian-goodies called debiman-support which adds a dman command to the package. We could then link to the source code when/if this is accepted by the package maintainer...

Here's the source code now:

https://anonscm.debian.org/git/collab-maint/debian-goodies.git/tree/dman?h=debiman-support

How does that sound?

anarcat commented 7 years ago

The alternative to this, of course, is to package debiman directly (see #1) and make this a distinct binary package people can just install directly. Given the complexity (or I guess my unfamiliarity) with golang + debian packaging, I am tempted to just stick with this easier option, as it also doesn't involve a rewrite in go. ;)

in other news, i asked to merge the branch in debian-goodies in #860920. let's see how that flies.

stapelberg commented 7 years ago

Linking to the source of a Debian package is fine with me.

xtaran commented 7 years ago

debian-goodies 0.70 with dman included has just been uploaded to Debian experimental.

jbicha commented 7 years ago

@dustinkirkland See above and https://tracker.debian.org/news/844034

anarcat commented 7 years ago

alright, I've linked to dman in the FAQ - we just need to merge the branch to complete this issue I believe - see #1 for the pipe dreams of packaging debiman and rewriting dman and so on. ;)

dustinkirkland commented 7 years ago

Hey team -- first off, I'm super pleased that you find dman useful! I love it, as well.

I would be very happy for Debian to ship dman in a Debian package, of course. I would just politely ask that you make it work with both Debian and Ubuntu, with a simple conditional that you can key directly off of the info you get out of the lsb-release.

As such, we'll be glad to use the same dman in Ubuntu as you use in Debian.

Thanks! @DustinKirkland

anarcat commented 7 years ago

On 2017-04-28 12:18:14, Dustin Kirkland wrote:

Hey team -- first off, I'm super pleased that you find dman useful! I love it, as well.

I would be very happy for Debian to ship dman in a Debian package, of course. I would just politely ask that you make it work with both Debian and Ubuntu, with a simple conditional that you can key directly off of the info you get out of the lsb-release.

As such, we'll be glad to use the same dman in Ubuntu as you use in Debian.

Hi!

I think that's a great idea. Unfortunately, I made some changes to the script and I am not sure it still works against manpages.ubuntu.com right now. I am certain, however, that the debian-goodies maintainer will happily welcome patches to fix that!

The source code is here:

https://anonscm.debian.org/git/collab-maint/debian-goodies.git/tree/dman

And you can file a pull request / send a patch on the Debian-goodies package:

https://tracker.debian.org/pkg/debian-goodies

Thanks!

xtaran commented 7 years ago

@anarcat, @dustinkirkland: Yes, actually I had this issue in my mind already, too. Patches are indeed very welcome.

anarcat commented 7 years ago

i tried this simple patch to try and support ubuntu (based on the no-suite branch):

diff --git a/dman b/dman
index a44ecf1..fea73f7 100755
--- a/dman
+++ b/dman
@@ -61,7 +61,14 @@ if echo $LOCALE | grep -E -q "^(C|en)"; then
        LOCDOT=".en"
 fi

-BASE_URL="https://manpages.debian.org"
+case $(lsb_release -s -i) in
+    Ubuntu)
+        BASE_URL="https://manpages.ubuntu.com"
+        ;;
+    *)
+        BASE_URL="https://manpages.debian.org"
+        ;;
+esac

 mandir=`mktemp -d dman.XXXXXX`
 trap "rm -rf $mandir" EXIT HUP INT QUIT TERM

unfortunately that doesn't work, because we look into debiman-specific paths and ubuntu's manpage service lacks the required redirections:

./dman: not found: https://manpages.ubuntu.com/stretch/crontab.fr.gz
./dman: not found: https://manpages.ubuntu.com/stretch/crontab.gz
./dman: not found: https://manpages.ubuntu.com/crontab.fr.gz
./dman: not found: https://manpages.ubuntu.com/crontab.gz
./dman: Could not fetch manpage from given locations.

even when using the manpages.gz prefix, this fails:

./dman: not found: https://manpages.ubuntu.com/manpages.gz/stretch/crontab.fr.gz
./dman: not found: https://manpages.ubuntu.com/manpages.gz/stretch/crontab.gz
./dman: not found: https://manpages.ubuntu.com/manpages.gz/crontab.fr.gz
./dman: not found: https://manpages.ubuntu.com/manpages.gz/crontab.gz
./dman: Could not fetch manpage from given locations.

the original dman had quite a bit of logic to guess the filesystem paths on the other end. we got rid of a lot of that for performance reasons: in the worst case, the original version was doing 9 requests to the server - we lowered that to 2, and jumped back to 4 to support multiple suites.

so i'm not sure how to deal with this. the logic will get really hairy if we need to support site-specific patterns like this. all i can think of now is rewrite the thing in python to support parameter expansion - e.g. https://%(base_url)s/%(suite)s/%(man)s.%(lang)s.gz is debiman, and ubuntu is https://%(base_url)s/%(lang)s/man%(section)s/%(man)s.%(section)s.gz... maybe this could be done in bash too, but you'd get nasty eval()s...

anarcat commented 7 years ago

we have opened #861611 for Ubuntu support in dman, so this is pretty much complete here. i have opened a separate issue (#79) for some optimizations that could make dman perform a little better as well.

Debian / debiman

dman support #57