Debiman crashes

[xfbs]

Hey guys!

Really awesome software. I was trying to get it work for me because I wanted to play with a large repository of man pages. Unfortunately, I had some issues. First, getting it to build on bullseye with golang failed, and go get does not produce very pretty error messages. Eventually I figured out that I had to install golang from backports. Also, I had to run go install, which was not very obvious (to someone who is not familiar with Go). I think there is some headroom for improvement around the tooling.

But anyways, once I did get it compiled, I wanted to get going. I tried something like this:

~/go/bin/debiman -serving_dir ./man -remote_mirror http://mirror.eu.oeandone.net -sync_codenames bullseye -only_render_pkgs=qelectrotech,i3-wm,cron

I tried several remote mirrors, including https://deb.debian.org/ and some others. Also different combinations of commands, codenames, and just no other options. But it keeps crashing, and the error messages are not exactly helpful either. Here's some output:

2023/01/11 18:26:10 globalview.go:292: package "testing/worklog" has errors: [Trying to interpret path "man1/man1/worklog.1.gz": Cannot parse language "man1": language: tag is not well-formed]
2023/01/11 18:26:10 globalview.go:292: package "testing/libccrtp-doc" has errors: [Trying to interpret path "man9/_build_libccrtp-eppWJD_libccrtp-2.0.9_src_.9_src_.3.gz": file name ("_build_libccrtp-eppWJD_libccrtp-2.0.9_src_.9_src_.3.gz") does not match regexp \.9([^.]*)\.gz$ Trying to interpret path "man9/_build_libccrtp-eppWJD_libccrtp-2.0.9_src_ccrtp_.9_src_ccrtp_.3.gz": file name ("_build_libccrtp-eppWJD_libccrtp-2.0.9_src_ccrtp_.9_src_ccrtp_.3.gz") does not match regexp \.9([^.]*)\.gz$]
2023/01/11 18:26:10 globalview.go:292: package "testing/faust" has errors: [Trying to interpret path "py/man1/faust2sc.1.gz": Cannot parse language "py": language: subtag "py" is well-formed but unknown]
2023/01/11 18:26:10 globalview.go:292: package "testing/po4a" has errors: [Trying to interpret path "zh_CHS/man1/msguntypot.1p.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man1/po4a-display-man.1.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man1/po4a-display-pod.1.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man1/po4a-normalize.1p.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man1/po4a-translate.1p.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man1/po4a-updatepo.1p.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man1/po4a.1p.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::AsciiDoc.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::BibTeX.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Chooser.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Common.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Dia.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Docbook.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Guide.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Halibut.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Ini.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::KernelHelp.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::LaTeX.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Man.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Po.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Pod.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::RubyDoc.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Sgml.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::TeX.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Texinfo.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Text.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::TransTractor.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Wml.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Xhtml.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Xml.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man3/Locale::Po4a::Yaml.3pm.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown Trying to interpret path "zh_CHS/man7/po4a.7.gz": Cannot parse language "zh_CHS": language: subtag "chs" is well-formed but unknown]
2023/01/11 18:26:10 globalview.go:292: package "testing/doris" has errors: [Trying to interpret path "py/man1/getorb_rs2.1.gz": Cannot parse language "py": language: subtag "py" is well-formed but unknown Trying to interpret path "py/man1/rs2_dump_data.1.gz": Cannot parse language "py": language: subtag "py" is well-formed but unknown Trying to interpret path "py/man1/rs2_dump_header2doris.1.gz": Cannot parse language "py": language: subtag "py" is well-formed but unknown Trying to interpret path "py/man1/tsx_dump_data.1.gz": Cannot parse language "py": language: subtag "py" is well-formed but unknown Trying to interpret path "py/man1/tsx_dump_header2doris.1.gz": Cannot parse language "py": language: subtag "py" is well-formed but unknown]
2023/01/11 18:26:10 globalview.go:292: package "testing/dhcp-probe" has errors: [Trying to interpret path "cf/man5/dhcp_probe.5.gz": Cannot parse language "cf": language: subtag "cf" is well-formed but unknown]
2023/01/11 18:26:10 globalview.go:292: package "testing/partclone" has errors: [Trying to interpret path "dd/man8/partclone.8.gz": Cannot parse language "dd": language: subtag "dd" is well-formed but unknown]
2023/01/11 18:26:10 main.go:120: gathered packages of all suites, total 33464 packages
bullseye/sfnt2woff-zopfli: 2023/01/11 18:26:10 findClosestFile(src="./usr/share/man/man1/woff2sfnt-zopfli.1.gz", name="/usr/share/man/man1/sfnt2woff-zopfli.1.gz")
bullseye/sfnt2woff-zopfli: 2023/01/11 18:26:10 parsing "/usr/share/man/man1/sfnt2woff-zopfli.1.gz" as man: <nil>

Any idea what is going on? I understand that this is a Debian internal tool, but I figured that if I open an issue here and we figure it out, might help others out too.

Cheers, @xfbs

[stapelberg]

Hey @xfbs 👋

First, getting it to build on bullseye with golang failed, and go get does not produce very pretty error messages. Eventually I figured out that I had to install golang from backports.

Yeah, the version in Debian stable will almost always be too old. We generally develop debiman with the currently released Go version, and accept fixes that make it work with the previous version (currently Go 1.19 and Go 1.18, but in about a month that’ll be Go 1.20 and Go 1.19). I’ll add a note to the README.

Also, I had to run go install, which was not very obvious (to someone who is not familiar with Go).

Yeah, the switch from go get to go install is somewhat recent, and we didn’t update the README since then. I’ll fix that.

But it keeps crashing, and the error messages are not exactly helpful either. Here's some output:

I don’t see a crash in your output. The output of debiman will always contain errors, as the packages in Debian often contain quality issues — for example, in Trying to interpret path "man1/man1/worklog.1.gz", the maintainer should fix the package to not install into an extra man1 subdirectory (precisely one is enough).

Can you say more about the symptoms, and can you attach the full stdout+stderr log please? Does debiman hang for you? Or does it return but not produce output?

[xfbs]

Hey, I was able to figure it out! Small error on my part, although I think that the output could perhaps be improved.

The issue was me setting

-serving_dir ./man

This is a relative path, but it should have been an absolute one. I found somewhere in the code that it is prefixed with a leading slash. The output does not really indicate it, but it crashes then because of that. Setting it to

-serving_dir $(pwd)/man

Fixes the issue.

Cheers, @xfbs

[stapelberg]

Thanks, I could reproduce the issue with that hint! Because debiman changes into -serving_dir, a relative location was referring to a different path after the chdir call. This is now fixed with commit 3b686fd792abc4ba0ce8ae9b86ec84d268fa9100