sgml tests are failing

[newbluemoon]

Hi,

I maintain po4a for Void Linux and ran into a test failure when onsgmls tries to verify/process the test basic.sgml according to the docbook DTD.

In Sgml.pm there are two invocations of onsgmls: line 408: ... system("onsgmls -p < $tmpfile") ... and line 861: my $cmd = "onsgmls -l -E 0 -wno-valid < $tmpfile" ... that error out.

The reason it fails is that the OpenSP package comes with a sgml catalog entry which has a line SGMLDECL unicode.sd (unicode.sd comes with OpenSP). When I remove this line or the entire catalog onsgmls falls back to its built in default sgml declaration and the test succeeds.

The same when I run the test command outside of po4a like onsgmls -p < basic.sgml. However, when I change the command to onsgmls -p basic.sgml (i.e. without the < redirection) it passes.

So when I patch the < out of Sgml.pm the above commands succeed, but the test fails a bit later when the output is compared.

    #   Failed test 'Provided command (retcode: 256)'
    #   at t/Testhelper.pm line 138.
    # Produced file tmp/fmt/sgml:
    # (end of tmp/fmt/sgml)
    # -------------------------------------------------------------
    # Looks like you failed 1 test of 7.

#   Failed test 'Format fmt/sgml/basic.sgml'
#   at t/Testhelper.pm line 510.

    #   Failed test 'Provided command (retcode: 256)'
    #   at t/Testhelper.pm line 138.
    # Produced file tmp/fmt/sgml:
    # (end of tmp/fmt/sgml)
    # -------------------------------------------------------------
    # Looks like you failed 1 test of 7.

#   Failed test 'Format fmt/sgml/attributes-order.sgml'
#   at t/Testhelper.pm line 510.
# Looks like you failed 2 tests of 2.
t/fmt-sgml.t .......... 
# Subtest: Format fmt/sgml/basic.sgml
    # Change directory to fmt/sgml
    ok 1 - Normalizing Format fmt/sgml/basic.sgml
    #   Pass: perl po4a-normalize -f sgml --quiet --pot t/tmp/fmt/sgml/basic.pot --localized t/tmp/fmt/sgml/basic.norm  basic.sgml > t/tmp/fmt/sgml/basic.norm.stderr 2>&1 (retcode: 0)
    ok 2 - Translating Format fmt/sgml/basic.sgml
    ok 3 -   Command: perl BUILDPATH/po4a-translate -f sgml  --master basic.sgml --po BUILDPATH/t/fmt/sgml/basic.po --localized BUILDPATH/t/tmp/fmt/sgml/basic.trans > BUILDPATH/t/tmp/fmt/sgml/basic.trans.stderr 2>&1
    ok 4 - perl BUILDPATH/po4a-updatepo -f sgml  --master basic.sgml --po BUILDPATH/t/tmp/fmt/sgml/basic.po_updated > BUILDPATH/t/tmp/fmt/sgml/update.stderr 2>&1
    # Change directory back to /builddir/po4a-0.64/t
    ok 5 - diff -uN fmt/sgml/basic.norm.stderr BUILDPATH/t/tmp/fmt/sgml/basic.norm.stderr  > tmp/fmt/sgml/basic.cmd_output 2>&1
    ok 6 - PODIFF   fmt/sgml/basic.pot  tmp/fmt/sgml/basic.pot  > tmp/fmt/sgml/basic.cmd_output 2>&1
    not ok 7 - Provided command (retcode: 256)
    # Expected retcode: 0
    # FAILED command: diff -u fmt/sgml/basic.norm   tmp/fmt/sgml/basic.norm  > tmp/fmt/sgml/basic.cmd_output 2>&1
    # Command output:
    # | --- fmt/sgml/basic.norm 2021-09-12 20:48:54.000000000 +0000
    # | +++ tmp/fmt/sgml/basic.norm 2021-11-11 08:48:16.254554400 +0000
    # | @@ -19,12 +19,7 @@
    # |     <holder>&dhusername; and &dhuusername;</holder>
    # |    </copyright>
    # |   </refentryinfo>
    # | - <refnamediv>
    # | -  <refname></refname>
    # | -  <refpurpose></refpurpose>
    # | - </refnamediv>
    # |   <refsect1>
    # | -  <title></title>
    # |    <para>
    # |      Help Options:
    # |    </para>
    # (end of output)
    1..7
not ok 1 - Format fmt/sgml/basic.sgml
# Subtest: Format fmt/sgml/attributes-order.sgml
    # Change directory to fmt/sgml
    ok 1 - Normalizing Format fmt/sgml/attributes-order.sgml
    #   Pass: perl po4a-normalize -f sgml --quiet --pot t/tmp/fmt/sgml/attributes-order.pot --localized t/tmp/fmt/sgml/attributes-order.norm  attributes-order.sgml > t/tmp/fmt/sgml/attributes-order.norm.stderr 2>&1 (retcode: 0)
    ok 2 - Translating Format fmt/sgml/attributes-order.sgml
    ok 3 -   Command: perl BUILDPATH/po4a-translate -f sgml  --master attributes-order.sgml --po BUILDPATH/t/fmt/sgml/attributes-order.po --localized BUILDPATH/t/tmp/fmt/sgml/attributes-order.trans > BUILDPATH/t/tmp/fmt/sgml/attributes-order.trans.stderr 2>&1
    ok 4 - perl BUILDPATH/po4a-updatepo -f sgml  --master attributes-order.sgml --po BUILDPATH/t/tmp/fmt/sgml/attributes-order.po_updated > BUILDPATH/t/tmp/fmt/sgml/update.stderr 2>&1
    # Change directory back to /builddir/po4a-0.64/t
    ok 5 - diff -uN fmt/sgml/attributes-order.norm.stderr BUILDPATH/t/tmp/fmt/sgml/attributes-order.norm.stderr  > tmp/fmt/sgml/attributes-order.cmd_output 2>&1
    ok 6 - PODIFF   fmt/sgml/attributes-order.pot  tmp/fmt/sgml/attributes-order.pot  > tmp/fmt/sgml/attributes-order.cmd_output 2>&1
    not ok 7 - Provided command (retcode: 256)
    # Expected retcode: 0
    # FAILED command: diff -u fmt/sgml/attributes-order.norm   tmp/fmt/sgml/attributes-order.norm  > tmp/fmt/sgml/attributes-order.cmd_output 2>&1
    # Command output:
    # | --- fmt/sgml/attributes-order.norm  2021-09-12 20:48:54.000000000 +0000
    # | +++ tmp/fmt/sgml/attributes-order.norm  2021-11-11 08:48:16.847555060 +0000
    # | @@ -1,10 +1,5 @@
    # |  <!doctype refentry PUBLIC "-//OASIS//DTD DocBook V4.1//EN"><refentry>
    # | - <refnamediv>
    # | -  <refname></refname>
    # | -  <refpurpose></refpurpose>
    # | - </refnamediv>
    # |   <refsect1>
    # | -  <title></title>
    # |    <para>
    # |      <link audiance="IRC" vendor="billy" version="1"></link>
    # |  
    # (end of output)
    1..7
not ok 2 - Format fmt/sgml/attributes-order.sgml
1..2
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/2 subtests 

What I can’t quite figure out is if the problem is distribution or po4a or OpenSP related (e.g. some unicode issue?). What is the purpse of using a shell redirection in onsgmls -p < $tmpfile? All tutorials that I found do it without.

Thank you very much for any insight in advance!

[mquinson]

Hello,

I changed the piping of input into the onsgmls command, as you advised. I think that it was historical in po4a. Please remember that once upon a time, po4a was using osgmls when onsgmls did not exist yet. So I'm not surprised if this module gets a bit rusty with time.

The tests are still working for me after the change, but I seem to understand that they fail for you. Maybe we should manage to get the debug output of onsgmls or something? Running po4a on that file should not be hard, but I have to go right now. I may find some time to investiguate further later this week. Feel free to ping me otherwise.

[mquinson]

Here is how to run the tests manually. But maybe the test itself is broken:

$ cd t/fmt/sgml
$ LC_ALL=C PERL5LIB=../../../lib/ ../../../po4a-normalize -f sgml --pot basic.pot --localized basic.norm basic.sgml --verbose -o debug=onsgmls
CMD=onsgmls -l -E 0 -wno-valid /tmp/po4a-fPLj.sgml |
onsgmls:/tmp/po4a-fPLj.sgml:27:11:E: start tag for "REFNAMEDIV" omitted, but its declaration does not permit this
onsgmls:/tmp/po4a-fPLj.sgml:27:11:E: start tag for "REFNAME" omitted, but its declaration does not permit this
onsgmls:/tmp/po4a-fPLj.sgml:27:11:E: no start tag specified for implied empty element "REFNAME"
onsgmls:/tmp/po4a-fPLj.sgml:27:11:E: start tag for "REFPURPOSE" omitted, but its declaration does not permit this
onsgmls:/tmp/po4a-fPLj.sgml:27:11:E: no start tag specified for implied empty element "REFPURPOSE"
onsgmls:/tmp/po4a-fPLj.sgml:27:11:E: no start tag specified for implied empty element "REFNAMEDIV"
onsgmls:/tmp/po4a-fPLj.sgml:28:9:E: start tag for "TITLE" omitted, but its declaration does not permit this
onsgmls:/tmp/po4a-fPLj.sgml:28:9:E: no start tag specified for implied empty element "TITLE"
po4a::sgml: msgid skipped to help translators (contains only tags)
po4a::sgml: Warning: onsgmls produced some errors.  This is usually caused by po4a, which modifies the input and restores it afterwards, causing the input of onsgmls to be invalid.  This is
            usually safe, but you may wish to verify the generated document with onsgmls -wno-valid.
Write the normalized document to basic.norm.
Write the normalized PO file to basic.pot.

Maybe we should modify basic.sgml to get ride of these warnings/errors in the formatting. I guess that my version of onsgmls+po4a adds empty tags for the missing ones, while yours sticks to the content and does not produce empty elements for the missing ones.

If you manage to come up with a patch to basic.sgml file and other basic.* friends, I'd be really thankful.

[newbluemoon]

First, thank you very much for looking into it!

Here is my output:

$ LC_ALL=C PERL5LIB=../../../lib/ ../../../po4a-normalize -f sgml --pot basic.pot --localized basic.norm basic.sgml --verbose -o debug=onsgmls
CMD=onsgmls -l -E 0 -wno-valid /tmp/po4a-IH3D.sgml |
onsgmls:/tmp/po4a-IH3D.sgml:43:14:E: "REFENTRY" not finished but document ended
onsgmls:/tmp/po4a-IH3D.sgml:43:14:E: end tag for "REFENTRY" omitted, but OMITTAG NO was specified
onsgmls:/tmp/po4a-IH3D.sgml:8:0: start tag was here
po4a::sgml: msgid skipped to help translators (contains only tags)
po4a::sgml: Warning: onsgmls produced some errors.  This is usually caused by po4a, which modifies the input and restores it afterwards, causing the input of onsgmls to be invalid.  This is usually safe, but you may wish to verify the generated document with onsgmls -wno-valid.
Write the normalized document to basic.norm.
Write the normalized PO file to basic.pot.

onsgmls is version 1.5.2, perl-SGMLSpm version 1.1, docbook 4.5, po4a 0.65

Sure, I can try to come up to a patch for the basic.* files, but just so that you know I’m not that familiar with sgml stuff (but I will give it a shot anyway).

[mquinson]

I can't understand why your onsgmls is configured with OMITTAG NO while mine is not, but I'm rather sure that this is the cause of the breakage. Providing valid sgml files as a test suite is certainly the way to go there.

Thanks in advance.

PS: I have onsgmls 1.5.2, perl-SGMLSpm 1.03ii (but the changelog says that the new upstream release is mostly the integration of the debian changes that I should already have on my debian), docbook 4.5 and po4a 0.65

[newbluemoon]

Debian patches out all declarations shipped with docbook while Void tries to stay as close to upstream sources as possible and so there are DATATAG NO and OMITTAG NO in docbook.dcl. When I set them to YES all tests pass.

But I agree that the test should be run against the unpatched docbook settings.

[newbluemoon]

No, it’s only OMITTAG that needs to be set to YES.

[newbluemoon]

330 is my attempt at fixing the sgml test files. :)