search

maxmind / MaxMind-DB-Writer-perl

Create MaxMind DB database files
https://metacpan.org/release/MaxMind-DB-Writer/
Other
77 stars 27 forks source link

FTBFS: test failures on some architectures #112

Closed gregoa closed 1 year ago

gregoa commented 4 years ago

We have the following bug reported to the Debian package of MaxMind-DB-Writer, c.f. https://bugs.debian.org/968362

It doesn't seem to be a bug in the packaging, so you may want to take a look. Thanks!

------8<-----------8<-----------8<-----------8<-----------8<-----

Source: libmaxmind-db-writer-perl
Version: 0.300003-1
Severity: serious
Tags: upstream ftbfs
Justification: fails to build from source (but built successfully in the past)

libmaxmind-db-writer-perl never built on all architectures:

https://buildd.debian.org/status/logs.php?pkg=libmaxmind-db-writer-perl

The history of the uploads goes like this:

0.300003-1: testsuite disabled
Result: successful build on all architectures where all build dependencies
are available (esp. libmath-int128-perl is missing on quite a few).

0.300003-2: testsuite enabled but tests needing Test::HexDifferences
skipped (as it was not yet packaged)
Result: additional failures in the tests on ppc64, s390x, sparc64
As (only) s390x is a release architecture, the package never migrated
to testing.

0.300003-3: all tests are run after libtest-hexdifferences entered
the archive.
Result: same as for 0.300003-2

Logs of the failures (for 0.300003-3):

ppc64:
https://buildd.debian.org/status/fetch.php?pkg=libmaxmind-db-writer-perl&arch=ppc64&ver=0.300003-3&stamp=1596597051&raw=0

sparc64:
https://buildd.debian.org/status/fetch.php?pkg=libmaxmind-db-writer-perl&arch=sparc64&ver=0.300003-3&stamp=1596597463&raw=0

s390x:
https://buildd.debian.org/status/fetch.php?pkg=libmaxmind-db-writer-perl&arch=s390x&ver=0.300003-3&stamp=1596578449&raw=0

The failing tests are always the same, quoting from the s390x log:

Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
t/MaxMind/DB/Writer/Tree-freeze-thaw.t ..................... 
    1..0
not ok 1 - No tests run for subtest "Tree with 256 networks - IPv4 only - 24-bit records"
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/1 subtests 
[…]
Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
t/MaxMind/DB/Writer/Tree-output/freeze-thaw-record-size.t .. 
Dubious, test returned 255 (wstat 65280, 0xff00)
No subtests run 
[…]
Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
t/MaxMind/DB/Writer/Tree-record-collisions.t ............... 
[…]
Dubious, test returned 255 (wstat 65280, 0xff00)
All 21 subtests passed 
[…]
    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

    #   Failed test 'Run without exceptions'
    #   at t/MaxMind/DB/Writer/Tree-thaw-merge.t line 86.
    # Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.
    # Looks like you failed 1 test of 1.

t/MaxMind/DB/Writer/Tree-thaw-merge.t ...................... 
    not ok 1 - Run without exceptions
    1..1
not ok 1 - check defaults work
    not ok 1 - Run without exceptions
    1..1
not ok 2 - check no merging explictly
    not ok 1 - Run without exceptions
    1..1
not ok 3 - check no merging and none explictly
    not ok 1 - Run without exceptions
    1..1
not ok 4 - set mrc in constructor, toplevel in thaw
    not ok 1 - Run without exceptions
    1..1
not ok 5 - set toplevel in constructor
    not ok 1 - Run without exceptions
    1..1
not ok 6 - set recurse in constructor
    not ok 1 - Run without exceptions
    1..1
not ok 7 - set mrc only in constructor
    not ok 1 - Run without exceptions
    1..1
not ok 8 - set toplevel only in constructor
    not ok 1 - Run without exceptions
    1..1
not ok 9 - set recurse only in constructor
    not ok 1 - Run without exceptions
    1..1
not ok 10 - set toplevel only in thaw
    not ok 1 - Run without exceptions
    1..1
not ok 11 - set mrc off in constructor, toplevel in thaw
    not ok 1 - Run without exceptions
    1..1
not ok 12 - set none in constructor, toplevel only in thaw
    not ok 1 - Run without exceptions
    1..1
not ok 13 - set recurse only in thaw
    not ok 1 - Run without exceptions
    1..1
not ok 14 - set mrc off in constructor, recurse in thaw
ok 15 - no (unexpected) warnings (via done_testing)
1..15
Dubious, test returned 14 (wstat 3584, 0xe00)
Failed 14/15 subtests 
[…]
Test Summary Report
-------------------
t/MaxMind/DB/Writer/Tree-freeze-thaw.t                   (Wstat: 65280 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t/MaxMind/DB/Writer/Tree-output/freeze-thaw-record-size.t (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t/MaxMind/DB/Writer/Tree-record-collisions.t             (Wstat: 65280 Tests: 21 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
t/MaxMind/DB/Writer/Tree-thaw-merge.t                    (Wstat: 3584 Tests: 15 Failed: 14)
  Failed tests:  1-14
  Non-zero exit status: 14
Files=39, Tests=446, 16 wallclock secs ( 0.19 usr  0.02 sys +  9.19 cusr  0.52 csys =  9.92 CPU)
Result: FAIL
Failed 4/39 test programs. 15/446 subtests failed.

So basically always the same:
Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 600 at /<<PKGBUILDDIR>>/blib/lib/MaxMind/DB/Writer/Tree.pm line 403.

The tests were run with libsereal-{de,}encoder-perl 4.018+ds-1, srl_decoder.c
is in libsereal-decoder-perl.

Cheers,
gregor

------8<-----------8<-----------8<-----------8<-----------8<-----

Thanks for considering, gregor herrmann, Debian Perl Group

oschwald commented 4 years ago

I believe these are all big endian architectures. Do you happen to know if it is failing on all big endian architectures that Debian supports?

oschwald commented 4 years ago

One thing that I am confused by is 0.300003-1 seems to have built on most of these architectures but 0.300003-3 doesn't. Are you sure it isn't a packaging issue or something in the build environment that has changed?

gregoa commented 4 years ago

On Thu, 13 Aug 2020 08:25:59 -0700, Gregory Oschwald wrote:

I believe these are all big endian architectures. Do you happen to know if it is failing on all big endian architectures that Debian supports?

According to https://wiki.debian.org/ArchitectureSpecificsMemo the big-endian architectures are avr32 hppa m68k mips mips64 mipsn32 powerpc ppc64 s390 s390x sparc sparc64.

Not all of them have build daemons or are {still,already} alive but from the existing ones we see at https://buildd.debian.org/status/package.php?p=libmaxmind-db-writer-perl

hppa: no libmath-int128-perl m68k: no libmath-int128-perl powerpc: no libmath-int128-perl ppc64: failure s390x: failure sparc64: failure

So yeah, this indeed looks like a BE issue, at least there is no successful test on any big-endian machine.

Cheers, gregor

-- .''. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 . ' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe - NP: David Bowie: Modern Love

gregoa commented 4 years ago

On Thu, 13 Aug 2020 08:34:11 -0700, Gregory Oschwald wrote:

One thing that I am confused by is 0.300003-1 seems to have built on most of these architectures but 0.300003-3 doesn't.

0.300003-1 has built because the testsuite was disabled for this upload. [0]

Are you sure it isn't a packaging issue or something in the build environment that has changed?

I'm sure the change was that I have removed the test disabling :)

Cheers, gregor

[0] If you look at https://buildd.debian.org/status/fetch.php?pkg=libmaxmind-db-writer-perl&arch=s390x&ver=0.300003-1&stamp=1594592481&raw=0

you see:

   debian/rules override_dh_auto_test
make[1]: Entering directory '/<<PKGBUILDDIR>>'
true
make[1]: Leaving directory '/<<PKGBUILDDIR>>'

and no test harness output whatsoever

-- .''. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 . ' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe - NP: David Bowie: Modern Love

oschwald commented 4 years ago

Thanks for explaining that. It sounds like there might be a bug in the freeze/thaw code on big endian architectures.

gregoa commented 4 years ago

Hi,

I just wanted to ask if you got a chance to look into this issue? (All Debian build logs at https://buildd.debian.org/status/logs.php?pkg=libmaxmind-db-writer-perl )

Cheers, gregor

oschwald commented 4 years ago

We haven't had a chance to look at it too closely besides identifying it as likely an issue on big endian architectures. Unfortunately, we don't have easy access to big endian hardware. I suppose it may be possible to replicate the issue using QEMU.

Most of our future development is going into the Go mmdbwriter library, which should be a full replacement. Is the Perl writer a dependency of something in Debian? I glanced at the package page and it wasn't obvious to me.

gregoa commented 4 years ago

Thanks for the quick reply!

I guess we can live with the situation where it's not available for BE architectures; as for the question why it was packaged originally, maybe @FedericoCeratto can answer. (Technically it's indeed not a dependency of another package right now.)

oschwald commented 1 year ago

We are deprecating this library and archiving the repo. As mentioned above, we have a Go library for writing MMDB files that is being actively maintained.