Unit test translator.t fails on Debian

[tgreenx]

On a clean Debian 12.5 install, latest develop, the following tests fail:

~/zonemaster/zonemaster-backend$ git log -1 --oneline
21212d4 (HEAD -> develop, origin/develop) Merge pull request #1169 from MichaelTimbert/update-MANIFEST

~/zonemaster/zonemaster-backend$ prove t/translator.t 
t/translator.t .. 1/? 
#   Failed test 'Translating a GLOBAL_VERSION message tag works'
#   at t/translator.t line 46.
#                   'Using version v5.0.0 of the Zonemaster engine.'
#     doesn't match '(?^u:\AUtilisation de la version .* du moteur Zonemaster\.\Z)'

#   Failed test 'Translating a B02_NS_BROKEN message works'
#   at t/translator.t line 63.
#                   'Broken response from name server "ns1.example" on an SOA query.'
#     doesn't match '(?^u:\AR�ponse cass�e du serveur de noms )'

#   Failed test 'Translation is a string of Unicode codepoints, not bytes'
#   at t/translator.t line 65.
#                   'Broken response from name server "ns1.example" on an SOA query.'
#     doesn't match '(?^u:cass\x{e9}e)'

#   Failed test 'Translating a backend-specific TEST_DIED message tag works'
#   at t/translator.t line 82.
#                   'An error occured and Zonemaster could not start or finish the test.'
#     doesn't match '(?^u:\AUne erreur est survenue )'

#   Failed test 'Translating Consistency01 gives a string of Unicode codepoints'
#   at t/translator.t line 91.
#                   'SOA serial number consistency'
#     doesn't match '(?^u:\ACoh\x{e9}rence du num\x{e9}ro de s\x{e9}rie)'
# Looks like you failed 5 tests of 9.
t/translator.t .. Dubious, test returned 5 (wstat 1280, 0x500)
Failed 5/9 subtests 

Test Summary Report
-------------------
t/translator.t (Wstat: 1280 (exited 5) Tests: 9 Failed: 5)
  Failed tests:  5-9
  Non-zero exit status: 5
Files=1, Tests=9,  5 wallclock secs ( 0.16 usr  0.03 sys +  4.48 cusr  0.34 csys =  5.01 CPU)
Result: FAIL

On another Debian 12 machine, development environment and latest develop, this unit file fails too but in a different manner:

~zonemaster/zonemaster-backend$ git log -1 --oneline
21212d4 (HEAD -> develop, upstream/develop) Merge pull request #1169 from MichaelTimbert/update-MANIFEST

~zonemaster/zonemaster-backend$ prove t/translator.t
t/translator.t .. 1/?
#   Failed test 'Translating a backend-specific TEST_DIED message tag works'
#   at t/translator.t line 82.
#                   'An error occured and Zonemaster could not start or finish the test.'
#     doesn't match '(?^u:\AUne erreur est survenue )'
# Looks like you failed 1 test of 9.
t/translator.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/9 subtests

Test Summary Report
-------------------
t/translator.t (Wstat: 256 (exited 1) Tests: 9 Failed: 1)
  Failed test:  8
  Non-zero exit status: 1
Files=1, Tests=9,  1 wallclock secs ( 0.02 usr  0.01 sys +  0.65 cusr  0.03 csys =  0.71 CPU)
Result: FAIL

Both environment have the exact same locales installed:

$ locale -a
C
C.utf8
da_DK.utf8
en_US.utf8
fi_FI.utf8
fr_FR.utf8
nb_NO.utf8
POSIX
sv_SE.utf8

[matsduf]

I also tested installing on a clean Debian 12 (12.5) and it fails, but 3 of 9 tests:

t/translator.t ............. 1/? 
#   Failed test 'Translating a B02_NS_BROKEN message works'
#   at t/translator.t line 63.
#                   'R?ponse cass?e du serveur de noms "ns1.example" ? une requ?te de type SOA.'
#     doesn't match '(?^u:\AR?ponse cass?e du serveur de noms )'

#   Failed test 'Translation is a string of Unicode codepoints, not bytes'
#   at t/translator.t line 65.
#                   'R?ponse cass?e du serveur de noms "ns1.example" ? une requ?te de type SOA.'
#     doesn't match '(?^u:cass\x{e9}e)'

#   Failed test 'Translating Consistency01 gives a string of Unicode codepoints'
#   at t/translator.t line 91.
#                   'Coh?rence du num?ro de s?rie du SOA'
#     doesn't match '(?^u:\ACoh\x{e9}rence du num\x{e9}ro de s\x{e9}rie)'
# Looks like you failed 3 tests of 9.
t/translator.t ............. Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/9 subtests 

Everything else passes.

[matsduf]

Still on Debian 12, if you first run

unset LANG ; unset LC_CTYPE ; unset LANGUAGE ; export LC_ALL=C.UTF-8

Then the unit tests will pass:

$ PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(1, 'inc', 'blib/lib', 'blib/arch')" t/translator.t 
t/translator.t .. 
ok 1 - use Zonemaster::Backend::Translator;
ok 2 - The class (or class-like) 'Zonemaster::Backend::Translator' isa 'Zonemaster::Engine::Translator'
ok 3 - 'Zonemaster::Backend::Translator->instance()' isa 'Zonemaster::Backend::Translator'
ok 4 - Setting locale to 'fr_FR.UTF-8' works
ok 5 - Translating a GLOBAL_VERSION message tag works
ok 6 - Translating a B02_NS_BROKEN message works
ok 7 - Translation is a string of Unicode codepoints, not bytes
ok 8 - Translating a backend-specific TEST_DIED message tag works
ok 9 - Translating Consistency01 gives a string of Unicode codepoints
1..9
ok
All tests successful.
Files=1, Tests=9,  1 wallclock secs ( 0.02 usr  0.01 sys +  0.73 cusr  0.09 csys =  0.85 CPU)
Result: PASS

[matsduf]

The new code in #1175 makes it pass in Debian 12, but not in Travis.

$ printenv | egrep "LC|LANG"
LC_ALL=C
$ PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(1, 'inc', 'blib/lib', 'blib/arch')" t/translator-new.t 
t/translator-new.t .. 
ok 1 - use Zonemaster::Backend::Translator;
ok 2 - The class (or class-like) 'Zonemaster::Backend::Translator' isa 'Zonemaster::Engine::Translator'
ok 3 - 'Zonemaster::Backend::Translator->instance()' isa 'Zonemaster::Backend::Translator'
ok 4 - Setting locale to 'fr_FR.UTF-8' works
ok 5 - Translating a GLOBAL_VERSION message tag works
ok 6 - Translating a B02_NS_BROKEN message works
ok 7 - Translation is a string of Unicode codepoints, not bytes
ok 8 - Translating a backend-specific TEST_DIED message tag works
ok 9 - Translating Consistency01 gives a string of Unicode codepoints
1..9
ok
All tests successful.
Files=1, Tests=9,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.69 cusr  0.09 csys =  0.80 CPU)
Result: PASS

[matsduf]

When I test t/translator.t without the modification in #1175 and with locale set to "C" (and not e.g. "C.UTF-8") then it fails on FreeBSD, but it still passes on Ubuntu 22.04). With the modification in #1175 it passes on FreeBSD even when the locale is set to "C".

[tgreenx]

On Ubuntu 24.04

Works when LANG is set:

$ printenv | grep -i utf
LANG=en_US.UTF-8
$ git log -1 --oneline
21212d4 (HEAD -> develop, origin/develop) Merge pull request #1169 from MichaelTimbert/update-MANIFEST
$ prove t/translator.t 
t/translator.t .. ok   
All tests successful.
Files=1, Tests=9,  1 wallclock secs ( 0.05 usr  0.01 sys +  1.21 cusr  0.16 csys =  1.43 CPU)
Result: PASS

Fails when LANG is unset:

$ unset LANG
$ printenv | grep -i utf
$ prove t/translator.t 
t/translator.t .. 1/? 
#   Failed test 'Translating a B02_NS_BROKEN message works'
#   at t/translator.t line 63.
#                   'R?ponse cass?e du serveur de noms "ns1.example" ? une requ?te de type SOA.'
#     doesn't match '(?^u:\AR�ponse cass�e du serveur de noms )'

#   Failed test 'Translation is a string of Unicode codepoints, not bytes'
#   at t/translator.t line 65.
#                   'R?ponse cass?e du serveur de noms "ns1.example" ? une requ?te de type SOA.'
#     doesn't match '(?^u:cass\x{e9}e)'

#   Failed test 'Translating Consistency01 gives a string of Unicode codepoints'
#   at t/translator.t line 91.
#                   'Coh?rence du num?ro de s?rie du SOA'
#     doesn't match '(?^u:\ACoh\x{e9}rence du num\x{e9}ro de s\x{e9}rie)'
# Looks like you failed 3 tests of 9.
t/translator.t .. Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/9 subtests 

Test Summary Report
-------------------
t/translator.t (Wstat: 768 (exited 3) Tests: 9 Failed: 3)
  Failed tests:  6-7, 9
  Non-zero exit status: 3
Files=1, Tests=9,  1 wallclock secs ( 0.05 usr  0.01 sys +  1.63 cusr  0.20 csys =  1.89 CPU)
Result: FAIL

[matsduf]

@tgreenx, with the fix in #1175 Ubuntu 24.04 passes even when LANG is unset, doesn't it?

[tgreenx]

@tgreenx, with the fix in #1175 Ubuntu 24.04 passes even when LANG is unset, doesn't it?

Yes, and I reported that in the PR: https://github.com/zonemaster/zonemaster-backend/pull/1175#pullrequestreview-2144949401

[tgreenx]

On my Debian 12 development environment though (i.e. no installation with cpnam), the fix in #1175 doesn't solve the singular error I get (with either LANG set or unset):

$ printenv | egrep "LC|LANG|utf|UTF"
LANG=en_US.UTF-8
$ git log -1 --oneline
81d55a4 (HEAD -> test-PR1175) Keeps Jammy for Postgresql
$ prove t/translator.t
t/translator.t .. 1/?
#   Failed test 'Translating a backend-specific TEST_DIED message tag works'
#   at t/translator.t line 100.
#                   'An error occured and Zonemaster could not start or finish the test.'
#     doesn't match '(?^u:\AUne erreur est survenue )'
# Looks like you failed 1 test of 9.
t/translator.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/9 subtests

Test Summary Report
-------------------
t/translator.t (Wstat: 256 (exited 1) Tests: 9 Failed: 1)
  Failed test:  8
  Non-zero exit status: 1
Files=1, Tests=9,  1 wallclock secs ( 0.01 usr  0.00 sys +  0.40 cusr  0.03 csys =  0.44 CPU)
Result: FAIL

$ unset LANG
$ printenv | egrep "LC|LANG|utf|UTF"
$ prove t/translator.t
t/translator.t .. 1/?
#   Failed test 'Translating a backend-specific TEST_DIED message tag works'
#   at t/translator.t line 100.
#                   'An error occured and Zonemaster could not start or finish the test.'
#     doesn't match '(?^u:\AUne erreur est survenue )'
# Looks like you failed 1 test of 9.
t/translator.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/9 subtests

Test Summary Report
-------------------
t/translator.t (Wstat: 256 (exited 1) Tests: 9 Failed: 1)
  Failed test:  8
  Non-zero exit status: 1
Files=1, Tests=9,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.40 cusr  0.02 csys =  0.43 CPU)
Result: FAIL

[matsduf]

@tgreenx, I have also tested on Debian 12, a clean installation, and I get a different result. What is the difference?

Below translator.t is the version current develop branch, and translator-new.t is the updated version from #1175.

$ printenv | grep -i utf
LANG=C.UTF-8

$ PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'inc', 'blib/lib', 'blib/arch')" t/translator.t 
t/translator.t .. ok   
All tests successful.
Files=1, Tests=9,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.67 cusr  0.06 csys =  0.75 CPU)
Result: PASS

1$ PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'inc', 'blib/lib', 'blib/arch')" t/translator-new.t 
t/translator-new.t .. ok   
All tests successful.
Files=1, Tests=9,  1 wallclock secs ( 0.02 usr  0.01 sys +  0.67 cusr  0.05 csys =  0.75 CPU)
Result: PASS

$ unset LANG
$ printenv | grep -i utf

$ PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'inc', 'blib/lib', 'blib/arch')" t/translator.t 
t/translator.t .. 1/? 
#   Failed test 'Translating a B02_NS_BROKEN message works'
#   at t/translator.t line 63.
#                   'R?ponse cass?e du serveur de noms "ns1.example" ? une requ?te de type SOA.'
#     doesn't match '(?^u:\AR?ponse cass?e du serveur de noms )'

#   Failed test 'Translation is a string of Unicode codepoints, not bytes'
#   at t/translator.t line 65.
#                   'R?ponse cass?e du serveur de noms "ns1.example" ? une requ?te de type SOA.'
#     doesn't match '(?^u:cass\x{e9}e)'

#   Failed test 'Translating Consistency01 gives a string of Unicode codepoints'
#   at t/translator.t line 91.
#                   'Coh?rence du num?ro de s?rie du SOA'
#     doesn't match '(?^u:\ACoh\x{e9}rence du num\x{e9}ro de s\x{e9}rie)'
# Looks like you failed 3 tests of 9.
t/translator.t .. Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/9 subtests 

Test Summary Report
-------------------
t/translator.t (Wstat: 768 (exited 3) Tests: 9 Failed: 3)
  Failed tests:  6-7, 9
  Non-zero exit status: 3
Files=1, Tests=9,  1 wallclock secs ( 0.02 usr  0.00 sys +  0.64 cusr  0.07 csys =  0.73 CPU)
Result: FAIL
Failed 1/1 test programs. 3/9 subtests failed.

$ PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'inc', 'blib/lib', 'blib/arch')" t/translator-new.t 
t/translator-new.t .. ok   
All tests successful.
Files=1, Tests=9,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.67 cusr  0.04 csys =  0.73 CPU)
Result: PASS

[tgreenx]

It works on a clean Debian 12 installation for me too. But not on a Debian 12 development environment (no installation). See my previous comment. @MichaelTimbert also got the same failing subtest on Ubuntu 22.04 (see https://github.com/zonemaster/zonemaster-backend/pull/1167#issuecomment-2186474371). He has yet to test it again with the fix in #1175 though.

[matsduf]

It would be interesting to understand why it fail sometimes.