Closed bitstreamout closed 11 years ago
plk
commented
11 years ago Hmm, have you pulled the latest? I merged into the master branch last night ready for the release so you should pull master and try that. For me, the master branch passes all tests cleanly.
bitstreamout
commented
11 years ago On Thu, Jul 04, 2013 at 02:13:28AM -0700, plk wrote:
Hmm, have you pulled the latest? I merged into the master branch last night ready for the release so you should pull master and try that. For me, the master branch passes all tests cleanly.
This is what I see:
[ 815s] Checking prerequisites...
[ 815s] requires:
[ 815s] ! Mozilla::CA is not installed
[ 815s] ! Unicode::Collate (0.97) is installed, but we need version >= 0.98
[ 815s]
[ 815s] ERRORS/WARNINGS FOUND IN PREREQUISITES. You may wish to install the versions
[ 815s] of the modules indicated above before proceeding with this installation
[ 815s]
[ 815s] Run 'Build installdeps' to install missing prerequisites.
[ 815s]
[ 815s] Created MYMETA.yml and MYMETA.json
[ 815s] Creating new 'Build' script for 'biblatex-biber' version '1.7'
[ 815s] + ./Build build flags=-j8
[ 815s] Building biblatex-biber
[ 815s] + rm t/remote-files.t
[ 815s] + ./Build test
[ 821s] t/basic-misc.t ........ ok
[ 822s] t/bcfvalidation.t ..... ok
[ 822s] t/biblatexml.t ........ ok
[ 823s] t/bibtex-aliases.t .... ok
[ 823s] t/bibtex-forms.t ...... ok
[ 823s] t/configfile.t ........ ok
[ 824s] t/crossrefs.t ......... ok
[ 825s] t/dm-constraints.t .... ok
[ 826s] t/dm-dateformats.t .... ok
[ 827s] t/encoding.t .......... ok
[ 828s] t/endnotexml.t ........ ok
[ 828s] t/extratitle.t ........ ok
[ 829s] t/extratitleyear.t .... ok
[ 829s] t/extrayear.t ......... ok
[ 830s] t/full.t .............. ok
[ 836s] t/labelalpha.t ........ ok
[ 837s] t/labelname.t ......... ok
[ 839s] t/names.t ............. ok
[ 840s] t/options.t ........... ok
[ 840s] t/pod-coverage.t ...... ok
[ 840s] t/pod.t ............... ok
[ 841s] t/related-entries.t ... ok
[ 841s] t/ris.t ............... ok
[ 842s] t/sections-complex.t .. ok
[ 843s] t/sections.t .......... ok
[ 843s] t/set-dynamic.t ....... ok
[ 843s] t/set-legacy.t ........ ok
[ 844s] t/set-static.t ........ ok
[ 844s] t/skips.t ............. ok
[ 845s] t/sort-case.t ......... ok
[ 845s] t/sort-complex.t ...... ok
[ 848s] t/sort-order.t ........ ok
[ 849s] t/sort-uc.t ........... ok
[ 850s]
[ 850s] # Failed test 'Editor type/class'
[ 850s] # at t/sorting.t line 186.
[ 850s] # got: 'redactor,Jaffe!Philipp,Loewenfeld!Samuel#Kaltenbrunner!Ferdinand#Ewald!Paul'
[ 850s] # expected: 'redactor,Jaff351!Philipp,Loewenfeld!Samuel#Kaltenbrunner!Ferdinand#Ewald!Paul'
[ 862s] # Looks like you failed 1 test of 42.
[ 862s] t/sorting.t ...........
[ 862s] Dubious, test returned 1 (wstat 256, 0x100)
[ 862s] Failed 1/42 subtests
[ 862s] t/sortlists.t ......... ok
[ 862s]
[ 862s] # Failed test 'bltxml tool mode - 1'
[ 862s] # at t/tool-bltxml.t line 210.
[ 862s] Wide character in print at /usr/lib/perl5/5.18.0/Test/Builder.pm line 1759.
[ 862s] # got: '<?xml version="1.0" encoding="UTF-8"?>
[ 862s] #
[ 862s] #
[ 862s] #
Next I've tried to make an su/chroot into the build system. First run `./Build test' works flawless, then the next run shows:
abuild@noether:~/rpmbuild/BUILD/biblatex-biber-1.7> ./Build test
t/basic-misc.t ........ ok
t/bcfvalidation.t ..... ok
t/biblatexml.t ........ ok
t/bibtex-aliases.t .... ok
t/bibtex-forms.t ...... ok
t/configfile.t ........ ok
t/crossrefs.t ......... ok
t/dm-constraints.t .... ok
t/dm-dateformats.t .... ok
t/encoding.t .......... ok
t/endnotexml.t ........ ok
t/extratitle.t ........ ok
t/extratitleyear.t .... ok
t/extrayear.t ......... ok
t/full.t .............. 1/5
t/full.t .............. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/5 subtests
t/labelalpha.t ........ ok
t/labelname.t ......... ok
t/names.t ............. ok
t/options.t ........... ok
t/pod-coverage.t ...... ok
t/pod.t ............... ok
t/related-entries.t ... ok
t/ris.t ............... ok
t/sections-complex.t .. ok
t/sections.t .......... ok
t/set-dynamic.t ....... ok
t/set-legacy.t ........ ok
t/set-static.t ........ ok
t/skips.t ............. ok
t/sort-case.t ......... ok
t/sort-complex.t ...... ok
t/sort-order.t ........ ok
t/sort-uc.t ........... ok
t/sorting.t ........... ok
t/sortlists.t ......... ok
t/tool-bltxml.t ....... 1/2
Wide character in print at /usr/lib/perl5/5.18.0/Test/Builder.pm line 1759.
[...]
t/tool-bltxml.t ....... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests
t/tool.t .............. ok
t/uniqueness.t ........ ok
t/utils.t ............. 1/22 Use of uninitialized value in concatenation (.) or string at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/LaTeX/Recode.pm line 234.
Use of uninitialized value in concatenation (.) or string at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/LaTeX/Recode.pm line 235.
Wide character in print at /usr/lib/perl5/5.18.0/Test/Builder.pm line 1759.
Use of uninitialized value in concatenation (.) or string at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/LaTeX/Recode.pm line 234. Use of uninitialized value in concatenation (.) or string at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/LaTeX/Recode.pm line 234.
Wide character in print at /usr/lib/perl5/5.18.0/Test/Builder.pm line 1759.
Use of uninitialized value in concatenation (.) or string at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/LaTeX/Recode.pm line 235.
Use of uninitialized value in concatenation (.) or string at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/LaTeX/Recode.pm line 234.
t/utils.t ............. Dubious, test returned 4 (wstat 1024, 0x400)
Failed 4/22 subtests
(less 1 skipped subtest: 17 okay)
t/xdata.t ............. ok
t/zoterordfxml.t ...... ok
Test Summary Report
t/full.t (Wstat: 256 Tests: 5 Failed: 1) Failed test: 2 Non-zero exit status: 1 t/tool-bltxml.t (Wstat: 256 Tests: 2 Failed: 1) Failed test: 1 Non-zero exit status: 1 t/utils.t (Wstat: 1024 Tests: 22 Failed: 4) Failed tests: 8, 10-11, 19 Non-zero exit status: 4 Files=41, Tests=945, 56 wallclock secs ( 0.21 usr 0.05 sys + 54.69 cusr 0.68 csys = 55.63 CPU) Result: FAIL Failed 3/41 test programs. 6/945 subtests failed.
... IMHO there is a race condition
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
bitstreamout
commented
11 years ago On Thu, Jul 04, 2013 at 02:13:28AM -0700, plk wrote:
Hmm, have you pulled the latest? I merged into the master branch last night ready for the release so you should pull master and try that. For me, the master branch passes all tests cleanly.
Yet some other random errors:
t/names.t ............. 1/54
Use of uninitialized value in concatenation (.) or string at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/LaTeX/Recode.pm line 235. Use of uninitialized value in concatenation (.) or string at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/LaTeX/Recode.pm line 235. t/names.t ............. 50/54 # Looks like you failed 2 tests of 54. t/names.t ............. Dubious, test returned 2 (wstat 512, 0x200) Failed 2/54 subtests
t/basic-misc.t ........ 4/55
t/basic-misc.t ........ 16/55
t/basic-misc.t ........ 29/55
t/basic-misc.t ........ Dubious, test returned 7 (wstat 1792, 0x700)
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago Looks like you have a broken install. One thing that looks very suspicious is your perl Text::BibTeX module - that appears to be broken if not the perl install itself - these are certainly not biber issues.
bitstreamout
commented
11 years ago On Thu, Jul 04, 2013 at 03:40:34AM -0700, plk wrote:
Looks like you have a broken install. One thing that looks very suspicious is your perl Text::BibTeX module - that appears to be broken if not the perl install itself - these are certainly not biber issues.
Hmmm ...
abuild@noether:~/rpmbuild/BUILD/biblatex-biber-1.7> rpm -qi --whatprovides 'perl(Text::BibTeX)' Name : perl-Text-BibTeX Version : 0.66 Release : 4.4 Architecture: x86_64 Install Date: Fri Jun 28 10:24:32 2013 Group : Development/Libraries/Perl Size : 483887 License : Artistic-1.0 or GPL-1.0+
abuild@noether:~/rpmbuild/BUILD/biblatex-biber-1.7> rpm -qi perl
Name : perl
Version : 5.18.0
Release : 1.1
Architecture: x86_64
Install Date: Fri Jun 28 10:23:59 2013
Group : Development/Languages/Perl
Size : 38907986
License : Artistic-1.0 or GPL-2.0+
does this mean that perl-Text-BibTeX 0.66 is to old for biblatex-biber-1.7/1.6
Werner
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago I just saw the other part you sent. It is very mysterious that the second test run fails but the first not. Looking at the errors, it seems to related to UTF-8 and locales perhaps?
bitstreamout
commented
11 years ago On Thu, Jul 04, 2013 at 03:50:56AM -0700, plk wrote:
I just saw the other part you sent. It is very mysterious that the second test run fails but the first not. Looking at the errors, it seems to related to UTF-8 and locales perhaps?
I've tried several locale ... first was LANG=POSIX and LC_CTYPE=en_US.UTF-8 then I've tried LANG=POSIX and LC_CTYPE unset, then LANG=en_US.UTF-8, next LANG=en_US, LANG=de_DE.UTF-8, LANG=en_US.ISO-8859-1 ... but the random behaviour still remains :(
Werner
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago This looks like a broken perl install or some really broken modules - I have never seen anything like this before ...
bitstreamout
commented
11 years ago On Thu, Jul 04, 2013 at 07:49:31AM -0700, plk wrote:
This looks like a broken perl install or some really broken modules - I have never seen anything like this before ...
Hmmm ... we're talking about a clean chroot environment used to build a package. Clean means that all packages required will be installed from the same project. All packages used for such a build are build with the help of virtual systems only started and installed to build only one package.
How this is done you can see at https://build.opensuse.org/
The Opensuse Build Service (OBS) provides this build scheme on fresh started virtual build host and for developers also for local systems.
For perl and its most used modules we have the main project devel:languages:perl
https://build.opensuse.org/project/show?project=devel%3Alanguages%3Aperl
as well as a log of sub projects CPAN-A upto CPAN-Z
https://build.opensuse.org/project/subprojects?project=devel%3Alanguages%3Aperl
all build and packed on a virtual build host.
Werner
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago I'm not sure how I can help here - I am sure if you just install perl5.16 on a normal opensuse box, run
perl ./Build.Pl ./Build installdeps ./Build test
it will work fine and so this is something to do with OBS.I think this will have to closed as too localised.
bitstreamout
commented
11 years ago On Fri, Jul 05, 2013 at 12:38:17AM -0700, plk wrote:
I'm not sure how I can help here - I am sure if you just install perl5.16 on a normal opensuse box, run
perl ./Build.Pl ./Build installdeps ./Build test
it will work fine and so this is something to do with OBS.I think this will have to closed as too localised.
Does this mean that biblatex-biber-1.6 and above is not perl 5.18 safe? The installdeps tree is a nogo for a clean build environment as it would install foreign packages without any control about security and legal certainty. Beside this there is no remote network in such build environment.
Werner
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago The should be no perl version issues but biber 1.6+ require perl 5.16+. I have not tested 5.18 yet but will do so.
bitstreamout
commented
11 years ago On Fri, Jul 05, 2013 at 01:49:44AM -0700, plk wrote:
The should be no perl version issues but biber 1.6+ require perl 5.16+. I have not tested 5.18 yet but will do so.
Found this
-> https://aur.archlinux.org/packages/biblatex-biber/?comments=all
that is that perl 5.18 now includes perl-Unicode-Collate ... maybe there the cause of the trouble is located.
Werner
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago No, perl has included Unicode::Collate for a long time. biber requests a more recent version than the bundled one though so it has to be updated usually. Now Unicode Collate changed its install behaviour with version 0.90 so that any update installs to site libs instead of overwriting the bundles module ... perhaps that is related. To update U::C, I used the cpan module otherwise, as mentioned in the thread you pointed to, you can be prompted to update all of perl as U::C is a core module.
bitstreamout
commented
11 years ago On Fri, Jul 05, 2013 at 02:14:09AM -0700, plk wrote:
No, perl has included Unicode::Collate for a long time. biber requests a more recent version than the bundled one though so it has to be updated usually. Now Unicode Collate changed its install behaviour with version 0.90 so that any update installs to site libs instead of overwriting the bundles module ... perhaps that is related. To update U::C, I used the cpan module otherwise, as mentioned in the thread you pointed to, you can be prompted to update all of perl as U::C is a core module.
See https://rt.cpan.org/Public/Bug/Display.html?id=85621
... this would explain why t/tool-bltxml.t fails that often, maybe there are more places in biber where length in bytes are assumed but perl 5.18 now uses count of (wide) characters
Werner
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago This shouldn't be possible in biber - it's UTF-8 only internally.
bitstreamout
commented
11 years ago On Fri, Jul 05, 2013 at 02:14:09AM -0700, plk wrote:
No, perl has included Unicode::Collate for a long time. biber requests a more recent version than the bundled one though so it has to be updated usually. Now Unicode Collate changed its install behaviour with version 0.90 so that any update installs to site libs instead of overwriting the bundles module ... perhaps that is related. To update U::C, I used the cpan module otherwise, as mentioned in the thread you pointed to, you can be prompted to update all of perl as U::C is a core module.
Found in an strace the following:
write(2, "Use of uninitialized value in subroutine entry at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/Utils.pm line 939.\n", 131) = 131
caused by a
strace -s 256 -e write -f ./Build test --test_files t/tool-bltxml.t
at 939 in lib/Biber/Utils.pm I found
sub biber_decode_utf8 { return NFD(decode_utf8(shift));# Unicode NFD boundary }
Our perl guru's here told me that it looks like biblatex-biber is not safe for perl 5.18 with its hash randomization, see http://search.cpan.org/dist/perl-5.18.0/pod/perldelta.pod#Hash_randomization and that "Wide character in print" of the error message caused by t/tool-bltxml.t may depict that the output stream used by Test::Builder is not utf8 safe.
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago I don't depend on hash order in the code so I think this is unlikely. I still need to test with perl 5.18 though. The wide character is normal for test failures as it exposes some UTF-8 to the test harness, returned in the error strings. Biber is UTF-8 safe.
bitstreamout
commented
11 years ago On Mon, Jul 08, 2013 at 01:47:51AM -0700, plk wrote:
I don't depend on hash order in the code so I think this is unlikely. I still need to test with perl 5.18 though. The wide character is normal for test failures as it exposes some UTF-8 to the test harness, returned in the error strings. Biber is UTF-8 safe.
Hmmm ... I've just tried out a little test with perl 5.16 and with perl 5.18 with the following test scriptlet:
biblatex-biber-1.7> cat test.perl my %hash1 = ( a => 1, b => 2, c => 3); my %hash2 = %hash1; print join " ", keys %hash1; print "\n"; print join " ", keys %hash2; print "\n";
with perl 5.18 from within the build system:
biblatex-biber-1.7> perl test.perl c b a a b c biblatex-biber-1.7> perl test.perl b a c c a b biblatex-biber-1.7> perl test.perl b a c a c b biblatex-biber-1.7> perl test.perl c b a b a c
but with perl 5.16
werner/Tests> perl test.perl c a b c a b werner/Tests> perl test.perl c a b c a b werner/Tests> perl test.perl c a b c a b werner/Tests> perl test.perl c a b c a b
don't know if this simple example reflects the results I've seen with biblatex-biber-1.7 and perl 5.18 but the random results (fail/success) may indicate a similar behaviour:
biblatex-biber-1.7> find -name '.p' | xargs grep 'join' | grep -E 'keys %' ./lib/Biber/Config.pm: return scalar keys %{$CONFIG->{state}{uniquelistcount}{minyear}{join("\x{10FFFD}", @$minyearnamelist)}{$year}}; ./lib/Biber/Input/file/bibtex.pm: $logger->debug('Text::BibTeX cache keys: ' . join(', ', keys %{$cache->{data}{$filename}})); ./lib/Biber/LaTeX/Recode.pm: $remaps->{$set}{$type}{re} = '[' . join('', keys %{$remaps->{$set}{$type}{map}}) . ']'; ./lib/Biber/LaTeX/Recode.pm: $r_remaps->{$set}{$type}{re} = '[' . join('', keys %{$rremaps->{$set}{$type}{map}}) . ']'; ./lib/Biber/LaTeX/Recode.pm: $remaps->{$set}{$type}{re} = join('|', map { /[+-)(]/ ? '\' . $ : $_ } keys %{$remaps->{$set}{$type}{map}}); ./lib/Biber/LaTeX/Recode.pm: $rremaps->{$set}{$type}{re} = join('|', map { /[+-)(]/ ? '\' . $ : $_ } keys %{$r_remaps->{$set}{$type}{map}}); ./lib/Biber/LaTeX/Recode.pm: $remaps->{$set}{$type}{re} = join('|', keys %{$remaps->{$set}{$type}{map}}); ./lib/Biber/LaTeX/Recode.pm: $r_remaps->{$set}{$type}{re} = join('|', keys %{$r_remaps->{$set}{$type}{map}});
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago I will get a perl 5.18 install to test.
bitstreamout
commented
11 years ago On Mon, Jul 08, 2013 at 04:59:36AM -0700, plk wrote:
I will get a perl 5.18 install to test.
Thank you :)
JFYI as I've just found the module 'Tie::IxHash' which helps a lot but may have some other side effects like menory consumption:
biblatex-biber-1.7> cat test use Tie::IxHash; my @keys = (a, b, c); tie my %hash1, 'Tie::IxHash'; tie my %hash2, 'Tie::IxHash'; %hash1 = (a => 1, b => 2, c => 3); %hash2 = %hash1; print join " ", keys %hash1; print "\n"; print join " ", keys %hash2; print "\n"; untie %hash1; untie %hash2; biblatex-biber-1.7> perl test a b c a b c
Werner
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago I have just run the test suite under Windows Strawberry perl 5.18 and it's all fine for me ...
bitstreamout
commented
11 years ago On Mon, Jul 08, 2013 at 11:35:53PM -0700, plk wrote:
I have just run the test suite under Windows Strawberry perl 5.18 and it's all fine for me ...
And your're sure that the perl 5.18 on Windows[tm] has enabled randomization of the hash table implementation. Do have tried my test perl script? If there are no random results the major feature of perl 5.18 is not enabled.
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago Yes, I get random results on the perl 5.18 with which I ran the tests. I don't think this is the problem - I don't rely on the order of hashes in biber - no perl programmer who has a clue has ever done that anyway - this is why it's possible for perl 5.18 to make this change at all ...
bitstreamout
commented
11 years ago On Tue, Jul 09, 2013 at 12:19:17AM -0700, plk wrote:
Yes, I get random results on the perl 5.18 with which I ran the tests. I don't think this is the problem - I don't rely on the order of hashes in biber - no perl programmer who has a clue has ever done that anyway - this is why it's possible for perl 5.18 to make this change at all ...
Just removed almost all modules and have done `./Build installdeps' as root but the problem remains, there are random errors mainly with missing or wrong UTF-8 multi byte characters and t/tool-bltxml.t fails very often.
Just to be noticed: the `./Build installdeps' had not shown any fatal error only some test had been skipped do missed modules.
Now what can I do as next?
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago I suspect a bad perl install at the moment. Did you compile perl yourself? Did all tests pass?
bitstreamout
commented
11 years ago On Tue, Jul 09, 2013 at 05:56:12AM -0700, plk wrote:
I suspect a bad perl install at the moment. Did you compile perl yourself? Did all tests pass?
Just have a look at https://build.opensuse.org/package/live_build_log/devel:languages:perl/perl/openSUSE_Factory/x86_64
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago This isn't a normal perl build run - it's some RPM chain build. I would start by getting the perl 5.18 source and compiling from scratch and then try to use the resulting perl. It is usually an easy compile on Linux.
bitstreamout
commented
11 years ago On Tue, Jul 09, 2013 at 11:18:35PM -0700, plk wrote:
This isn't a normal perl build run - it's some RPM chain build. I would start by getting the perl 5.18 source and compiling from scratch and then try to use the resulting perl. It is usually an easy compile on Linux.
It is ia complete build log done on a virtual system which including installation and activating the system as well as instalaltion all required packages and the build ist self followed by tests for file conflicts, compiler warnings anfd much more.
In this log you'll find after searching for the string
./runtests choosethat is line 22192 and below the results of the test suite.
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
bitstreamout
commented
11 years ago It looks like I'm not able to attach anything other that PNG, GIF, or JPG :(
plk
commented
11 years ago I can really only help when we show that this is a biber problem - so far this looks like a perl issue on the platform and until we can replicate this problem with a standard perl build, it's hard to be able to say anything useful. Can you compile perl normally on an openSUSE box and see what happens?
bitstreamout
commented
11 years ago On Wed, Jul 10, 2013 at 01:00:35AM -0700, plk wrote:
I can really only help when we show that this is a biber problem - so far this looks like a perl issue on the platform and until we can replicate this problem with a standard perl build, it's hard to be able to say anything useful. Can you compile perl normally on an openSUSE box and see what happens?
Had branched devel:languages:perl/perl, then disabled not required patches, switched the gcc version from 4.8 back to 4.7 and verified that. After the build I've run the command
make testwhich shows me a long list and finally:
All tests successful.
u=4.13 s=0.87 cu=358.74 cs=25.29 scripts=2319 tests=685709nevertheless exactly this version shows also random behaviour with biber.
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago Perhaps there is a forum where someone can verify this? Which version of SUSE? I have a VM somewhere but it won't be the latest.
bitstreamout
commented
11 years ago On Wed, Jul 10, 2013 at 04:42:50AM -0700, plk wrote:
Perhaps there is a forum where someone can verify this? Which version of SUSE? I have a VM somewhere but it won't be the latest.
It is openSUSE Factory within a chroot environment and/or a VM. Currently I run a build with perl 5.18 and the configure option
-DNO_HASH_SEED
simply to sure that this is out of question.
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago and with that flag, you still see the problems?
bitstreamout
commented
11 years ago On Wed, Jul 10, 2013 at 04:52:35AM -0700, plk wrote:
and with that flag, you still see the problems?
Accordingly to my simply test script this flag does not work as described in INSTALL :(( Indeed with biber I see exactyl the same behaviour with biber
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago What happend if you set the environment variable
PERL_HASH_SEED=0
? With my 5.18 install, this makes your previous test script return the same every time.
bitstreamout
commented
11 years ago On Wed, Jul 10, 2013 at 05:42:26AM -0700, plk wrote:
What happend if you set the environment variable
PERL_HASH_SEED=0
? With my 5.18 install, this makes your previous test script return the same every time.
This makes no difference ... also setting -DUSE_HASH_SEED_EXPLICIT and -Accflags='-DNO_HASH_SEED -DUSE_HASH_SEED_EXPLICIT' [1] ... the biber test suite remains random.
... maybe we see here a race condition
[1] ... now I see with perl -V
Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES MULTIPLICITY NO_HASH_SEED PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_PRESERVE_IVUV PERL_SAWAMPERSAND PERL_USE_SAFE_PUTENV USE_64_BIT_ALL USE_64_BIT_INT USE_HASH_SEED_EXPLICIT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API
and as the NO_HASH_SEED is only used in perl.c for printing compile time options
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago Hmm. I'm not convinved that this is even the issue then. Also, I never rely on hash order in biber anyway. I'm not sure what to recommend now - I could try to compile on a SUSE box but I'd need the same lib versions etc. as this looks quite low level. I'm not sure I'll have time for some time to set this up though.
bitstreamout
commented
11 years ago On Wed, Jul 10, 2013 at 06:35:34AM -0700, plk wrote:
Hmm. I'm not convinved that this is even the issue then. Also, I never rely on hash order in biber anyway. I'm not sure what to recommend now - I could try to compile on a SUSE box but I'd need the same lib versions etc. as this looks quite low level. I'm not sure I'll have time for some time to set this up though.
Q: Is the perl 5.18 you're using on Windows[tm] with or without thread support? This because I'm comparing two outputs of
PERLLIB=$PWD/blib/lib perl -DS t/tool-bltxml.t > log
one with success in first test and one with failed first test and they differ a lot in order ... also the other logs of failing runs do differ not only in temporary name but in order of execution.
Also tracing shows up those differences
PERLLIB=$PWD/blib/lib perl -Dt t/tool-bltxml.t.skipped 2> tlog
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago Ok, I seem to be able to reproduce this on an old linux box with a new 5.18 install. I suspect that the test suite module might rely on hash order ... urgh. I'm fairly certain you're right that this is hash randomisation issues but it's not so easy to track down as I thought (http://www.perlmonks.org/?node_id=1005122). Will look into it. This will be priority for biber 1.8. I have a nasty feeling that this is also due to overlooked has issue in some dependency module(s).
plk
commented
11 years ago If I set PERL_HASH_SEED=0, the tests are much better. full.t will always give problems if "which perl" returns a perl other that the one you are testing with - this test is now disabled by default in the new dev branch. I still get what looks like hash problems with tool-bltxml.t and I can't work that out yet.
plk
commented
11 years ago All tests reliably pass now for me with PERL_HASH_SEED=0 with perl 5.18 using the new 1.8 dev branch. I am looking into the hash issues which are in the Recode.pm module. Many thanks for persisting with this - I didn't quite realise how subtle the hash seed changes in perl 5.18 could be.
plk
commented
11 years ago Actually, can you try the dev branch now? It passes cleanly for me with new hash algorithm and perl 5.18 - I think I found the reason why and it would only impact tests.
bitstreamout
commented
11 years ago On Thu, Jul 11, 2013 at 02:03:26PM -0700, plk wrote:
Actually, can you try the dev branch now? It passed cleanly for me with new hash algorithm and perl 5.18 - I think I found the reason why and it would only impact tests.
... I've done yesterday a patch on biber 1.7 which makes the test suite survive ... only t/tool-bltxml.t shows a random order of the entries. I'll compare this with you're dev branch
Werner
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago That's interesting - the patch I made fixed tool-bltmx.t and when I tested, everything else too ...
bitstreamout
commented
11 years ago On Thu, Jul 11, 2013 at 11:45:09PM -0700, plk wrote:
That's interesting - the patch I made fixed tool-bltmx.t and this patch also resolved everything else.
The patch resolves the rare problems I've seen. Hopefully this channel accepts attachments as I try to attach the patch.
Werner
"Having a smoking section in a restaurant is like having a peeing section in a swimming pool." -- Edward Burr
plk
commented
11 years ago You can mail me your suggested patch - would be interested to see it. Philip [at] kime.org.uk
plk
commented
11 years ago Ok. Let's close this - I think we know what the problem is now and it is fixed in 1.8.
norbusan
commented
11 years ago Hi Philip, hi Werner,
could you specify which commit I would have to use on Debian? I am running into the same problems as mentioned above.
Werner: can you send my your patch per email?
By setting PERL_HASH_SEED=0 I get only one failed test, the tool-bltxml.t. But I don't know how to fix that one.
Any remarks would be great.
Thanks
Norbert
Seen in build log with t/utils.t
[ 876s] Use of uninitialized value in concatenation (.) or string at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/LaTeX/Recode.pm line 234. [ 876s] Use of uninitialized value in concatenation (.) or string at /home/abuild/rpmbuild/BUILD/biblatex-biber-1.7/blib/lib/Biber/LaTeX/Recode.pm line 235.
also from 10 tests of t/tool-bltxml.t 9 do fail ... guess there is a racy condition