p5pRT
commented
6 years ago Open p5pRT opened 6 years ago
p5pRT
commented
6 years ago p5pRT
commented
6 years ago On Mon\, 20 Aug 2018 23:42:55 GMT\, 11000@gmx.de wrote:
Hi\, I would like to provide a little patch that fixes a bug introduced somewhere around 5.26. The bug makes perl stall immediately because all hash keys will go into the same bucket\, because the hash-split is broken. Fortunately\, this bug is only triggered if you use the compile-time switch -Accflags=-DPERL_PERTURB_KEYS_DISABLED. I reckon i am the first person to ever use this switch since around 5.26! :) The patch i provided is very simple and fixes the bug by moving the in_collision flag assignment out of the #ifdef PERL_RANDOMIZE_KEYS. I think this assignment must always be included in the code\, no matter if the randomize keys feature is enabled or not. It is important that this assignment is computed always\, because the new hash split algorithm introduced somewhere around 5.26 will only split if the hash is 1) full (up to a certain level) and 2) a collision happens. If however the in_collision flag is never set\, the hash will never split. So we have to move its assignment out of that #ifdef. Regards\,Thomas Schweiger
Shouldn't the 2nd switch you mention be:
PERL_HASH_RANDOMIZE_KEYS
-- James E Keenan (jkeenan@cpan.org)
p5pRT
commented
6 years ago The RT System itself - Status changed from 'new' to 'open'
p5pRT
commented
6 years ago On Mon\, 20 Aug 2018 23:42:55 GMT\, 11000@gmx.de wrote:
Hi\, I would like to provide a little patch that fixes a bug introduced somewhere around 5.26. The bug makes perl stall immediately because all hash keys will go into the same bucket\, because the hash-split is broken. Fortunately\, this bug is only triggered if you use the compile-time switch -Accflags=-DPERL_PERTURB_KEYS_DISABLED. I reckon i am the first person to ever use this switch since around 5.26! :) The patch i provided is very simple and fixes the bug by moving the in_collision flag assignment out of the #ifdef PERL_RANDOMIZE_KEYS. I think this assignment must always be included in the code\, no matter if the randomize keys feature is enabled or not. It is important that this assignment is computed always\, because the new hash split algorithm introduced somewhere around 5.26 will only split if the hash is 1) full (up to a certain level) and 2) a collision happens. If however the in_collision flag is never set\, the hash will never split. So we have to move its assignment out of that #ifdef. Regards\,Thomas Schweiger
1. I tried to build Perl 5 blead by adding your switch to my regular configuration syntax:
##### sh ./Configure -des -Dusedevel -Accflags=-DPERL_PERTURB_KEYS_DISABLED #####
I get this previously unseen compile-time warning:
##### ... cc -c -DPERL_CORE -DPERL_PERTURB_KEYS_DISABLED -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c89 -O2 -Wall -Werror=declaration-after-statement -Werror=pointer-arith -Wextra -Wc++-compat -Wwrite-strings hv.c hv.c: In function ‘Perl_hv_rand_set’: hv.c:2344:23: warning: unused variable ‘iter’ [-Wunused-variable] struct xpvhv_aux *iter; ^ hv.c:2343:28: warning: unused parameter ‘hv’ [-Wunused-parameter] Perl_hv_rand_set(pTHX_ HV *hv\, U32 new_xhv_rand) { ^ hv.c:2343:36: warning: unused parameter ‘new_xhv_rand’ [-Wunused-parameter] Perl_hv_rand_set(pTHX_ HV *hv\, U32 new_xhv_rand) { ^ hv.c: In function ‘Perl_hv_common’: hv.c:886:8: warning: ‘in_collision’ may be used uninitialized in this function [-Wmaybe-uninitialized] if ( in_collision && DO_HSPLIT(xhv) ) { ^ ... #####
Then 'make' appeared to grind to a halt at:
##### ... Processing extracted/DJoinGroup.txt Processing extracted/DJoinType.txt Processing Jamo.txt Processing UnicodeData.txt #####
I did not attempt to proceed to 'make test'.
2. I created a branch and applied your patch to the branch:
smoke-me/jkeenan/133459-perturb
I configured and built perl as I normally do. 'make test_harness' PASS.
3. In the patch branched\, I added your switch to my regular configuration command. I got the same compile-time warning as mentioned above\, but 'make' did not appear to hang. However\, I got failures in three tests:
##### run/runenv.t (Wstat: 0 Tests: 106 Failed: 5) Failed tests: 46\, 48\, 50\, 52\, 54 op/each.t (Wstat: 0 Tests: 59 Failed: 1) Failed test: 56 ../ext/Devel-Peek/t/Peek.t (Wstat: 1280 Tests: 132 Failed: 5) Failed tests: 55\, 57\, 59\, 63\, 65 Non-zero exit status: 5 #####
Output attached.
-- James E Keenan (jkeenan@cpan.org)
p5pRT
commented
6 years ago # Failed test 56 - each() after insert produces warnings at op/each.t line 276 op/each.t ................... 1..59 ok 1 - keys ok 2 - values ok 3 - each count ok 4 - added a key ok 5 - resize ok 6 - same size ok 7 - still same size ok 8 - size 8 ok 9 - test scalar each ok 10 - test scalar keys resets iterator ok 11 - test iterator of each is being maintained ok 12 - test values keys resets iterator ok 13 - assign to keys does not shrink hash bucket array ok 14 - assignment to keys of a number not large enough does not change size ok 15 - keys (%hash) ok 16 ok 17 # DESTROY called ok 18 ok 19 - Check length of "\x{12345}" ok 20 - Check length of "\x{10fffd}" ok 21 - Check length of "\x{1234}" ok 22 - Check length of "\x{123}" ok 23 - Check length of "\022" ok 24 - copied unicode hash keys correctly? ok 25 - utf8 key in bytes hash ok 26 - bytes key in utf8 hash # non-utf8 ok 27 - if we got here change 8056 worked # utf8 ok 28 - change 8056 is thanks to Inaba Hiroto ok 29 - check encoding on EBCDIC ok 30 - check length ok 31 - check value ok 32 ok 33 - four different Unicode keys ok 34 - U+00FF -> 1 ok 35 - U+00C3 U+00BF -> 2 ok 36 - U+0100 -> 3 ok 37 - U+00C4 U+0080 -> 4 ok 38 - each demands argument ok 39 - keys demands argument ok 40 - values demands argument ok 41 - if(%foo) was true - my ok 42 - if(%foo) didnt mess with each (key) - my ok 43 - if(%foo) didnt mess with each (value) - my ok 44 - Got the expected number of keys - my ok 45 - Got the count of keys from %foo in scalar assignment context - my ok 46 - Got expected number of elements in list context - my ok 47 - if(%foo) was true - our ok 48 - if(%foo) didnt mess with each (key) - our ok 49 - if(%foo) didnt mess with each (value) - our ok 50 - Got the expected number of keys - our ok 51 - Got the count of keys from %foo in scalar assignment context - our ok 52 - Got expected number of elements in list context - our ok 53 - single key not yet freed ok 54 - single key now freed ok 55 - each on apparently empty hash does not leave RITER set not ok 56 - each() after insert produces warnings ok 57 - no warnings 'internal' silences each() after insert warnings ok 58 - each in list assignment ok 59 - values in list assignment Failed 1/59 subtests # Failed test 46 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr at run/runenv.t line 232 # Failed test 48 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr at run/runenv.t line 239 # Failed test 50 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stderr at run/runenv.t line 244 # Failed test 52 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stderr at run/runenv.t line 249 # Failed test 54 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stderr at run/runenv.t line 254 # Failed test 'ENAME on a stash' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAME = "RWOM" # ENAME = "RWOM" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAME = "RWOM" # ENAME = "RWOM" # \Z)' # # Failed test 'ENAMEs on a stash' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 3 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAME = "RWOM" # NAMECOUNT = 2 # ENAME = "RWOM"\, "KLANK" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 3 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAME = "RWOM" # NAMECOUNT = 2 # ENAME = "RWOM"\, "KLANK" # \Z)' # # Failed test 'ENAMEs on a stash with no NAME' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 3 # FLAGS = (OOK\,SHAREKEYS\,OVERLOAD) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAMECOUNT = -3 # ENAME = "RWOM"\, "KLANK" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 3 # FLAGS = \(OOK\,SHAREKEYS\,OVERLOAD\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAMECOUNT = -3 # ENAME = "RWOM"\, "KLANK" # \Z)' # # Failed test 'small hash after keys' # at t/Peek.t line 98. # # 'SV = IV(0x2e19c20) at 0x2e19c30 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25de050 # SV = PVHV(0x2d92a90) at 0x25de050 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2e23b70 (0:6\, 1:2) # hash quality = 125.0% # KEYS = 2 # FILL = 2 # MAX = 7 # RITER = -1 # EITER = 0x0 # # Elt "Beer" HASH = 0xaf2bb028 # SV = PV(0x2dfc2e0) at 0x2e154e0 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7670 "Foamy"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # Elt "Perl" HASH = 0xe22b31e6 # SV = PV(0x2dfc2d0) at 0x2e19f48 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7820 "Rules"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ \(0:[67]\,.*\) # hash quality = [0-9.]+% # KEYS = 2 # FILL = [12] # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # (?: Elt "(?:Perl|Beer)" HASH = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(POK\,(?:IsCOW\,)?pPOK\) # PV = 0x[[:xdigit:]]+ "(?:Rules|Foamy)"\\0 # CUR = \d+ # LEN = \d+ # COW_REFCNT = 1 # ){2}\Z)' # # Failed test 'small hash after keys and scalar' # at t/Peek.t line 98. # # 'SV = IV(0x2e19c20) at 0x2e19c30 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25de050 # SV = PVHV(0x2d92a90) at 0x25de050 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2e23b70 (0:6\, 1:2) # hash quality = 125.0% # KEYS = 2 # FILL = 2 # MAX = 7 # RITER = -1 # EITER = 0x0 # # Elt "Beer" HASH = 0xaf2bb028 # SV = PV(0x2dfc2e0) at 0x2e154e0 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7670 "Foamy"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # Elt "Perl" HASH = 0xe22b31e6 # SV = PV(0x2dfc2d0) at 0x2e19f48 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7820 "Rules"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ \(0:[67]\,.*\) # hash quality = [0-9.]+% # KEYS = 2 # FILL = ([12]) # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # (?: Elt "(?:Perl|Beer)" HASH = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(POK\,(?:IsCOW\,)?pPOK\) # PV = 0x[[:xdigit:]]+ "(?:Rules|Foamy)"\\0 # CUR = \d+ # LEN = \d+ # COW_REFCNT = 1 # ){2}\Z)' # run/runenv.t ................ 1..106 ok 1 - PERL5OPT => -w\,-e\,print $::x stdout ok 2 - PERL5OPT => -w\,-e\,print $::x stderr ok 3 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $::x stdout ok 4 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $::x stderr ok 5 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $x stdout ok 6 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $x stderr ok 7 - PERL5OPT => -Mstrict -w\,-I../lib\,-e\,print $x stdout ok 8 - PERL5OPT => -Mstrict -w\,-I../lib\,-e\,print $x stderr ok 9 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stdout ok 10 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stderr ok 11 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stdout ok 12 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stderr ok 13 - PERL5OPT => -MExporter\,-I../lib\,-e0 stdout ok 14 - PERL5OPT => -MExporter\,-I../lib\,-e0 stderr ok 15 - PERL5OPT => -MExporter -MExporter\,-I../lib\,-e0 stdout ok 16 - PERL5OPT => -MExporter -MExporter\,-I../lib\,-e0 stderr ok 17 - PERL5OPT => -Mstrict -Mwarnings\,-I../lib\,-e\,print "ok" if $INC{"strict.pm"} and $INC{"warnings.pm"} stdout ok 18 - PERL5OPT => -Mstrict -Mwarnings\,-I../lib\,-e\,print "ok" if $INC{"strict.pm"} and $INC{"warnings.pm"} stderr ok 19 - PERL5OPT => -I. -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stdout ok 20 - PERL5OPT => -I. -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stderr ok 21 - PERL5OPT => -I./ -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stdout ok 22 - PERL5OPT => -I./ -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stderr ok 23 - PERL5OPT => -w -w\,-e\,print $ENV{PERL5OPT} stdout ok 24 - PERL5OPT => -w -w\,-e\,print $ENV{PERL5OPT} stderr ok 25 - PERL5OPT => -t\,-e\,print ${^TAINT} stdout ok 26 - PERL5OPT => -t\,-e\,print ${^TAINT} stderr ok 27 - PERL5OPT => -W\,-I../lib\,-e\,local $^W = 0; no warnings; print $x stdout ok 28 - PERL5OPT => -W\,-I../lib\,-e\,local $^W = 0; no warnings; print $x stderr ok 29 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stdout ok 30 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stderr ok 31 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stdout ok 32 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stderr ok 33 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stdout ok 34 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stderr ok 35 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stdout ok 36 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stderr ok 37 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "foo" } @INC stdout ok 38 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "foo" } @INC stderr ok 39 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "bar" } @INC stdout ok 40 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "bar" } @INC stderr ok 41 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 42 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 43 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 44 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 45 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout not ok 46 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 47 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout not ok 48 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 49 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stdout not ok 50 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stderr ok 51 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stdout not ok 52 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stderr ok 53 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stdout not ok 54 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stderr ok 55 - PERL_HASH_SEED => 12345678\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 56 - PERL_HASH_SEED => 12345678\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 57 - PERL_HASH_SEED => 12\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 58 - PERL_HASH_SEED => 12\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 59 - PERL_HASH_SEED => 123456789\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 60 - PERL_HASH_SEED => 123456789\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 61 - No errors when determining @INC ok 62 - . is not in @INC ok 63 - No errors when determining unsafe @INC ok 64 - . last in unsafe @INC ok 65 - No errors when determining @INC for nothing ok 66 - expected number of elements in @INC for nothing ok 67 - expected elements in @INC for nothing ok 68 - No errors when determining @INC for something ok 69 - expected number of elements in @INC for something ok 70 - expected elements in @INC for something ok 71 - No errors when determining @INC for two things ok 72 - expected number of elements in @INC for two things ok 73 - expected elements in @INC for two things ok 74 - No errors when determining @INC for two things\, :: ok 75 - expected number of elements in @INC for two things\, :: ok 76 - expected elements in @INC for two things\, :: ok 77 - No errors when determining @INC for : at start ok 78 - expected number of elements in @INC for : at start ok 79 - expected elements in @INC for : at start ok 80 - No errors when determining @INC for : at end ok 81 - expected number of elements in @INC for : at end ok 82 - expected elements in @INC for : at end ok 83 - No errors when determining @INC for :: sandwich :: ok 84 - expected number of elements in @INC for :: sandwich :: ok 85 - expected elements in @INC for :: sandwich :: ok 86 - No errors when determining @INC for : ok 87 - expected number of elements in @INC for : ok 88 - expected elements in @INC for : ok 89 - No errors when determining @INC for :: ok 90 - expected number of elements in @INC for :: ok 91 - expected elements in @INC for :: ok 92 - No errors when determining @INC for ::: ok 93 - expected number of elements in @INC for ::: ok 94 - expected elements in @INC for ::: ok 95 - No errors when determining @INC for two things and : ok 96 - expected number of elements in @INC for two things and : ok 97 - expected elements in @INC for two things and : ok 98 - No errors when determining @INC for : and two things ok 99 - expected number of elements in @INC for : and two things ok 100 - expected elements in @INC for : and two things ok 101 - No errors when determining @INC for : two things : ok 102 - expected number of elements in @INC for : two things : ok 103 - expected elements in @INC for : two things : ok 104 - No errors when determining @INC for three things ok 105 - expected number of elements in @INC for three things ok 106 - expected elements in @INC for three things Failed 5/106 subtests # Looks like you failed 5 tests of 132. ../ext/Devel-Peek/t/Peek.t .. ok 1 - assignment of immediate constant (string) ok 2 - assignment of immediate constant (string) (unchanged by dump) ok 3 - immediate constant (string) ok 4 - immediate constant (string) (unchanged by dump) ok 5 - assignment of immediate constant (integer) ok 6 - assignment of immediate constant (integer) (unchanged by dump) ok 7 - immediate constant (integer) ok 8 - immediate constant (integer) (unchanged by dump) ok 9 - assignment of immediate constant (integer) ok 10 - assignment of immediate constant (integer) (unchanged by dump) ok 11 - result of addition ok 12 - result of addition (unchanged by dump) ok 13 - floating point value ok 14 - floating point value (unchanged by dump) ok 15 - integer constant ok 16 - integer constant (unchanged by dump) ok 17 - undef ok 18 - undef (unchanged by dump) ok 19 - reference to scalar ok 20 - reference to scalar (unchanged by dump) ok 21 - reference to array ok 22 - reference to array (unchanged by dump) ok 23 - reference to hash ok 24 - reference to hash (unchanged by dump) ok 25 - reference to anon sub with empty prototype ok 26 - reference to anon sub with empty prototype (unchanged by dump) ok 27 - reference to named subroutine without prototype ok 28 - reference to named subroutine without prototype (unchanged by dump) ok 29 - reference to regexp ok 30 - reference to regexp (unchanged by dump) ok 31 - reference to blessed hash ok 32 - reference to blessed hash (unchanged by dump) ok 33 - typeglob ok 34 - typeglob (unchanged by dump) ok 35 - string with Unicode ok 36 - string with Unicode (unchanged by dump) ok 37 - reference to hash containing Unicode ok 38 - reference to hash containing Unicode (unchanged by dump) ok 39 - scalar with pos magic ok 40 - scalar with pos magic (unchanged by dump) ok 41 - tainted value in %ENV ok 42 - tainted value in %ENV (unchanged by dump) ok 43 - blessed reference ok 44 - blessed reference (unchanged by dump) ok 45 - constant subroutine ok 46 - constant subroutine (unchanged by dump) ok 47 - isUV should show on PVMG ok 48 - isUV should show on PVMG (unchanged by dump) ok 49 - IO ok 50 - IO (unchanged by dump) ok 51 - FORMAT ok 52 - FORMAT (unchanged by dump) ok 53 - blessing to a class with embedded NUL characters ok 54 - blessing to a class with embedded NUL characters (unchanged by dump) not ok 55 - ENAME on a stash # line 773 ok 56 - ENAME on a stash (unchanged by dump) not ok 57 - ENAMEs on a stash # line 796 ok 58 - ENAMEs on a stash (unchanged by dump) not ok 59 - ENAMEs on a stash with no NAME # line 820 ok 60 - ENAMEs on a stash with no NAME (unchanged by dump) ok 61 - small hash ok 62 - small hash (unchanged by dump) not ok 63 - small hash after keys # line 871 ok 64 - small hash after keys (unchanged by dump) not ok 65 - small hash after keys and scalar # line 901 ok 66 - small hash after keys and scalar (unchanged by dump) ok 67 - Dump @array ok 68 - Dump @array (unchanged by dump) ok 69 - Dump @array\,1 ok 70 - Dump @array\,1 (unchanged by dump) ok 71 - Dump %hash ok 72 - Dump %hash (unchanged by dump) ok 73 - rvalue substr ok 74 - rvalue substr (unchanged by dump) ok 75 - Dump; ok 76 - Dump() ok 77 # skip Not built with usemymalloc ok 78 # skip Not built with usemymalloc ok 79 - regular string constant ok 80 - regular string constant (unchanged by dump) ok 81 - string constant now an FBM ok 82 - string constant now an FBM (unchanged by dump) ok 83 - Not allowed to study an FBM ok 84 - string constant still an FBM ok 85 - string constant still an FBM (unchanged by dump) ok 86 - regular string constant ok 87 - regular string constant (unchanged by dump) ok 88 - Our studies were successful ok 89 - string constant quite unaffected ok 90 - string constant quite unaffected (unchanged by dump) ok 91 - correct index ok 92 - string constant now FBMed ok 93 - string constant now FBMed (unchanged by dump) ok 94 - Our studies were successful ok 95 - string constant still FBMed ok 96 - string constant still FBMed (unchanged by dump) ok 97 - second string also unaffected ok 98 - second string also unaffected (unchanged by dump) ok 99 - no crash with DeadCode ok 100 - UTF-8 in a regular expression ok 101 - UTF-8 in a regular expression (unchanged by dump) ok 102 - SvREFCNT on non-scalar ok 103 - requires prototype ok 104 - GVGV's are correctly escaped for latin1 :: latin1 ok 105 - GVGV's are correctly escaped for UTF8 :: UTF8 ok 106 - GVGV's are correctly escaped for latin1 :: UTF8 ok 107 - GVGV's are correctly escaped for UTF8 :: latin1 ok 108 - GVGV's are correctly escaped for UTF8 :: latin 1 :: UTF8 ok 109 - NAME is correctly escaped for UTF8 globs ok 110 - GvSTASH is correctly escaped for UTF8 globs ok 111 - EGV is correctly escaped for UTF8 globs ok 112 - NAME is correctly escaped for UTF8 globs with latin1 stashes ok 113 - GvSTASH is correctly escaped for UTF8 globs with latin1 stashes ok 114 - EGV is correctly escaped for UTF8 globs with latin1 stashes ok 115 - STASH for blessed hashrefs is correct ok 116 - PROTOTYPE is escaped correctly ok 117 - PROTOTYPE works on lexical subs ok 118 - OUTSIDE works ok 119 - OUTSIDE + UTF8 works ok 120 - NAME is correct escaped for UTF8 stashes ok 121 - ENAME is correct escaped for UTF8 stashes ok 122 - NAME is correct escaped for latin 1 stashes ok 123 - ENAME is correct escaped for latin 1 stashes ok 124 - NAME is correct escaped for latin1 + UTF8 stashes ok 125 - ENAME is correct escaped for latin1 + UTF8 stashes ok 126 - NAME is correct escaped for UTF8 + latin1 stashes ok 127 - ENAME is correct escaped for UTF8 + latin1 stashes ok 128 - DumpProg() + package declaration ok 129 - DumpProg() + UTF8 package declaration ok 130 - use utf8; sub ヌ::ネ {1}; ヌ::ネ;use Devel::Pe... ok 131 - use utf8; ヌ: { last ヌ }use Devel::Peek "DumpPr... ok 132 - DumpProg() has no 'Attempt to free X prematurely' warning 1..132 Dubious\, test returned 5 (wstat 1280\, 0x500) Failed 5/132 subtests (less 2 skipped subtests: 125 okay)
Test Summary Report
op/each.t (Wstat: 0 Tests: 59 Failed: 1) Failed test: 56 run/runenv.t (Wstat: 0 Tests: 106 Failed: 5) Failed tests: 46\, 48\, 50\, 52\, 54 ../ext/Devel-Peek/t/Peek.t (Wstat: 1280 Tests: 132 Failed: 5) Failed tests: 55\, 57\, 59\, 63\, 65 Non-zero exit status: 5 Files=3\, Tests=297\, 0 wallclock secs ( 0.05 usr 0.00 sys + 0.16 cusr 0.05 csys = 0.26 CPU) Result: FAIL
p5pRT
commented
6 years ago Thats right! Thank you! It should be PERL_HASH_RANDOMIZE_KEYS of course :)
On Wed\, Aug 22\, 2018 at 5:23 PM\, James E Keenan via RT \< perlbug-followup@perl.org> wrote:
On Mon\, 20 Aug 2018 23:42:55 GMT\, 11000@gmx.de wrote:
Hi\, I would like to provide a little patch that fixes a bug introduced somewhere around 5.26. The bug makes perl stall immediately because all hash keys will go into the same bucket\, because the hash-split is broken. Fortunately\, this bug is only triggered if you use the compile-time switch -Accflags=-DPERL_PERTURB_KEYS_DISABLED. I reckon i am the first person to ever use this switch since around 5.26! :) The patch i provided is very simple and fixes the bug by moving the in_collision flag assignment out of the #ifdef PERL_RANDOMIZE_KEYS. I think this assignment must always be included in the code\, no matter if the randomize keys feature is enabled or not. It is important that this assignment is computed always\, because the new hash split algorithm introduced somewhere around 5.26 will only split if the hash is 1) full (up to a certain level) and 2) a collision happens. If however the in_collision flag is never set\, the hash will never split. So we have to move its assignment out of that #ifdef. Regards\,Thomas Schweiger
Shouldn't the 2nd switch you mention be:
PERL_HASH_RANDOMIZE_KEYS
-- James E Keenan (jkeenan@cpan.org)
p5pRT
commented
6 years ago Thank you for looking into this! Looks good. This is exactly what i observed.
Regarding the compile time warning: I also observed the warning of in_collision being uninitialized (without the patch)\, which makes sense. The warning disappears after the patch is applied.
The other warnings about usused variables though - i don't know where they come from. They must be related to the use of the switch somehow but they are probably not related to this issue.
Regarding the failed tests: I can also confirm some tests are failing when compiled with the switch but i did not care much about it because in hv.h\, it is stated that some tests may fail. So this is probably fine.
------- exerpt from hv.h ------ /* These control hash traversal randomization and the environment variable PERL_PERTURB_KEYS. * Currently disabling this functionality will break a few tests\, but should otherwise work fine. * See perlrun for more details. */
#if defined(PERL_PERTURB_KEYS_DISABLED)
On Wed\, Aug 22\, 2018 at 11:07 PM\, James E Keenan via RT \< perlbug-followup@perl.org> wrote:
On Mon\, 20 Aug 2018 23:42:55 GMT\, 11000@gmx.de wrote:
Hi\, I would like to provide a little patch that fixes a bug introduced somewhere around 5.26. The bug makes perl stall immediately because all hash keys will go into the same bucket\, because the hash-split is broken. Fortunately\, this bug is only triggered if you use the compile-time switch -Accflags=-DPERL_PERTURB_KEYS_DISABLED. I reckon i am the first person to ever use this switch since around 5.26! :) The patch i provided is very simple and fixes the bug by moving the in_collision flag assignment out of the #ifdef PERL_RANDOMIZE_KEYS. I think this assignment must always be included in the code\, no matter if the randomize keys feature is enabled or not. It is important that this assignment is computed always\, because the new hash split algorithm introduced somewhere around 5.26 will only split if the hash is 1) full (up to a certain level) and 2) a collision happens. If however the in_collision flag is never set\, the hash will never split. So we have to move its assignment out of that #ifdef. Regards\,Thomas Schweiger
1. I tried to build Perl 5 blead by adding your switch to my regular configuration syntax:
##### sh ./Configure -des -Dusedevel -Accflags=-DPERL_PERTURB_KEYS_DISABLED #####
I get this previously unseen compile-time warning:
##### ... cc -c -DPERL_CORE -DPERL_PERTURB_KEYS_DISABLED -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c89 -O2 -Wall -Werror=declaration-after-statement -Werror=pointer-arith -Wextra -Wc++-compat -Wwrite-strings hv.c hv.c: In function ‘Perl_hv_rand_set’: hv.c:2344:23: warning: unused variable ‘iter’ [-Wunused-variable] struct xpvhv_aux *iter; ^ hv.c:2343:28: warning: unused parameter ‘hv’ [-Wunused-parameter] Perl_hv_rand_set(pTHX_ HV *hv\, U32 new_xhv_rand) { ^ hv.c:2343:36: warning: unused parameter ‘new_xhv_rand’ [-Wunused-parameter] Perl_hv_rand_set(pTHX_ HV *hv\, U32 new_xhv_rand) { ^ hv.c: In function ‘Perl_hv_common’: hv.c:886:8: warning: ‘in_collision’ may be used uninitialized in this function [-Wmaybe-uninitialized] if ( in_collision && DO_HSPLIT(xhv) ) { ^ ... #####
Then 'make' appeared to grind to a halt at:
##### ... Processing extracted/DJoinGroup.txt Processing extracted/DJoinType.txt Processing Jamo.txt Processing UnicodeData.txt #####
I did not attempt to proceed to 'make test'.
2. I created a branch and applied your patch to the branch:
smoke-me/jkeenan/133459-perturb
I configured and built perl as I normally do. 'make test_harness' PASS.
3. In the patch branched\, I added your switch to my regular configuration command. I got the same compile-time warning as mentioned above\, but 'make' did not appear to hang. However\, I got failures in three tests:
##### run/runenv.t (Wstat: 0 Tests: 106 Failed: 5) Failed tests: 46\, 48\, 50\, 52\, 54 op/each.t (Wstat: 0 Tests: 59 Failed: 1) Failed test: 56 ../ext/Devel-Peek/t/Peek.t (Wstat: 1280 Tests: 132 Failed: 5) Failed tests: 55\, 57\, 59\, 63\, 65 Non-zero exit status: 5 #####
Output attached.
-- James E Keenan (jkeenan@cpan.org)
# Failed test 56 - each() after insert produces warnings at op/each.t line 276 op/each.t ................... 1..59 ok 1 - keys ok 2 - values ok 3 - each count ok 4 - added a key ok 5 - resize ok 6 - same size ok 7 - still same size ok 8 - size 8 ok 9 - test scalar each ok 10 - test scalar keys resets iterator ok 11 - test iterator of each is being maintained ok 12 - test values keys resets iterator ok 13 - assign to keys does not shrink hash bucket array ok 14 - assignment to keys of a number not large enough does not change size ok 15 - keys (%hash) ok 16 ok 17 # DESTROY called ok 18 ok 19 - Check length of "\x{12345}" ok 20 - Check length of "\x{10fffd}" ok 21 - Check length of "\x{1234}" ok 22 - Check length of "\x{123}" ok 23 - Check length of "\022" ok 24 - copied unicode hash keys correctly? ok 25 - utf8 key in bytes hash ok 26 - bytes key in utf8 hash # non-utf8 ok 27 - if we got here change 8056 worked # utf8 ok 28 - change 8056 is thanks to Inaba Hiroto ok 29 - check encoding on EBCDIC ok 30 - check length ok 31 - check value ok 32 ok 33 - four different Unicode keys ok 34 - U+00FF -> 1 ok 35 - U+00C3 U+00BF -> 2 ok 36 - U+0100 -> 3 ok 37 - U+00C4 U+0080 -> 4 ok 38 - each demands argument ok 39 - keys demands argument ok 40 - values demands argument ok 41 - if(%foo) was true - my ok 42 - if(%foo) didnt mess with each (key) - my ok 43 - if(%foo) didnt mess with each (value) - my ok 44 - Got the expected number of keys - my ok 45 - Got the count of keys from %foo in scalar assignment context - my ok 46 - Got expected number of elements in list context - my ok 47 - if(%foo) was true - our ok 48 - if(%foo) didnt mess with each (key) - our ok 49 - if(%foo) didnt mess with each (value) - our ok 50 - Got the expected number of keys - our ok 51 - Got the count of keys from %foo in scalar assignment context - our ok 52 - Got expected number of elements in list context - our ok 53 - single key not yet freed ok 54 - single key now freed ok 55 - each on apparently empty hash does not leave RITER set not ok 56 - each() after insert produces warnings ok 57 - no warnings 'internal' silences each() after insert warnings ok 58 - each in list assignment ok 59 - values in list assignment Failed 1/59 subtests # Failed test 46 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr at run/runenv.t line 232 # Failed test 48 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr at run/runenv.t line 239 # Failed test 50 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stderr at run/runenv.t line 244 # Failed test 52 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stderr at run/runenv.t line 249 # Failed test 54 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stderr at run/runenv.t line 254 # Failed test 'ENAME on a stash' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAME = "RWOM" # ENAME = "RWOM" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAME = "RWOM" # ENAME = "RWOM" # \Z)' # # Failed test 'ENAMEs on a stash' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 3 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAME = "RWOM" # NAMECOUNT = 2 # ENAME = "RWOM"\, "KLANK" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 3 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAME = "RWOM" # NAMECOUNT = 2 # ENAME = "RWOM"\, "KLANK" # \Z)' # # Failed test 'ENAMEs on a stash with no NAME' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 3 # FLAGS = (OOK\,SHAREKEYS\,OVERLOAD) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAMECOUNT = -3 # ENAME = "RWOM"\, "KLANK" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 3 # FLAGS = \(OOK\,SHAREKEYS\,OVERLOAD\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAMECOUNT = -3 # ENAME = "RWOM"\, "KLANK" # \Z)' # # Failed test 'small hash after keys' # at t/Peek.t line 98. # # 'SV = IV(0x2e19c20) at 0x2e19c30 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25de050 # SV = PVHV(0x2d92a90) at 0x25de050 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2e23b70 (0:6\, 1:2) # hash quality = 125.0% # KEYS = 2 # FILL = 2 # MAX = 7 # RITER = -1 # EITER = 0x0 # # Elt "Beer" HASH = 0xaf2bb028 # SV = PV(0x2dfc2e0) at 0x2e154e0 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7670 "Foamy"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # Elt "Perl" HASH = 0xe22b31e6 # SV = PV(0x2dfc2d0) at 0x2e19f48 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7820 "Rules"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ \(0:[67]\,.*\) # hash quality = [0-9.]+% # KEYS = 2 # FILL = [12] # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # (?: Elt "(?:Perl|Beer)" HASH = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(POK\,(?:IsCOW\,)?pPOK\) # PV = 0x[[:xdigit:]]+ "(?:Rules|Foamy)"\\0 # CUR = \d+ # LEN = \d+ # COW_REFCNT = 1 # ){2}\Z)' # # Failed test 'small hash after keys and scalar' # at t/Peek.t line 98. # # 'SV = IV(0x2e19c20) at 0x2e19c30 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25de050 # SV = PVHV(0x2d92a90) at 0x25de050 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2e23b70 (0:6\, 1:2) # hash quality = 125.0% # KEYS = 2 # FILL = 2 # MAX = 7 # RITER = -1 # EITER = 0x0 # # Elt "Beer" HASH = 0xaf2bb028 # SV = PV(0x2dfc2e0) at 0x2e154e0 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7670 "Foamy"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # Elt "Perl" HASH = 0xe22b31e6 # SV = PV(0x2dfc2d0) at 0x2e19f48 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7820 "Rules"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ \(0:[67]\,.*\) # hash quality = [0-9.]+% # KEYS = 2 # FILL = ([12]) # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # (?: Elt "(?:Perl|Beer)" HASH = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(POK\,(?:IsCOW\,)?pPOK\) # PV = 0x[[:xdigit:]]+ "(?:Rules|Foamy)"\\0 # CUR = \d+ # LEN = \d+ # COW_REFCNT = 1 # ){2}\Z)' # run/runenv.t ................ 1..106 ok 1 - PERL5OPT => -w\,-e\,print $::x stdout ok 2 - PERL5OPT => -w\,-e\,print $::x stderr ok 3 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $::x stdout ok 4 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $::x stderr ok 5 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $x stdout ok 6 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $x stderr ok 7 - PERL5OPT => -Mstrict -w\,-I../lib\,-e\,print $x stdout ok 8 - PERL5OPT => -Mstrict -w\,-I../lib\,-e\,print $x stderr ok 9 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stdout ok 10 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stderr ok 11 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stdout ok 12 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stderr ok 13 - PERL5OPT => -MExporter\,-I../lib\,-e0 stdout ok 14 - PERL5OPT => -MExporter\,-I../lib\,-e0 stderr ok 15 - PERL5OPT => -MExporter -MExporter\,-I../lib\,-e0 stdout ok 16 - PERL5OPT => -MExporter -MExporter\,-I../lib\,-e0 stderr ok 17 - PERL5OPT => -Mstrict -Mwarnings\,-I../lib\,-e\,print "ok" if $INC{" strict.pm"} and $INC{"warnings.pm"} stdout ok 18 - PERL5OPT => -Mstrict -Mwarnings\,-I../lib\,-e\,print "ok" if $INC{" strict.pm"} and $INC{"warnings.pm"} stderr ok 19 - PERL5OPT => -I. -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stdout ok 20 - PERL5OPT => -I. -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stderr ok 21 - PERL5OPT => -I./ -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stdout ok 22 - PERL5OPT => -I./ -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stderr ok 23 - PERL5OPT => -w -w\,-e\,print $ENV{PERL5OPT} stdout ok 24 - PERL5OPT => -w -w\,-e\,print $ENV{PERL5OPT} stderr ok 25 - PERL5OPT => -t\,-e\,print ${^TAINT} stdout ok 26 - PERL5OPT => -t\,-e\,print ${^TAINT} stderr ok 27 - PERL5OPT => -W\,-I../lib\,-e\,local $^W = 0; no warnings; print $x stdout ok 28 - PERL5OPT => -W\,-I../lib\,-e\,local $^W = 0; no warnings; print $x stderr ok 29 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stdout ok 30 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stderr ok 31 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stdout ok 32 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stderr ok 33 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stdout ok 34 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stderr ok 35 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stdout ok 36 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stderr ok 37 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "foo" } @INC stdout ok 38 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "foo" } @INC stderr ok 39 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "bar" } @INC stdout ok 40 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "bar" } @INC stderr ok 41 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 42 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 43 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 44 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 45 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout not ok 46 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 47 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout not ok 48 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 49 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stdout not ok 50 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stderr ok 51 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stdout not ok 52 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stderr ok 53 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stdout not ok 54 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stderr ok 55 - PERL_HASH_SEED => 12345678\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 56 - PERL_HASH_SEED => 12345678\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 57 - PERL_HASH_SEED => 12\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 58 - PERL_HASH_SEED => 12\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 59 - PERL_HASH_SEED => 123456789\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 60 - PERL_HASH_SEED => 123456789\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 61 - No errors when determining @INC ok 62 - . is not in @INC ok 63 - No errors when determining unsafe @INC ok 64 - . last in unsafe @INC ok 65 - No errors when determining @INC for nothing ok 66 - expected number of elements in @INC for nothing ok 67 - expected elements in @INC for nothing ok 68 - No errors when determining @INC for something ok 69 - expected number of elements in @INC for something ok 70 - expected elements in @INC for something ok 71 - No errors when determining @INC for two things ok 72 - expected number of elements in @INC for two things ok 73 - expected elements in @INC for two things ok 74 - No errors when determining @INC for two things\, :: ok 75 - expected number of elements in @INC for two things\, :: ok 76 - expected elements in @INC for two things\, :: ok 77 - No errors when determining @INC for : at start ok 78 - expected number of elements in @INC for : at start ok 79 - expected elements in @INC for : at start ok 80 - No errors when determining @INC for : at end ok 81 - expected number of elements in @INC for : at end ok 82 - expected elements in @INC for : at end ok 83 - No errors when determining @INC for :: sandwich :: ok 84 - expected number of elements in @INC for :: sandwich :: ok 85 - expected elements in @INC for :: sandwich :: ok 86 - No errors when determining @INC for : ok 87 - expected number of elements in @INC for : ok 88 - expected elements in @INC for : ok 89 - No errors when determining @INC for :: ok 90 - expected number of elements in @INC for :: ok 91 - expected elements in @INC for :: ok 92 - No errors when determining @INC for ::: ok 93 - expected number of elements in @INC for ::: ok 94 - expected elements in @INC for ::: ok 95 - No errors when determining @INC for two things and : ok 96 - expected number of elements in @INC for two things and : ok 97 - expected elements in @INC for two things and : ok 98 - No errors when determining @INC for : and two things ok 99 - expected number of elements in @INC for : and two things ok 100 - expected elements in @INC for : and two things ok 101 - No errors when determining @INC for : two things : ok 102 - expected number of elements in @INC for : two things : ok 103 - expected elements in @INC for : two things : ok 104 - No errors when determining @INC for three things ok 105 - expected number of elements in @INC for three things ok 106 - expected elements in @INC for three things Failed 5/106 subtests # Looks like you failed 5 tests of 132. ../ext/Devel-Peek/t/Peek.t .. ok 1 - assignment of immediate constant (string) ok 2 - assignment of immediate constant (string) (unchanged by dump) ok 3 - immediate constant (string) ok 4 - immediate constant (string) (unchanged by dump) ok 5 - assignment of immediate constant (integer) ok 6 - assignment of immediate constant (integer) (unchanged by dump) ok 7 - immediate constant (integer) ok 8 - immediate constant (integer) (unchanged by dump) ok 9 - assignment of immediate constant (integer) ok 10 - assignment of immediate constant (integer) (unchanged by dump) ok 11 - result of addition ok 12 - result of addition (unchanged by dump) ok 13 - floating point value ok 14 - floating point value (unchanged by dump) ok 15 - integer constant ok 16 - integer constant (unchanged by dump) ok 17 - undef ok 18 - undef (unchanged by dump) ok 19 - reference to scalar ok 20 - reference to scalar (unchanged by dump) ok 21 - reference to array ok 22 - reference to array (unchanged by dump) ok 23 - reference to hash ok 24 - reference to hash (unchanged by dump) ok 25 - reference to anon sub with empty prototype ok 26 - reference to anon sub with empty prototype (unchanged by dump) ok 27 - reference to named subroutine without prototype ok 28 - reference to named subroutine without prototype (unchanged by dump) ok 29 - reference to regexp ok 30 - reference to regexp (unchanged by dump) ok 31 - reference to blessed hash ok 32 - reference to blessed hash (unchanged by dump) ok 33 - typeglob ok 34 - typeglob (unchanged by dump) ok 35 - string with Unicode ok 36 - string with Unicode (unchanged by dump) ok 37 - reference to hash containing Unicode ok 38 - reference to hash containing Unicode (unchanged by dump) ok 39 - scalar with pos magic ok 40 - scalar with pos magic (unchanged by dump) ok 41 - tainted value in %ENV ok 42 - tainted value in %ENV (unchanged by dump) ok 43 - blessed reference ok 44 - blessed reference (unchanged by dump) ok 45 - constant subroutine ok 46 - constant subroutine (unchanged by dump) ok 47 - isUV should show on PVMG ok 48 - isUV should show on PVMG (unchanged by dump) ok 49 - IO ok 50 - IO (unchanged by dump) ok 51 - FORMAT ok 52 - FORMAT (unchanged by dump) ok 53 - blessing to a class with embedded NUL characters ok 54 - blessing to a class with embedded NUL characters (unchanged by dump) not ok 55 - ENAME on a stash # line 773 ok 56 - ENAME on a stash (unchanged by dump) not ok 57 - ENAMEs on a stash # line 796 ok 58 - ENAMEs on a stash (unchanged by dump) not ok 59 - ENAMEs on a stash with no NAME # line 820 ok 60 - ENAMEs on a stash with no NAME (unchanged by dump) ok 61 - small hash ok 62 - small hash (unchanged by dump) not ok 63 - small hash after keys # line 871 ok 64 - small hash after keys (unchanged by dump) not ok 65 - small hash after keys and scalar # line 901 ok 66 - small hash after keys and scalar (unchanged by dump) ok 67 - Dump @array ok 68 - Dump @array (unchanged by dump) ok 69 - Dump @array\,1 ok 70 - Dump @array\,1 (unchanged by dump) ok 71 - Dump %hash ok 72 - Dump %hash (unchanged by dump) ok 73 - rvalue substr ok 74 - rvalue substr (unchanged by dump) ok 75 - Dump; ok 76 - Dump() ok 77 # skip Not built with usemymalloc ok 78 # skip Not built with usemymalloc ok 79 - regular string constant ok 80 - regular string constant (unchanged by dump) ok 81 - string constant now an FBM ok 82 - string constant now an FBM (unchanged by dump) ok 83 - Not allowed to study an FBM ok 84 - string constant still an FBM ok 85 - string constant still an FBM (unchanged by dump) ok 86 - regular string constant ok 87 - regular string constant (unchanged by dump) ok 88 - Our studies were successful ok 89 - string constant quite unaffected ok 90 - string constant quite unaffected (unchanged by dump) ok 91 - correct index ok 92 - string constant now FBMed ok 93 - string constant now FBMed (unchanged by dump) ok 94 - Our studies were successful ok 95 - string constant still FBMed ok 96 - string constant still FBMed (unchanged by dump) ok 97 - second string also unaffected ok 98 - second string also unaffected (unchanged by dump) ok 99 - no crash with DeadCode ok 100 - UTF-8 in a regular expression ok 101 - UTF-8 in a regular expression (unchanged by dump) ok 102 - SvREFCNT on non-scalar ok 103 - requires prototype ok 104 - GVGV's are correctly escaped for latin1 :: latin1 ok 105 - GVGV's are correctly escaped for UTF8 :: UTF8 ok 106 - GVGV's are correctly escaped for latin1 :: UTF8 ok 107 - GVGV's are correctly escaped for UTF8 :: latin1 ok 108 - GVGV's are correctly escaped for UTF8 :: latin 1 :: UTF8 ok 109 - NAME is correctly escaped for UTF8 globs ok 110 - GvSTASH is correctly escaped for UTF8 globs ok 111 - EGV is correctly escaped for UTF8 globs ok 112 - NAME is correctly escaped for UTF8 globs with latin1 stashes ok 113 - GvSTASH is correctly escaped for UTF8 globs with latin1 stashes ok 114 - EGV is correctly escaped for UTF8 globs with latin1 stashes ok 115 - STASH for blessed hashrefs is correct ok 116 - PROTOTYPE is escaped correctly ok 117 - PROTOTYPE works on lexical subs ok 118 - OUTSIDE works ok 119 - OUTSIDE + UTF8 works ok 120 - NAME is correct escaped for UTF8 stashes ok 121 - ENAME is correct escaped for UTF8 stashes ok 122 - NAME is correct escaped for latin 1 stashes ok 123 - ENAME is correct escaped for latin 1 stashes ok 124 - NAME is correct escaped for latin1 + UTF8 stashes ok 125 - ENAME is correct escaped for latin1 + UTF8 stashes ok 126 - NAME is correct escaped for UTF8 + latin1 stashes ok 127 - ENAME is correct escaped for UTF8 + latin1 stashes ok 128 - DumpProg() + package declaration ok 129 - DumpProg() + UTF8 package declaration ok 130 - use utf8; sub ヌ::ネ {1}; ヌ::ネ;use Devel::Pe... ok 131 - use utf8; ヌ: { last ヌ }use Devel::Peek "DumpPr... ok 132 - DumpProg() has no 'Attempt to free X prematurely' warning 1..132 Dubious\, test returned 5 (wstat 1280\, 0x500) Failed 5/132 subtests (less 2 skipped subtests: 125 okay)
Test Summary Report ------------------- op/each.t (Wstat: 0 Tests: 59 Failed: 1) Failed test: 56 run/runenv.t (Wstat: 0 Tests: 106 Failed: 5) Failed tests: 46\, 48\, 50\, 52\, 54 ../ext/Devel-Peek/t/Peek.t (Wstat: 1280 Tests: 132 Failed: 5) Failed tests: 55\, 57\, 59\, 63\, 65 Non-zero exit status: 5 Files=3\, Tests=297\, 0 wallclock secs ( 0.05 usr 0.00 sys + 0.16 cusr 0.05 csys = 0.26 CPU) Result: FAIL
p5pRT
commented
6 years ago Is it possible you could explain why you are building like this and which hash function you are using?
Yves
On Fri\, 24 Aug 2018\, 17:08 fnordpole@gmail.com\, \fnordpole@​gmail\.com wrote:
Thank you for looking into this! Looks good. This is exactly what i observed.
Regarding the compile time warning: I also observed the warning of in_collision being uninitialized (without the patch)\, which makes sense. The warning disappears after the patch is applied.
The other warnings about usused variables though - i don't know where they come from. They must be related to the use of the switch somehow but they are probably not related to this issue.
Regarding the failed tests: I can also confirm some tests are failing when compiled with the switch but i did not care much about it because in hv.h\, it is stated that some tests may fail. So this is probably fine.
------- exerpt from hv.h ------ /* These control hash traversal randomization and the environment variable PERL_PERTURB_KEYS. * Currently disabling this functionality will break a few tests\, but should otherwise work fine. * See perlrun for more details. */
#if defined(PERL_PERTURB_KEYS_DISABLED) -------------------------------
On Wed\, Aug 22\, 2018 at 11:07 PM\, James E Keenan via RT \< perlbug-followup@perl.org> wrote:
On Mon\, 20 Aug 2018 23:42:55 GMT\, 11000@gmx.de wrote:
Hi\, I would like to provide a little patch that fixes a bug introduced somewhere around 5.26. The bug makes perl stall immediately because all hash keys will go into the same bucket\, because the hash-split is broken. Fortunately\, this bug is only triggered if you use the compile-time switch -Accflags=-DPERL_PERTURB_KEYS_DISABLED. I reckon i am the first person to ever use this switch since around 5.26! :) The patch i provided is very simple and fixes the bug by moving the in_collision flag assignment out of the #ifdef PERL_RANDOMIZE_KEYS. I think this assignment must always be included in the code\, no matter if the randomize keys feature is enabled or not. It is important that this assignment is computed always\, because the new hash split algorithm introduced somewhere around 5.26 will only split if the hash is 1) full (up to a certain level) and 2) a collision happens. If however the in_collision flag is never set\, the hash will never split. So we have to move its assignment out of that #ifdef. Regards\,Thomas Schweiger
1. I tried to build Perl 5 blead by adding your switch to my regular configuration syntax:
##### sh ./Configure -des -Dusedevel -Accflags=-DPERL_PERTURB_KEYS_DISABLED #####
I get this previously unseen compile-time warning:
##### ... cc -c -DPERL_CORE -DPERL_PERTURB_KEYS_DISABLED -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c89 -O2 -Wall -Werror=declaration-after-statement -Werror=pointer-arith -Wextra -Wc++-compat -Wwrite-strings hv.c hv.c: In function ‘Perl_hv_rand_set’: hv.c:2344:23: warning: unused variable ‘iter’ [-Wunused-variable] struct xpvhv_aux *iter; ^ hv.c:2343:28: warning: unused parameter ‘hv’ [-Wunused-parameter] Perl_hv_rand_set(pTHX_ HV *hv\, U32 new_xhv_rand) { ^ hv.c:2343:36: warning: unused parameter ‘new_xhv_rand’ [-Wunused-parameter] Perl_hv_rand_set(pTHX_ HV *hv\, U32 new_xhv_rand) { ^ hv.c: In function ‘Perl_hv_common’: hv.c:886:8: warning: ‘in_collision’ may be used uninitialized in this function [-Wmaybe-uninitialized] if ( in_collision && DO_HSPLIT(xhv) ) { ^ ... #####
Then 'make' appeared to grind to a halt at:
##### ... Processing extracted/DJoinGroup.txt Processing extracted/DJoinType.txt Processing Jamo.txt Processing UnicodeData.txt #####
I did not attempt to proceed to 'make test'.
2. I created a branch and applied your patch to the branch:
smoke-me/jkeenan/133459-perturb
I configured and built perl as I normally do. 'make test_harness' PASS.
3. In the patch branched\, I added your switch to my regular configuration command. I got the same compile-time warning as mentioned above\, but 'make' did not appear to hang. However\, I got failures in three tests:
##### run/runenv.t (Wstat: 0 Tests: 106 Failed: 5) Failed tests: 46\, 48\, 50\, 52\, 54 op/each.t (Wstat: 0 Tests: 59 Failed: 1) Failed test: 56 ../ext/Devel-Peek/t/Peek.t (Wstat: 1280 Tests: 132 Failed: 5) Failed tests: 55\, 57\, 59\, 63\, 65 Non-zero exit status: 5 #####
Output attached.
-- James E Keenan (jkeenan@cpan.org)
# Failed test 56 - each() after insert produces warnings at op/each.t line 276 op/each.t ................... 1..59 ok 1 - keys ok 2 - values ok 3 - each count ok 4 - added a key ok 5 - resize ok 6 - same size ok 7 - still same size ok 8 - size 8 ok 9 - test scalar each ok 10 - test scalar keys resets iterator ok 11 - test iterator of each is being maintained ok 12 - test values keys resets iterator ok 13 - assign to keys does not shrink hash bucket array ok 14 - assignment to keys of a number not large enough does not change size ok 15 - keys (%hash) ok 16 ok 17 # DESTROY called ok 18 ok 19 - Check length of "\x{12345}" ok 20 - Check length of "\x{10fffd}" ok 21 - Check length of "\x{1234}" ok 22 - Check length of "\x{123}" ok 23 - Check length of "\022" ok 24 - copied unicode hash keys correctly? ok 25 - utf8 key in bytes hash ok 26 - bytes key in utf8 hash # non-utf8 ok 27 - if we got here change 8056 worked # utf8 ok 28 - change 8056 is thanks to Inaba Hiroto ok 29 - check encoding on EBCDIC ok 30 - check length ok 31 - check value ok 32 ok 33 - four different Unicode keys ok 34 - U+00FF -> 1 ok 35 - U+00C3 U+00BF -> 2 ok 36 - U+0100 -> 3 ok 37 - U+00C4 U+0080 -> 4 ok 38 - each demands argument ok 39 - keys demands argument ok 40 - values demands argument ok 41 - if(%foo) was true - my ok 42 - if(%foo) didnt mess with each (key) - my ok 43 - if(%foo) didnt mess with each (value) - my ok 44 - Got the expected number of keys - my ok 45 - Got the count of keys from %foo in scalar assignment context - my ok 46 - Got expected number of elements in list context - my ok 47 - if(%foo) was true - our ok 48 - if(%foo) didnt mess with each (key) - our ok 49 - if(%foo) didnt mess with each (value) - our ok 50 - Got the expected number of keys - our ok 51 - Got the count of keys from %foo in scalar assignment context - our ok 52 - Got expected number of elements in list context - our ok 53 - single key not yet freed ok 54 - single key now freed ok 55 - each on apparently empty hash does not leave RITER set not ok 56 - each() after insert produces warnings ok 57 - no warnings 'internal' silences each() after insert warnings ok 58 - each in list assignment ok 59 - values in list assignment Failed 1/59 subtests # Failed test 46 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr at run/runenv.t line 232 # Failed test 48 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr at run/runenv.t line 239 # Failed test 50 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stderr at run/runenv.t line 244 # Failed test 52 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stderr at run/runenv.t line 249 # Failed test 54 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stderr at run/runenv.t line 254 # Failed test 'ENAME on a stash' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAME = "RWOM" # ENAME = "RWOM" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAME = "RWOM" # ENAME = "RWOM" # \Z)' # # Failed test 'ENAMEs on a stash' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 3 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAME = "RWOM" # NAMECOUNT = 2 # ENAME = "RWOM"\, "KLANK" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 3 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAME = "RWOM" # NAMECOUNT = 2 # ENAME = "RWOM"\, "KLANK" # \Z)' # # Failed test 'ENAMEs on a stash with no NAME' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 3 # FLAGS = (OOK\,SHAREKEYS\,OVERLOAD) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAMECOUNT = -3 # ENAME = "RWOM"\, "KLANK" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 3 # FLAGS = \(OOK\,SHAREKEYS\,OVERLOAD\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAMECOUNT = -3 # ENAME = "RWOM"\, "KLANK" # \Z)' # # Failed test 'small hash after keys' # at t/Peek.t line 98. # # 'SV = IV(0x2e19c20) at 0x2e19c30 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25de050 # SV = PVHV(0x2d92a90) at 0x25de050 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2e23b70 (0:6\, 1:2) # hash quality = 125.0% # KEYS = 2 # FILL = 2 # MAX = 7 # RITER = -1 # EITER = 0x0 # # Elt "Beer" HASH = 0xaf2bb028 # SV = PV(0x2dfc2e0) at 0x2e154e0 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7670 "Foamy"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # Elt "Perl" HASH = 0xe22b31e6 # SV = PV(0x2dfc2d0) at 0x2e19f48 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7820 "Rules"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ \(0:[67]\,.*\) # hash quality = [0-9.]+% # KEYS = 2 # FILL = [12] # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # (?: Elt "(?:Perl|Beer)" HASH = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(POK\,(?:IsCOW\,)?pPOK\) # PV = 0x[[:xdigit:]]+ "(?:Rules|Foamy)"\\0 # CUR = \d+ # LEN = \d+ # COW_REFCNT = 1 # ){2}\Z)' # # Failed test 'small hash after keys and scalar' # at t/Peek.t line 98. # # 'SV = IV(0x2e19c20) at 0x2e19c30 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25de050 # SV = PVHV(0x2d92a90) at 0x25de050 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2e23b70 (0:6\, 1:2) # hash quality = 125.0% # KEYS = 2 # FILL = 2 # MAX = 7 # RITER = -1 # EITER = 0x0 # # Elt "Beer" HASH = 0xaf2bb028 # SV = PV(0x2dfc2e0) at 0x2e154e0 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7670 "Foamy"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # Elt "Perl" HASH = 0xe22b31e6 # SV = PV(0x2dfc2d0) at 0x2e19f48 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7820 "Rules"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ \(0:[67]\,.*\) # hash quality = [0-9.]+% # KEYS = 2 # FILL = ([12]) # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # (?: Elt "(?:Perl|Beer)" HASH = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(POK\,(?:IsCOW\,)?pPOK\) # PV = 0x[[:xdigit:]]+ "(?:Rules|Foamy)"\\0 # CUR = \d+ # LEN = \d+ # COW_REFCNT = 1 # ){2}\Z)' # run/runenv.t ................ 1..106 ok 1 - PERL5OPT => -w\,-e\,print $::x stdout ok 2 - PERL5OPT => -w\,-e\,print $::x stderr ok 3 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $::x stdout ok 4 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $::x stderr ok 5 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $x stdout ok 6 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $x stderr ok 7 - PERL5OPT => -Mstrict -w\,-I../lib\,-e\,print $x stdout ok 8 - PERL5OPT => -Mstrict -w\,-I../lib\,-e\,print $x stderr ok 9 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stdout ok 10 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stderr ok 11 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stdout ok 12 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stderr ok 13 - PERL5OPT => -MExporter\,-I../lib\,-e0 stdout ok 14 - PERL5OPT => -MExporter\,-I../lib\,-e0 stderr ok 15 - PERL5OPT => -MExporter -MExporter\,-I../lib\,-e0 stdout ok 16 - PERL5OPT => -MExporter -MExporter\,-I../lib\,-e0 stderr ok 17 - PERL5OPT => -Mstrict -Mwarnings\,-I../lib\,-e\,print "ok" if $INC{" strict.pm"} and $INC{"warnings.pm"} stdout ok 18 - PERL5OPT => -Mstrict -Mwarnings\,-I../lib\,-e\,print "ok" if $INC{" strict.pm"} and $INC{"warnings.pm"} stderr ok 19 - PERL5OPT => -I. -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stdout ok 20 - PERL5OPT => -I. -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stderr ok 21 - PERL5OPT => -I./ -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stdout ok 22 - PERL5OPT => -I./ -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stderr ok 23 - PERL5OPT => -w -w\,-e\,print $ENV{PERL5OPT} stdout ok 24 - PERL5OPT => -w -w\,-e\,print $ENV{PERL5OPT} stderr ok 25 - PERL5OPT => -t\,-e\,print ${^TAINT} stdout ok 26 - PERL5OPT => -t\,-e\,print ${^TAINT} stderr ok 27 - PERL5OPT => -W\,-I../lib\,-e\,local $^W = 0; no warnings; print $x stdout ok 28 - PERL5OPT => -W\,-I../lib\,-e\,local $^W = 0; no warnings; print $x stderr ok 29 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stdout ok 30 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stderr ok 31 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stdout ok 32 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stderr ok 33 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stdout ok 34 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stderr ok 35 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stdout ok 36 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stderr ok 37 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "foo" } @INC stdout ok 38 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "foo" } @INC stderr ok 39 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "bar" } @INC stdout ok 40 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "bar" } @INC stderr ok 41 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 42 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 43 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 44 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 45 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout not ok 46 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 47 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout not ok 48 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 49 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stdout not ok 50 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stderr ok 51 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stdout not ok 52 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stderr ok 53 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stdout not ok 54 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stderr ok 55 - PERL_HASH_SEED => 12345678\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 56 - PERL_HASH_SEED => 12345678\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 57 - PERL_HASH_SEED => 12\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 58 - PERL_HASH_SEED => 12\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 59 - PERL_HASH_SEED => 123456789\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 60 - PERL_HASH_SEED => 123456789\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 61 - No errors when determining @INC ok 62 - . is not in @INC ok 63 - No errors when determining unsafe @INC ok 64 - . last in unsafe @INC ok 65 - No errors when determining @INC for nothing ok 66 - expected number of elements in @INC for nothing ok 67 - expected elements in @INC for nothing ok 68 - No errors when determining @INC for something ok 69 - expected number of elements in @INC for something ok 70 - expected elements in @INC for something ok 71 - No errors when determining @INC for two things ok 72 - expected number of elements in @INC for two things ok 73 - expected elements in @INC for two things ok 74 - No errors when determining @INC for two things\, :: ok 75 - expected number of elements in @INC for two things\, :: ok 76 - expected elements in @INC for two things\, :: ok 77 - No errors when determining @INC for : at start ok 78 - expected number of elements in @INC for : at start ok 79 - expected elements in @INC for : at start ok 80 - No errors when determining @INC for : at end ok 81 - expected number of elements in @INC for : at end ok 82 - expected elements in @INC for : at end ok 83 - No errors when determining @INC for :: sandwich :: ok 84 - expected number of elements in @INC for :: sandwich :: ok 85 - expected elements in @INC for :: sandwich :: ok 86 - No errors when determining @INC for : ok 87 - expected number of elements in @INC for : ok 88 - expected elements in @INC for : ok 89 - No errors when determining @INC for :: ok 90 - expected number of elements in @INC for :: ok 91 - expected elements in @INC for :: ok 92 - No errors when determining @INC for ::: ok 93 - expected number of elements in @INC for ::: ok 94 - expected elements in @INC for ::: ok 95 - No errors when determining @INC for two things and : ok 96 - expected number of elements in @INC for two things and : ok 97 - expected elements in @INC for two things and : ok 98 - No errors when determining @INC for : and two things ok 99 - expected number of elements in @INC for : and two things ok 100 - expected elements in @INC for : and two things ok 101 - No errors when determining @INC for : two things : ok 102 - expected number of elements in @INC for : two things : ok 103 - expected elements in @INC for : two things : ok 104 - No errors when determining @INC for three things ok 105 - expected number of elements in @INC for three things ok 106 - expected elements in @INC for three things Failed 5/106 subtests # Looks like you failed 5 tests of 132. ../ext/Devel-Peek/t/Peek.t .. ok 1 - assignment of immediate constant (string) ok 2 - assignment of immediate constant (string) (unchanged by dump) ok 3 - immediate constant (string) ok 4 - immediate constant (string) (unchanged by dump) ok 5 - assignment of immediate constant (integer) ok 6 - assignment of immediate constant (integer) (unchanged by dump) ok 7 - immediate constant (integer) ok 8 - immediate constant (integer) (unchanged by dump) ok 9 - assignment of immediate constant (integer) ok 10 - assignment of immediate constant (integer) (unchanged by dump) ok 11 - result of addition ok 12 - result of addition (unchanged by dump) ok 13 - floating point value ok 14 - floating point value (unchanged by dump) ok 15 - integer constant ok 16 - integer constant (unchanged by dump) ok 17 - undef ok 18 - undef (unchanged by dump) ok 19 - reference to scalar ok 20 - reference to scalar (unchanged by dump) ok 21 - reference to array ok 22 - reference to array (unchanged by dump) ok 23 - reference to hash ok 24 - reference to hash (unchanged by dump) ok 25 - reference to anon sub with empty prototype ok 26 - reference to anon sub with empty prototype (unchanged by dump) ok 27 - reference to named subroutine without prototype ok 28 - reference to named subroutine without prototype (unchanged by dump) ok 29 - reference to regexp ok 30 - reference to regexp (unchanged by dump) ok 31 - reference to blessed hash ok 32 - reference to blessed hash (unchanged by dump) ok 33 - typeglob ok 34 - typeglob (unchanged by dump) ok 35 - string with Unicode ok 36 - string with Unicode (unchanged by dump) ok 37 - reference to hash containing Unicode ok 38 - reference to hash containing Unicode (unchanged by dump) ok 39 - scalar with pos magic ok 40 - scalar with pos magic (unchanged by dump) ok 41 - tainted value in %ENV ok 42 - tainted value in %ENV (unchanged by dump) ok 43 - blessed reference ok 44 - blessed reference (unchanged by dump) ok 45 - constant subroutine ok 46 - constant subroutine (unchanged by dump) ok 47 - isUV should show on PVMG ok 48 - isUV should show on PVMG (unchanged by dump) ok 49 - IO ok 50 - IO (unchanged by dump) ok 51 - FORMAT ok 52 - FORMAT (unchanged by dump) ok 53 - blessing to a class with embedded NUL characters ok 54 - blessing to a class with embedded NUL characters (unchanged by dump) not ok 55 - ENAME on a stash # line 773 ok 56 - ENAME on a stash (unchanged by dump) not ok 57 - ENAMEs on a stash # line 796 ok 58 - ENAMEs on a stash (unchanged by dump) not ok 59 - ENAMEs on a stash with no NAME # line 820 ok 60 - ENAMEs on a stash with no NAME (unchanged by dump) ok 61 - small hash ok 62 - small hash (unchanged by dump) not ok 63 - small hash after keys # line 871 ok 64 - small hash after keys (unchanged by dump) not ok 65 - small hash after keys and scalar # line 901 ok 66 - small hash after keys and scalar (unchanged by dump) ok 67 - Dump @array ok 68 - Dump @array (unchanged by dump) ok 69 - Dump @array\,1 ok 70 - Dump @array\,1 (unchanged by dump) ok 71 - Dump %hash ok 72 - Dump %hash (unchanged by dump) ok 73 - rvalue substr ok 74 - rvalue substr (unchanged by dump) ok 75 - Dump; ok 76 - Dump() ok 77 # skip Not built with usemymalloc ok 78 # skip Not built with usemymalloc ok 79 - regular string constant ok 80 - regular string constant (unchanged by dump) ok 81 - string constant now an FBM ok 82 - string constant now an FBM (unchanged by dump) ok 83 - Not allowed to study an FBM ok 84 - string constant still an FBM ok 85 - string constant still an FBM (unchanged by dump) ok 86 - regular string constant ok 87 - regular string constant (unchanged by dump) ok 88 - Our studies were successful ok 89 - string constant quite unaffected ok 90 - string constant quite unaffected (unchanged by dump) ok 91 - correct index ok 92 - string constant now FBMed ok 93 - string constant now FBMed (unchanged by dump) ok 94 - Our studies were successful ok 95 - string constant still FBMed ok 96 - string constant still FBMed (unchanged by dump) ok 97 - second string also unaffected ok 98 - second string also unaffected (unchanged by dump) ok 99 - no crash with DeadCode ok 100 - UTF-8 in a regular expression ok 101 - UTF-8 in a regular expression (unchanged by dump) ok 102 - SvREFCNT on non-scalar ok 103 - requires prototype ok 104 - GVGV's are correctly escaped for latin1 :: latin1 ok 105 - GVGV's are correctly escaped for UTF8 :: UTF8 ok 106 - GVGV's are correctly escaped for latin1 :: UTF8 ok 107 - GVGV's are correctly escaped for UTF8 :: latin1 ok 108 - GVGV's are correctly escaped for UTF8 :: latin 1 :: UTF8 ok 109 - NAME is correctly escaped for UTF8 globs ok 110 - GvSTASH is correctly escaped for UTF8 globs ok 111 - EGV is correctly escaped for UTF8 globs ok 112 - NAME is correctly escaped for UTF8 globs with latin1 stashes ok 113 - GvSTASH is correctly escaped for UTF8 globs with latin1 stashes ok 114 - EGV is correctly escaped for UTF8 globs with latin1 stashes ok 115 - STASH for blessed hashrefs is correct ok 116 - PROTOTYPE is escaped correctly ok 117 - PROTOTYPE works on lexical subs ok 118 - OUTSIDE works ok 119 - OUTSIDE + UTF8 works ok 120 - NAME is correct escaped for UTF8 stashes ok 121 - ENAME is correct escaped for UTF8 stashes ok 122 - NAME is correct escaped for latin 1 stashes ok 123 - ENAME is correct escaped for latin 1 stashes ok 124 - NAME is correct escaped for latin1 + UTF8 stashes ok 125 - ENAME is correct escaped for latin1 + UTF8 stashes ok 126 - NAME is correct escaped for UTF8 + latin1 stashes ok 127 - ENAME is correct escaped for UTF8 + latin1 stashes ok 128 - DumpProg() + package declaration ok 129 - DumpProg() + UTF8 package declaration ok 130 - use utf8; sub ヌ::ネ {1}; ヌ::ネ;use Devel::Pe... ok 131 - use utf8; ヌ: { last ヌ }use Devel::Peek "DumpPr... ok 132 - DumpProg() has no 'Attempt to free X prematurely' warning 1..132 Dubious\, test returned 5 (wstat 1280\, 0x500) Failed 5/132 subtests (less 2 skipped subtests: 125 okay)
Test Summary Report ------------------- op/each.t (Wstat: 0 Tests: 59 Failed: 1) Failed test: 56 run/runenv.t (Wstat: 0 Tests: 106 Failed: 5) Failed tests: 46\, 48\, 50\, 52\, 54 ../ext/Devel-Peek/t/Peek.t (Wstat: 1280 Tests: 132 Failed: 5) Failed tests: 55\, 57\, 59\, 63\, 65 Non-zero exit status: 5 Files=3\, Tests=297\, 0 wallclock secs ( 0.05 usr 0.00 sys + 0.16 cusr 0.05 csys = 0.26 CPU) Result: FAIL
p5pRT
commented
6 years ago I design ASICs and run lots of simulations of electrical circuits. The simulations produce lots of numbers (Voltages\, Currents\, etc.). I store them in SQLite and I process these SQLite databases using perl to make computations and create reports.
When i have fetched the data from a database\, i build a tree of nested hashes in perl that contains the actual data-values at its leafs. The keys of the hashes between the root-node and the leafs are the parameter values of a the simulation run. This kind of data structure fits particularly well. I then traverse through these trees and calculate all sorts of things from the data.
Some average numbers: The tree depth is up to around 20 layers and a hash contains on average around 1 to 1000 keys (usually numbers\, but sometimes strings of approx. 1 to 100 characters). The run time of these scripts varies but it is easily in the range of 30 minutes for larger amounts of data. I wonder whether there are any low hanging fruits that i can grab to speed this process up.
So i analyzed my code using nytprof (a lot of time goes into IO of course) but also into computation. So I optimized the tree traversal code from an algorithmic viewpoint. Now that only micro-optimizations are left\, i shifted the focus to perl itself. (Any low-hanging fruits there?)
I have seen that you have put forth a lot of effort into making the hash infrastructure of perl attack proof (and this certainly makes sense).
My custom built perl however lives in a completely offline laboratory and security is not a topic. (If it could be 1 percent faster i would trade in all the security for this performance gain). Thats my motivation behind this effort. I have no benchmarks yet\, so i can't tell if it is worth it. Still working on it and trying things.
Regarding the hash-function itself: I would love to try out other hash-functions (especially the less secure but faster ones). I have seen that Yves once had a git-branch which contained a version of perl that provided a larger number of hash-functions to select from\, like crc32 and murmur3. Unfortunately these were removed from the code and the hash-infrastructure was upgraded in between to the new "state-based-hash". I could not in reasonable time figure out how i could get these hash functions back into the current git perl with the new hash infrastructure. (I am neither a perl nor a hash expert). So i left the hash function at its default. (StadtX\, i believe).
Regards\, Thomas
On Fri\, Aug 24\, 2018 at 5:46 PM\, demerphq \demerphq@​gmail\.com wrote:
Is it possible you could explain why you are building like this and which hash function you are using?
Yves
On Fri\, 24 Aug 2018\, 17:08 fnordpole@gmail.com\, \fnordpole@​gmail\.com wrote:
Thank you for looking into this! Looks good. This is exactly what i observed.
Regarding the compile time warning: I also observed the warning of in_collision being uninitialized (without the patch)\, which makes sense. The warning disappears after the patch is applied.
The other warnings about usused variables though - i don't know where they come from. They must be related to the use of the switch somehow but they are probably not related to this issue.
Regarding the failed tests: I can also confirm some tests are failing when compiled with the switch but i did not care much about it because in hv.h\, it is stated that some tests may fail. So this is probably fine.
------- exerpt from hv.h ------ /* These control hash traversal randomization and the environment variable PERL_PERTURB_KEYS. * Currently disabling this functionality will break a few tests\, but should otherwise work fine. * See perlrun for more details. */
#if defined(PERL_PERTURB_KEYS_DISABLED) -------------------------------
On Wed\, Aug 22\, 2018 at 11:07 PM\, James E Keenan via RT \< perlbug-followup@perl.org> wrote:
On Mon\, 20 Aug 2018 23:42:55 GMT\, 11000@gmx.de wrote:
Hi\, I would like to provide a little patch that fixes a bug introduced somewhere around 5.26. The bug makes perl stall immediately because all hash keys will go into the same bucket\, because the hash-split is broken. Fortunately\, this bug is only triggered if you use the compile-time switch -Accflags=-DPERL_PERTURB_KEYS_DISABLED. I reckon i am the first person to ever use this switch since around 5.26! :) The patch i provided is very simple and fixes the bug by moving the in_collision flag assignment out of the #ifdef PERL_RANDOMIZE_KEYS. I think this assignment must always be included in the code\, no matter if the randomize keys feature is enabled or not. It is important that this assignment is computed always\, because the new hash split algorithm introduced somewhere around 5.26 will only split if the hash is 1) full (up to a certain level) and 2) a collision happens. If however the in_collision flag is never set\, the hash will never split. So we have to move its assignment out of that #ifdef. Regards\,Thomas Schweiger
1. I tried to build Perl 5 blead by adding your switch to my regular configuration syntax:
##### sh ./Configure -des -Dusedevel -Accflags=-DPERL_PERTURB_KEYS_DISABLED #####
I get this previously unseen compile-time warning:
##### ... cc -c -DPERL_CORE -DPERL_PERTURB_KEYS_DISABLED -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c89 -O2 -Wall -Werror=declaration-after-statement -Werror=pointer-arith -Wextra -Wc++-compat -Wwrite-strings hv.c hv.c: In function ‘Perl_hv_rand_set’: hv.c:2344:23: warning: unused variable ‘iter’ [-Wunused-variable] struct xpvhv_aux *iter; ^ hv.c:2343:28: warning: unused parameter ‘hv’ [-Wunused-parameter] Perl_hv_rand_set(pTHX_ HV *hv\, U32 new_xhv_rand) { ^ hv.c:2343:36: warning: unused parameter ‘new_xhv_rand’ [-Wunused-parameter] Perl_hv_rand_set(pTHX_ HV *hv\, U32 new_xhv_rand) { ^ hv.c: In function ‘Perl_hv_common’: hv.c:886:8: warning: ‘in_collision’ may be used uninitialized in this function [-Wmaybe-uninitialized] if ( in_collision && DO_HSPLIT(xhv) ) { ^ ... #####
Then 'make' appeared to grind to a halt at:
##### ... Processing extracted/DJoinGroup.txt Processing extracted/DJoinType.txt Processing Jamo.txt Processing UnicodeData.txt #####
I did not attempt to proceed to 'make test'.
2. I created a branch and applied your patch to the branch:
smoke-me/jkeenan/133459-perturb
I configured and built perl as I normally do. 'make test_harness' PASS.
3. In the patch branched\, I added your switch to my regular configuration command. I got the same compile-time warning as mentioned above\, but 'make' did not appear to hang. However\, I got failures in three tests:
##### run/runenv.t (Wstat: 0 Tests: 106 Failed: 5) Failed tests: 46\, 48\, 50\, 52\, 54 op/each.t (Wstat: 0 Tests: 59 Failed: 1) Failed test: 56 ../ext/Devel-Peek/t/Peek.t (Wstat: 1280 Tests: 132 Failed: 5) Failed tests: 55\, 57\, 59\, 63\, 65 Non-zero exit status: 5 #####
Output attached.
-- James E Keenan (jkeenan@cpan.org)
# Failed test 56 - each() after insert produces warnings at op/each.t line 276 op/each.t ................... 1..59 ok 1 - keys ok 2 - values ok 3 - each count ok 4 - added a key ok 5 - resize ok 6 - same size ok 7 - still same size ok 8 - size 8 ok 9 - test scalar each ok 10 - test scalar keys resets iterator ok 11 - test iterator of each is being maintained ok 12 - test values keys resets iterator ok 13 - assign to keys does not shrink hash bucket array ok 14 - assignment to keys of a number not large enough does not change size ok 15 - keys (%hash) ok 16 ok 17 # DESTROY called ok 18 ok 19 - Check length of "\x{12345}" ok 20 - Check length of "\x{10fffd}" ok 21 - Check length of "\x{1234}" ok 22 - Check length of "\x{123}" ok 23 - Check length of "\022" ok 24 - copied unicode hash keys correctly? ok 25 - utf8 key in bytes hash ok 26 - bytes key in utf8 hash # non-utf8 ok 27 - if we got here change 8056 worked # utf8 ok 28 - change 8056 is thanks to Inaba Hiroto ok 29 - check encoding on EBCDIC ok 30 - check length ok 31 - check value ok 32 \https://maps.google.com/?q=32+%0D%0Aok+33&entry=gmail&source=g ok 33 - four different Unicode keys ok 34 - U+00FF -> 1 ok 35 - U+00C3 U+00BF -> 2 ok 36 - U+0100 -> 3 \https://maps.google.com/?q=3++%0D%0Aok+37&entry=gmail&source=g ok 37 - U+00C4 U+0080 -> 4 ok 38 - each demands argument ok 39 - keys demands argument ok 40 - values demands argument ok 41 - if(%foo) was true - my ok 42 - if(%foo) didnt mess with each (key) - my ok 43 - if(%foo) didnt mess with each (value) - my ok 44 - Got the expected number of keys - my ok 45 - Got the count of keys from %foo in scalar assignment context - my ok 46 - Got expected number of elements in list context - my ok 47 - if(%foo) was true - our ok 48 - if(%foo) didnt mess with each (key) - our ok 49 - if(%foo) didnt mess with each (value) - our ok 50 - Got the expected number of keys - our ok 51 - Got the count of keys from %foo in scalar assignment context - our ok 52 - Got expected number of elements in list context - our ok 53 - single key not yet freed ok 54 - single key now freed ok 55 - each on apparently empty hash does not leave RITER set not ok 56 - each() after insert produces warnings ok 57 - no warnings 'internal' silences each() after insert warnings ok 58 - each in list assignment ok 59 - values in list assignment Failed 1/59 subtests # Failed test 46 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr at run/runenv.t line 232 # Failed test 48 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr at run/runenv.t line 239 # Failed test 50 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stderr at run/runenv.t line 244 # Failed test 52 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stderr at run/runenv.t line 249 # Failed test 54 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stderr at run/runenv.t line 254 # Failed test 'ENAME on a stash' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAME = "RWOM" # ENAME = "RWOM" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAME = "RWOM" # ENAME = "RWOM" # \Z)' # # Failed test 'ENAMEs on a stash' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 3 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAME = "RWOM" # NAMECOUNT = 2 # ENAME = "RWOM"\, "KLANK" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 3 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAME = "RWOM" # NAMECOUNT = 2 # ENAME = "RWOM"\, "KLANK" # \Z)' # # Failed test 'ENAMEs on a stash with no NAME' # at t/Peek.t line 98. # # 'SV = IV(0x2e19f38) at 0x2e19f48 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25ddf48 # SV = PVHV(0x2d92a10) at 0x25ddf48 # REFCNT = 3 # FLAGS = (OOK\,SHAREKEYS\,OVERLOAD) # AUX_FLAGS = 0 # ARRAY = 0x2dd7290 # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # # NAMECOUNT = -3 # ENAME = "RWOM"\, "KLANK" # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 3 # FLAGS = \(OOK\,SHAREKEYS\,OVERLOAD\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ # KEYS = 0 # FILL = 0 # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # NAMECOUNT = -3 # ENAME = "RWOM"\, "KLANK" # \Z)' # # Failed test 'small hash after keys' # at t/Peek.t line 98. # # 'SV = IV(0x2e19c20) at 0x2e19c30 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25de050 # SV = PVHV(0x2d92a90) at 0x25de050 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2e23b70 (0:6\, 1:2) # hash quality = 125.0% # KEYS = 2 # FILL = 2 # MAX = 7 # RITER = -1 # EITER = 0x0 # # Elt "Beer" HASH = 0xaf2bb028 # SV = PV(0x2dfc2e0) at 0x2e154e0 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7670 "Foamy"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # Elt "Perl" HASH = 0xe22b31e6 # SV = PV(0x2dfc2d0) at 0x2e19f48 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7820 "Rules"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ \(0:[67]\,.*\) # hash quality = [0-9.]+% # KEYS = 2 # FILL = [12] # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # (?: Elt "(?:Perl|Beer)" HASH = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(POK\,(?:IsCOW\,)?pPOK\) # PV = 0x[[:xdigit:]]+ "(?:Rules|Foamy)"\\0 # CUR = \d+ # LEN = \d+ # COW_REFCNT = 1 # ){2}\Z)' # # Failed test 'small hash after keys and scalar' # at t/Peek.t line 98. # # 'SV = IV(0x2e19c20) at 0x2e19c30 # REFCNT = 1 # FLAGS = (ROK) # RV = 0x25de050 # SV = PVHV(0x2d92a90) at 0x25de050 # REFCNT = 2 # FLAGS = (OOK\,SHAREKEYS) # AUX_FLAGS = 0 # ARRAY = 0x2e23b70 (0:6\, 1:2) # hash quality = 125.0% # KEYS = 2 # FILL = 2 # MAX = 7 # RITER = -1 # EITER = 0x0 # # Elt "Beer" HASH = 0xaf2bb028 # SV = PV(0x2dfc2e0) at 0x2e154e0 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7670 "Foamy"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # Elt "Perl" HASH = 0xe22b31e6 # SV = PV(0x2dfc2d0) at 0x2e19f48 # REFCNT = 1 # FLAGS = (POK\,IsCOW\,pPOK) # PV = 0x2dd7820 "Rules"\0 # CUR = 5 # LEN = 10 # COW_REFCNT = 1 # ' # doesn't match '(?^ms:\A(?:ALLOCATED at .*? # )?SV = IV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(ROK\) # RV = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PVHV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 2 # FLAGS = \(OOK\,SHAREKEYS\) # AUX_FLAGS = 0 # ARRAY = 0x[[:xdigit:]]+ \(0:[67]\,.*\) # hash quality = [0-9.]+% # KEYS = 2 # FILL = ([12]) # MAX = 7 # RITER = -1 # EITER = 0x0 # RAND = 0x[[:xdigit:]]+ # (?: Elt "(?:Perl|Beer)" HASH = 0x[[:xdigit:]]+ # (?: ALLOCATED at .*? # )? SV = PV\(0x[[:xdigit:]]+\) at 0x[[:xdigit:]]+ # REFCNT = 1 # FLAGS = \(POK\,(?:IsCOW\,)?pPOK\) # PV = 0x[[:xdigit:]]+ "(?:Rules|Foamy)"\\0 # CUR = \d+ # LEN = \d+ # COW_REFCNT = 1 # ){2}\Z)' # run/runenv.t ................ 1..106 ok 1 - PERL5OPT => -w\,-e\,print $::x stdout ok 2 - PERL5OPT => -w\,-e\,print $::x stderr ok 3 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $::x stdout ok 4 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $::x stderr ok 5 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $x stdout ok 6 - PERL5OPT => -Mstrict\,-I../lib\,-e\,print $x stderr ok 7 - PERL5OPT => -Mstrict -w\,-I../lib\,-e\,print $x stdout ok 8 - PERL5OPT => -Mstrict -w\,-I../lib\,-e\,print $x stderr ok 9 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stdout ok 10 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stderr ok 11 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stdout ok 12 - PERL5OPT => -w -Mstrict\,-I../lib\,-e\,print $::x stderr ok 13 - PERL5OPT => -MExporter\,-I../lib\,-e0 stdout ok 14 - PERL5OPT => -MExporter\,-I../lib\,-e0 stderr ok 15 - PERL5OPT => -MExporter -MExporter\,-I../lib\,-e0 stdout ok 16 - PERL5OPT => -MExporter -MExporter\,-I../lib\,-e0 stderr ok 17 - PERL5OPT => -Mstrict -Mwarnings\,-I../lib\,-e\,print "ok" if $INC{" strict.pm"} and $INC{"warnings.pm"} stdout ok 18 - PERL5OPT => -Mstrict -Mwarnings\,-I../lib\,-e\,print "ok" if $INC{" strict.pm"} and $INC{"warnings.pm"} stderr ok 19 - PERL5OPT => -I. -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stdout ok 20 - PERL5OPT => -I. -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stderr ok 21 - PERL5OPT => -I./ -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stdout ok 22 - PERL5OPT => -I./ -MtmpOooof\,-e\,print "ok" if $INC{"tmpOooof.pm"} eq "tmpOooof.pm" stderr ok 23 - PERL5OPT => -w -w\,-e\,print $ENV{PERL5OPT} stdout ok 24 - PERL5OPT => -w -w\,-e\,print $ENV{PERL5OPT} stderr ok 25 - PERL5OPT => -t\,-e\,print ${^TAINT} stdout ok 26 - PERL5OPT => -t\,-e\,print ${^TAINT} stderr ok 27 - PERL5OPT => -W\,-I../lib\,-e\,local $^W = 0; no warnings; print $x stdout ok 28 - PERL5OPT => -W\,-I../lib\,-e\,local $^W = 0; no warnings; print $x stderr ok 29 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stdout ok 30 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stderr ok 31 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stdout ok 32 - PERLLIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stderr ok 33 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stdout ok 34 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "foobar" } @INC stderr ok 35 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stdout ok 36 - PERL5LIB => foobar:42\,-e\,print grep { $_ eq "42" } @INC stderr ok 37 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "foo" } @INC stdout ok 38 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "foo" } @INC stderr ok 39 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "bar" } @INC stdout ok 40 - PERL5LIB => foo\,PERLLIB => bar\,-e\,print grep { $_ eq "bar" } @INC stderr ok 41 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 42 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 43 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 44 - PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 45 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout not ok 46 - PERL_HASH_SEED => 0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 47 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout not ok 48 - PERL_HASH_SEED => 0x0\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 49 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stdout not ok 50 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 0\,-e\,1 stderr ok 51 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stdout not ok 52 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 1\,-e\,1 stderr ok 53 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stdout not ok 54 - PERL_HASH_SEED_DEBUG => 1\,PERL_PERTURB_KEYS => 2\,-e\,1 stderr ok 55 - PERL_HASH_SEED => 12345678\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 56 - PERL_HASH_SEED => 12345678\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 57 - PERL_HASH_SEED => 12\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 58 - PERL_HASH_SEED => 12\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 59 - PERL_HASH_SEED => 123456789\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stdout ok 60 - PERL_HASH_SEED => 123456789\,PERL_HASH_SEED_DEBUG => 1\,-e\,1 stderr ok 61 - No errors when determining @INC ok 62 - . is not in @INC ok 63 - No errors when determining unsafe @INC ok 64 - . last in unsafe @INC ok 65 - No errors when determining @INC for nothing ok 66 - expected number of elements in @INC for nothing ok 67 - expected elements in @INC for nothing ok 68 - No errors when determining @INC for something ok 69 - expected number of elements in @INC for something ok 70 - expected elements in @INC for something ok 71 - No errors when determining @INC for two things ok 72 - expected number of elements in @INC for two things ok 73 - expected elements in @INC for two things ok 74 - No errors when determining @INC for two things\, :: ok 75 - expected number of elements in @INC for two things\, :: ok 76 - expected elements in @INC for two things\, :: ok 77 - No errors when determining @INC for : at start ok 78 - expected number of elements in @INC for : at start ok 79 - expected elements in @INC for : at start ok 80 - No errors when determining @INC for : at end ok 81 - expected number of elements in @INC for : at end ok 82 - expected elements in @INC for : at end ok 83 - No errors when determining @INC for :: sandwich :: ok 84 - expected number of elements in @INC for :: sandwich :: ok 85 - expected elements in @INC for :: sandwich :: ok 86 - No errors when determining @INC for : ok 87 - expected number of elements in @INC for : ok 88 - expected elements in @INC for : ok 89 - No errors when determining @INC for :: ok 90 - expected number of elements in @INC for :: ok 91 - expected elements in @INC for :: ok 92 - No errors when determining @INC for ::: ok 93 - expected number of elements in @INC for ::: ok 94 - expected elements in @INC for ::: ok 95 - No errors when determining @INC for two things and : ok 96 - expected number of elements in @INC for two things and : ok 97 - expected elements in @INC for two things and : ok 98 - No errors when determining @INC for : and two things ok 99 - expected number of elements in @INC for : and two things ok 100 - expected elements in @INC for : and two things ok 101 - No errors when determining @INC for : two things : ok 102 - expected number of elements in @INC for : two things : ok 103 - expected elements in @INC for : two things : ok 104 - No errors when determining @INC for three things ok 105 - expected number of elements in @INC for three things ok 106 - expected elements in @INC for three things Failed 5/106 subtests # Looks like you failed 5 tests of 132. ../ext/Devel-Peek/t/Peek.t .. ok 1 - assignment of immediate constant (string) ok 2 - assignment of immediate constant (string) (unchanged by dump) ok 3 - immediate constant (string) ok 4 - immediate constant (string) (unchanged by dump) ok 5 - assignment of immediate constant (integer) ok 6 - assignment of immediate constant (integer) (unchanged by dump) ok 7 - immediate constant (integer) ok 8 - immediate constant (integer) (unchanged by dump) ok 9 - assignment of immediate constant (integer) ok 10 - assignment of immediate constant (integer) (unchanged by dump) ok 11 - result of addition ok 12 - result of addition (unchanged by dump) ok 13 - floating point value ok 14 - floating point value (unchanged by dump) ok 15 - integer constant ok 16 - integer constant (unchanged by dump) ok 17 - undef ok 18 - undef (unchanged by dump) ok 19 - reference to scalar ok 20 - reference to scalar (unchanged by dump) ok 21 - reference to array ok 22 - reference to array (unchanged by dump) ok 23 - reference to hash ok 24 - reference to hash (unchanged by dump) ok 25 - reference to anon sub with empty prototype ok 26 - reference to anon sub with empty prototype (unchanged by dump) ok 27 - reference to named subroutine without prototype ok 28 - reference to named subroutine without prototype (unchanged by dump) ok 29 - reference to regexp ok 30 - reference to regexp (unchanged by dump) ok 31 - reference to blessed hash ok 32 - reference to blessed hash (unchanged by dump) ok 33 - typeglob ok 34 - typeglob (unchanged by dump) ok 35 - string with Unicode ok 36 - string with Unicode (unchanged by dump) ok 37 - reference to hash containing Unicode ok 38 - reference to hash containing Unicode (unchanged by dump) ok 39 - scalar with pos magic ok 40 - scalar with pos magic (unchanged by dump) ok 41 - tainted value in %ENV ok 42 - tainted value in %ENV (unchanged by dump) ok 43 - blessed reference ok 44 - blessed reference (unchanged by dump) ok 45 - constant subroutine ok 46 - constant subroutine (unchanged by dump) ok 47 - isUV should show on PVMG ok 48 - isUV should show on PVMG (unchanged by dump) ok 49 - IO ok 50 - IO (unchanged by dump) ok 51 - FORMAT ok 52 - FORMAT (unchanged by dump) ok 53 - blessing to a class with embedded NUL characters ok 54 - blessing to a class with embedded NUL characters (unchanged by dump) not ok 55 - ENAME on a stash # line 773 ok 56 - ENAME on a stash (unchanged by dump) not ok 57 - ENAMEs on a stash # line 796 ok 58 - ENAMEs on a stash (unchanged by dump) not ok 59 - ENAMEs on a stash with no NAME # line 820 ok 60 - ENAMEs on a stash with no NAME (unchanged by dump) ok 61 - small hash ok 62 - small hash (unchanged by dump) not ok 63 - small hash after keys # line 871 ok 64 - small hash after keys (unchanged by dump) not ok 65 - small hash after keys and scalar # line 901 \https://maps.google.com/?q=901+%0D%0Aok+66&entry=gmail&source=g ok 66 - small hash after keys and scalar (unchanged by dump) ok 67 - Dump @array ok 68 - Dump @array (unchanged by dump) ok 69 - Dump @array\,1 ok 70 - Dump @array\,1 (unchanged by dump) ok 71 - Dump %hash ok 72 - Dump %hash (unchanged by dump) ok 73 - rvalue substr ok 74 - rvalue substr (unchanged by dump) ok 75 - Dump; ok 76 - Dump() ok 77 # skip Not built with usemymalloc ok 78 # skip Not built with usemymalloc ok 79 - regular string constant ok 80 - regular string constant (unchanged by dump) ok 81 - string constant now an FBM ok 82 - string constant now an FBM (unchanged by dump) ok 83 - Not allowed to study an FBM ok 84 - string constant still an FBM ok 85 - string constant still an FBM (unchanged by dump) ok 86 - regular string constant ok 87 - regular string constant (unchanged by dump) ok 88 - Our studies were successful ok 89 - string constant quite unaffected ok 90 - string constant quite unaffected (unchanged by dump) ok 91 - correct index ok 92 - string constant now FBMed ok 93 - string constant now FBMed (unchanged by dump) ok 94 - Our studies were successful ok 95 - string constant still FBMed ok 96 - string constant still FBMed (unchanged by dump) ok 97 - second string also unaffected ok 98 - second string also unaffected (unchanged by dump) ok 99 - no crash with DeadCode ok 100 - UTF-8 in a regular expression ok 101 - UTF-8 in a regular expression (unchanged by dump) ok 102 - SvREFCNT on non-scalar ok 103 - requires prototype ok 104 - GVGV's are correctly escaped for latin1 :: latin1 ok 105 - GVGV's are correctly escaped for UTF8 :: UTF8 ok 106 - GVGV's are correctly escaped for latin1 :: UTF8 ok 107 - GVGV's are correctly escaped for UTF8 :: latin1 ok 108 - GVGV's are correctly escaped for UTF8 :: latin 1 :: UTF8 ok 109 - NAME is correctly escaped for UTF8 globs ok 110 - GvSTASH is correctly escaped for UTF8 globs ok 111 - EGV is correctly escaped for UTF8 globs ok 112 - NAME is correctly escaped for UTF8 globs with latin1 stashes ok 113 - GvSTASH is correctly escaped for UTF8 globs with latin1 stashes ok 114 - EGV is correctly escaped for UTF8 globs with latin1 stashes ok 115 - STASH for blessed hashrefs is correct ok 116 - PROTOTYPE is escaped correctly ok 117 - PROTOTYPE works on lexical subs ok 118 - OUTSIDE works ok 119 - OUTSIDE + UTF8 works ok 120 - NAME is correct escaped for UTF8 stashes ok 121 - ENAME is correct escaped for UTF8 stashes ok 122 - NAME is correct escaped for latin 1 stashes ok 123 - ENAME is correct escaped for latin 1 stashes ok 124 - NAME is correct escaped for latin1 + UTF8 stashes ok 125 - ENAME is correct escaped for latin1 + UTF8 stashes ok 126 - NAME is correct escaped for UTF8 + latin1 stashes ok 127 - ENAME is correct escaped for UTF8 + latin1 stashes ok 128 - DumpProg() + package declaration ok 129 - DumpProg() + UTF8 package declaration ok 130 - use utf8; sub ヌ::ネ {1}; ヌ::ネ;use Devel::Pe... ok 131 - use utf8; ヌ: { last ヌ }use Devel::Peek "DumpPr... ok 132 - DumpProg() has no 'Attempt to free X prematurely' warning 1..132 Dubious\, test returned 5 (wstat 1280\, 0x500) Failed 5/132 subtests (less 2 skipped subtests: 125 okay)
Test Summary Report ------------------- op/each.t (Wstat: 0 Tests: 59 Failed: 1) Failed test: 56 run/runenv.t (Wstat: 0 Tests: 106 Failed: 5) Failed tests: 46\, 48\, 50\, 52\, 54 ../ext/Devel-Peek/t/Peek.t (Wstat: 1280 Tests: 132 Failed: 5) Failed tests: 55\, 57\, 59\, 63\, 65 Non-zero exit status: 5 Files=3\, Tests=297\, 0 wallclock secs ( 0.05 usr 0.00 sys + 0.16 cusr 0.05 csys = 0.26 CPU) Result: FAIL
p5pRT
commented
6 years ago On Fri\, 24 Aug 2018 at 23:00\, fnordpole@gmail.com \fnordpole@​gmail\.com wrote:
I design ASICs and run lots of simulations of electrical circuits. The simulations produce lots of numbers (Voltages\, Currents\, etc.). I store them in SQLite and I process these SQLite databases using perl to make computations and create reports.
When i have fetched the data from a database\, i build a tree of nested hashes in perl that contains the actual data-values at its leafs. The keys of the hashes between the root-node and the leafs are the parameter values of a the simulation run. This kind of data structure fits particularly well. I then traverse through these trees and calculate all sorts of things from the data.
Some average numbers: The tree depth is up to around 20 layers and a hash contains on average around 1 to 1000 keys (usually numbers\, but sometimes strings of approx. 1 to 100 characters). The run time of these scripts varies but it is easily in the range of 30 minutes for larger amounts of data. I wonder whether there are any low hanging fruits that i can grab to speed this process up.
So i analyzed my code using nytprof (a lot of time goes into IO of course) but also into computation. So I optimized the tree traversal code from an algorithmic viewpoint. Now that only micro-optimizations are left\, i shifted the focus to perl itself. (Any low-hanging fruits there?)
I have seen that you have put forth a lot of effort into making the hash infrastructure of perl attack proof (and this certainly makes sense).
My custom built perl however lives in a completely offline laboratory and security is not a topic. (If it could be 1 percent faster i would trade in all the security for this performance gain). Thats my motivation behind this effort. I have no benchmarks yet\, so i can't tell if it is worth it. Still working on it and trying things.
Regarding the hash-function itself: I would love to try out other hash-functions (especially the less secure but faster ones). I have seen that Yves once had a git-branch which contained a version of perl that provided a larger number of hash-functions to select from\, like crc32 and murmur3. Unfortunately these were removed from the code and the hash-infrastructure was upgraded in between to the new "state-based-hash". I could not in reasonable time figure out how i could get these hash functions back into the current git perl with the new hash infrastructure. (I am neither a perl nor a hash expert). So i left the hash function at its default. (StadtX\, i believe).
Interesting\, thank you for the explanation. Could you give me some kind of idea of the key sizes?
FWIW\, I doubt removing the key perturbance will change much of anything in the profile. It adds something like two instructions per hash store and iteration\, which is a drop in the bucket. But I am keen to hear your results. Also the hash function in later perls Stadtx is about as fast as it gets\, as is SBOX. We would hash keys \<= 24 bytes with SBOX and then longer strings with Stadtx. You may find that adjusting the SBOX32_MAX_LEN gives some speedup. (Try both raising and lowering it\, there are complex factors at play governing their relative speed.)
If really fast hash operations are really going to make your algorithm faster then you will want to avoid using Perl hashes and use something custom implemented in C/C++. Perls hash code has to deal with a lot of special cases that wont apply for a specific work load.
Yves
p5pRT
commented
6 years ago Yves\,
thank you for your suggestions! I bulit 4 versions of perl and varied the switches you suggested:
1) default (no switches at all) 2) PERL_PERTURB_KEYS_DISABLED 3) SBOX32_MAX_LEN=12 4) SBOX32_MAX_LEN=48
I created a benchmark script that runs a typical workload and profiled it it nytprof. FWIW\, I rebooted my machine before each run to reduce the intrinsic variation of the profiling itself.
These are the results
1) default (no switches at all) Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 81.0s (of 199s) Run 2: Profile of /home/tho/fx/Dev/run_noise_perf for 80.4s (of 198s)
2) PERL_PERTURB_KEYS_DISABLED Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 83.7s (of 202s) Run 2: Profile of /home/tho/fx/Dev/run_noise_perf for 81.9s (of 200s)
3) SBOX32_MAX_LEN=12 Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 80.6s (of 199s)
4) SBOX32_MAX_LEN=48 Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 80.7s (of 200s)
My conclusion is: - the differences between the various configurations for my typical workload seem to be in the same order as the intrinsic variation of the profiling. - more data would be needed to make a sound decision\, averaging out the intrinsic variations of the profiling. - my guess is that no big differences are to be expected. So\, apparently no low hanging fruits to harvest here.
FYI: The key-length is on average around 16 characters. (stringified floating point numbers\, just like this: 3981071.70553499)
Thank you again Yves\, for providing help. I really appreciated this little investigation. If performance of my scripts should become a bottleneck one day\, i would go for the C/C++ approach. But for the time being i enjoy perl pretty much and will leave it by this :)
Regards\, Thomas
On Sat\, Aug 25\, 2018 at 8:27 AM\, demerphq \demerphq@​gmail\.com wrote:
On Fri\, 24 Aug 2018 at 23:00\, fnordpole@gmail.com \fnordpole@​gmail\.com wrote:
I design ASICs and run lots of simulations of electrical circuits. The simulations produce lots of numbers (Voltages\, Currents\, etc.). I store them in SQLite and I process these SQLite databases using perl to make computations and create reports.
When i have fetched the data from a database\, i build a tree of nested hashes in perl that contains the actual data-values at its leafs. The keys of the hashes between the root-node and the leafs are the parameter values of a the simulation run. This kind of data structure fits particularly well. I then traverse through these trees and calculate all sorts of things from the data.
Some average numbers: The tree depth is up to around 20 layers and a hash contains on average around 1 to 1000 keys (usually numbers\, but sometimes strings of approx. 1 to 100 characters). The run time of these scripts varies but it is easily in the range of 30 minutes for larger amounts of data. I wonder whether there are any low hanging fruits that i can grab to speed this process up.
So i analyzed my code using nytprof (a lot of time goes into IO of course) but also into computation. So I optimized the tree traversal code from an algorithmic viewpoint. Now that only micro-optimizations are left\, i shifted the focus to perl itself. (Any low-hanging fruits there?)
I have seen that you have put forth a lot of effort into making the hash infrastructure of perl attack proof (and this certainly makes sense).
My custom built perl however lives in a completely offline laboratory and security is not a topic. (If it could be 1 percent faster i would trade in all the security for this performance gain). Thats my motivation behind this effort. I have no benchmarks yet\, so i can't tell if it is worth it. Still working on it and trying things.
Regarding the hash-function itself: I would love to try out other hash-functions (especially the less secure but faster ones). I have seen that Yves once had a git-branch which contained a version of perl that provided a larger number of hash-functions to select from\, like crc32 and murmur3. Unfortunately these were removed from the code and the hash-infrastructure was upgraded in between to the new "state-based-hash". I could not in reasonable time figure out how i could get these hash functions back into the current git perl with the new hash infrastructure. (I am neither a perl nor a hash expert). So i left the hash function at its default. (StadtX\, i believe).
Interesting\, thank you for the explanation. Could you give me some kind of idea of the key sizes?
FWIW\, I doubt removing the key perturbance will change much of anything in the profile. It adds something like two instructions per hash store and iteration\, which is a drop in the bucket. But I am keen to hear your results. Also the hash function in later perls Stadtx is about as fast as it gets\, as is SBOX. We would hash keys \<= 24 bytes with SBOX and then longer strings with Stadtx. You may find that adjusting the SBOX32_MAX_LEN gives some speedup. (Try both raising and lowering it\, there are complex factors at play governing their relative speed.)
If really fast hash operations are really going to make your algorithm faster then you will want to avoid using Perl hashes and use something custom implemented in C/C++. Perls hash code has to deal with a lot of special cases that wont apply for a specific work load.
Yves
p5pRT
commented
6 years ago On Sat\, 25 Aug 2018\, 18:53 fnordpole@gmail.com\, \fnordpole@​gmail\.com wrote:
Yves\,
thank you for your suggestions! I bulit 4 versions of perl and varied the switches you suggested:
1) default (no switches at all) 2) PERL_PERTURB_KEYS_DISABLED 3) SBOX32_MAX_LEN=12 4) SBOX32_MAX_LEN=48
I created a benchmark script that runs a typical workload and profiled it it nytprof. FWIW\, I rebooted my machine before each run to reduce the intrinsic variation of the profiling itself.
These are the results
1) default (no switches at all) Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 81.0s (of 199s) Run 2: Profile of /home/tho/fx/Dev/run_noise_perf for 80.4s (of 198s)
2) PERL_PERTURB_KEYS_DISABLED Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 83.7s (of 202s) Run 2: Profile of /home/tho/fx/Dev/run_noise_perf for 81.9s (of 200s)
3) SBOX32_MAX_LEN=12 Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 80.6s (of 199s)
4) SBOX32_MAX_LEN=48 Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 80.7s (of 200s)
My conclusion is: - the differences between the various configurations for my typical workload seem to be in the same order as the intrinsic variation of the profiling. - more data would be needed to make a sound decision\, averaging out the intrinsic variations of the profiling. - my guess is that no big differences are to be expected. So\, apparently no low hanging fruits to harvest here.
FYI: The key-length is on average around 16 characters. (stringified floating point numbers\, just like this: 3981071.70553499)
Thank you again Yves\, for providing help. I really appreciated this little investigation. If performance of my scripts should become a bottleneck one day\, i would go for the C/C++ approach. But for the time being i enjoy perl pretty much and will leave it by this :
Very interesting results. Thank you very much.
I just wanted to say I was not actually suggesting you move entirely away from perl but rather that you will likely find that a bit of effort in writing some well chosen XS tools which do the hashing will result in considerable speedups.
So for instance you say that your keys are floats\, when perl hashes these values it converts them to strings first\, which a)introduces the possibility of incorrect collisions due to truncation/rounding in the binary to strings conversion\, and b) introduces costly extra steps to every hash operation.
A carefully designed XS module on the other hand would likely use some form of integer hash function which would operate on the 8bytes of your float(NV in perl parlence) as a uint64t and be both much faster and avoid the collisions i mentioned earlier. Internally in perl we sometimes this alternative hashing mechanism ourselves.
Having said that it can take some artistry to actually make using XS like this faster than the perl run loop as marshalling from perl to XS is relatively slow. So for instance if you can batch up work from the perl layer to the c/xs layer so the c code does many hash operations per XS function call from perl then you will very likely see a significant speed up.
An example of the work you might avoid by doing this is the fact that mutating hash operations are typically done twice internally by perl\, once in the master hash table and once in the target table. Also keys are recounted and shared so every hash store triggers a refcount increment in the master key table.
Perl makes a set of tradeoffs with things like hashes\, trading speed for memory and vice versa in various ways that can impose a significant burden on specialist use case like yours.
Cheers Yves
p5pRT
commented
6 years ago Yves\,
thank you for your further comments and for pointing out the XS approach! I will certainly give it a try should performance become a bottleneck one day!
Regards\, Thomas
On Sun\, Aug 26\, 2018 at 12:52 PM\, demerphq \demerphq@​gmail\.com wrote:
On Sat\, 25 Aug 2018\, 18:53 fnordpole@gmail.com\, \fnordpole@​gmail\.com wrote:
Yves\,
thank you for your suggestions! I bulit 4 versions of perl and varied the switches you suggested:
1) default (no switches at all) 2) PERL_PERTURB_KEYS_DISABLED 3) SBOX32_MAX_LEN=12 4) SBOX32_MAX_LEN=48
I created a benchmark script that runs a typical workload and profiled it it nytprof. FWIW\, I rebooted my machine before each run to reduce the intrinsic variation of the profiling itself.
These are the results
1) default (no switches at all) Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 81.0s (of 199s) Run 2: Profile of /home/tho/fx/Dev/run_noise_perf for 80.4s (of 198s)
2) PERL_PERTURB_KEYS_DISABLED Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 83.7s (of 202s) Run 2: Profile of /home/tho/fx/Dev/run_noise_perf for 81.9s (of 200s)
3) SBOX32_MAX_LEN=12 Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 80.6s (of 199s)
4) SBOX32_MAX_LEN=48 Run 1: Profile of /home/tho/fx/Dev/run_noise_perf for 80.7s (of 200s)
My conclusion is: - the differences between the various configurations for my typical workload seem to be in the same order as the intrinsic variation of the profiling. - more data would be needed to make a sound decision\, averaging out the intrinsic variations of the profiling. - my guess is that no big differences are to be expected. So\, apparently no low hanging fruits to harvest here.
FYI: The key-length is on average around 16 characters. (stringified floating point numbers\, just like this: 3981071.70553499)
Thank you again Yves\, for providing help. I really appreciated this little investigation. If performance of my scripts should become a bottleneck one day\, i would go for the C/C++ approach. But for the time being i enjoy perl pretty much and will leave it by this :
Very interesting results. Thank you very much.
I just wanted to say I was not actually suggesting you move entirely away from perl but rather that you will likely find that a bit of effort in writing some well chosen XS tools which do the hashing will result in considerable speedups.
So for instance you say that your keys are floats\, when perl hashes these values it converts them to strings first\, which a)introduces the possibility of incorrect collisions due to truncation/rounding in the binary to strings conversion\, and b) introduces costly extra steps to every hash operation.
A carefully designed XS module on the other hand would likely use some form of integer hash function which would operate on the 8bytes of your float(NV in perl parlence) as a uint64t and be both much faster and avoid the collisions i mentioned earlier. Internally in perl we sometimes this alternative hashing mechanism ourselves.
Having said that it can take some artistry to actually make using XS like this faster than the perl run loop as marshalling from perl to XS is relatively slow. So for instance if you can batch up work from the perl layer to the c/xs layer so the c code does many hash operations per XS function call from perl then you will very likely see a significant speed up.
An example of the work you might avoid by doing this is the fact that mutating hash operations are typically done twice internally by perl\, once in the master hash table and once in the target table. Also keys are recounted and shared so every hash store triggers a refcount increment in the master key table.
Perl makes a set of tradeoffs with things like hashes\, trading speed for memory and vice versa in various ways that can impose a significant burden on specialist use case like yours.
Cheers Yves
Migrated from rt.perl.org#133459 (status was 'open')
Searchable as RT133459$