strange 0.000000% classification errors

[garfieldnate]

Even in the latest dev release, many platforms are failing badly, with all of the classification results giving 0.00000% for each possible outcome. All of *nix still fails, and for whatever reason 5.18.0 on Windows also fails with the same problem.

[garfieldnate]

I believe this has something to do with the use of ULONGs. It would be preferable to switch them from unsigned int to uint32_t, but that only works with C99 or newer. Is that acceptable?

[garfieldnate]

It does indeed look like the culprit: http://stackoverflow.com/questions/384502/what-is-the-bit-size-of-long-on-64-bit-windows

Darn Windows for being different! I guess the code would have been wrong on all 64 bit machines if it weren't, so maybe we should be happy that it was so we could solve this problem.

I think that using uint32_t is the best bet. It will make it work on *nix/Windows, 32/64 bit. Otherwise we'd have to do a bunch of #ifdefs.

I suppose also that USHORT shoudl be uint16_t?

[garfieldnate]

So, c1f2e97 produced an even worse error (did this on an ec2 64-bit Ubuntu):

t/01-classify.t ..
1..5
Initializing project /home/ubuntu/Algorithm-AM-2.37/t/data/chapter3
Time:  3:41:44
3 1 2
0/1   3:41:44
Given Context:  3 1 2, myCommentHere
If context is in data file then exclude
Number of data items: 5
Total Excluded: 5
Nulls: exclude
Gang: squared
Number of active variables: 3
*** buffer overflow detected ***: /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/bin/perl terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f18af34408c]
/lib/x86_64-linux-gnu/libc.so.6(+0x111020)[0x7f18af343020]
/home/ubuntu/Algorithm-AM-2.37/blib/arch/auto/Algorithm/AM/AM.so(normalize+0x99)[0x7f18ae1fd4b9]
/home/ubuntu/Algorithm-AM-2.37/blib/arch/auto/Algorithm/AM/AM.so(+0x2e61)[0x7f18ae1fee61]
/home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/bin/perl(Perl_pp_entersub+0x56d)[0x49d60d]
/home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/bin/perl(Perl_runops_standard+0x13)[0x496153]
/home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/bin/perl(perl_run+0x306)[0x439796]
/home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/bin/perl(main+0x105)[0x41db45]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f18af253de5]
/home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/bin/perl[0x41db7d]
======= Memory map: ========
00400000-0055c000 r-xp 00000000 ca:01 152666                             /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/bin/perl
0075b000-0075c000 r--p 0015b000 ca:01 152666                             /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/bin/perl
0075c000-00761000 rw-p 0015c000 ca:01 152666                             /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/bin/perl
00761000-00762000 rw-p 00000000 00:00 0
00c5b000-01545000 rw-p 00000000 00:00 0                                  [heap]
7f18ad368000-7f18ad37d000 r-xp 00000000 ca:01 396830                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f18ad37d000-7f18ad57c000 ---p 00015000 ca:01 396830                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f18ad57c000-7f18ad57d000 r--p 00014000 ca:01 396830                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f18ad57d000-7f18ad57e000 rw-p 00015000 ca:01 396830                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7f18ad57e000-7f18ad586000 r-xp 00000000 ca:01 403342                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Encode/Encode.so
7f18ad586000-7f18ad785000 ---p 00008000 ca:01 403342                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Encode/Encode.so
7f18ad785000-7f18ad786000 r--p 00007000 ca:01 403342                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Encode/Encode.so
7f18ad786000-7f18ad787000 rw-p 00008000 ca:01 403342                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Encode/Encode.so
7f18ad787000-7f18ad78d000 r-xp 00000000 ca:01 403380                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/PerlIO/encoding/encoding.so
7f18ad78d000-7f18ad98c000 ---p 00006000 ca:01 403380                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/PerlIO/encoding/encoding.so
7f18ad98c000-7f18ad98d000 r--p 00005000 ca:01 403380                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/PerlIO/encoding/encoding.so
7f18ad98d000-7f18ad98e000 rw-p 00006000 ca:01 403380                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/PerlIO/encoding/encoding.so
7f18ad98e000-7f18ad9de000 r-xp 00000000 ca:01 280265                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/re/re.so
7f18ad9de000-7f18adbdd000 ---p 00050000 ca:01 280265                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/re/re.so
7f18adbdd000-7f18adbde000 r--p 0004f000 ca:01 280265                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/re/re.so
7f18adbde000-7f18adbdf000 rw-p 00050000 ca:01 280265                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/re/re.so
7f18adbdf000-7f18adbf1000 r-xp 00000000 ca:01 403287                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/POSIX/POSIX.so
7f18adbf1000-7f18addf0000 ---p 00012000 ca:01 403287                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/POSIX/POSIX.so
7f18addf0000-7f18addf3000 r--p 00011000 ca:01 403287                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/POSIX/POSIX.so
7f18addf3000-7f18addf4000 rw-p 00014000 ca:01 403287                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/POSIX/POSIX.so
7f18addf4000-7f18addf8000 r-xp 00000000 ca:01 152852                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Fcntl/Fcntl.so
7f18addf8000-7f18adff7000 ---p 00004000 ca:01 152852                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Fcntl/Fcntl.so
7f18adff7000-7f18adff8000 r--p 00003000 ca:01 152852                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Fcntl/Fcntl.so
7f18adff8000-7f18adff9000 rw-p 00004000 ca:01 152852                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Fcntl/Fcntl.so
7f18adff9000-7f18adffb000 r-xp 00000000 ca:01 280257                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Cwd/Cwd.so
7f18adffb000-7f18ae1fa000 ---p 00002000 ca:01 280257                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Cwd/Cwd.so
7f18ae1fa000-7f18ae1fb000 r--p 00001000 ca:01 280257                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Cwd/Cwd.so
7f18ae1fb000-7f18ae1fc000 rw-p 00002000 ca:01 280257                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Cwd/Cwd.so
7f18ae1fc000-7f18ae200000 r-xp 00000000 ca:01 154565                     /home/ubuntu/Algorithm-AM-2.37/blib/arch/auto/Algorithm/AM/AM.so
7f18ae200000-7f18ae3ff000 ---p 00004000 ca:01 154565                     /home/ubuntu/Algorithm-AM-2.37/blib/arch/auto/Algorithm/AM/AM.so
7f18ae3ff000-7f18ae400000 r--p 00003000 ca:01 154565                     /home/ubuntu/Algorithm-AM-2.37/blib/arch/auto/Algorithm/AM/AM.so
7f18ae400000-7f18ae401000 rw-p 00004000 ca:01 154565                     /home/ubuntu/Algorithm-AM-2.37/blib/arch/auto/Algorithm/AM/AM.so
7f18ae401000-7f18ae40a000 r-xp 00000000 ca:01 405144                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Params/Validate/XS/XS.so
7f18ae40a000-7f18ae609000 ---p 00009000 ca:01 405144                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Params/Validate/XS/XS.so
7f18ae609000-7f18ae60a000 r--p 00008000 ca:01 405144                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Params/Validate/XS/XS.so
7f18ae60a000-7f18ae60b000 rw-p 00009000 ca:01 405144                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Params/Validate/XS/XS.so
7f18ae60b000-7f18ae611000 r-xp 00000000 ca:01 405335                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Package/Stash/XS/XS.so
7f18ae611000-7f18ae810000 ---p 00006000 ca:01 405335                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Package/Stash/XS/XS.so
7f18ae810000-7f18ae811000 r--p 00005000 ca:01 405335                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Package/Stash/XS/XS.so
7f18ae811000-7f18ae812000 rw-p 00006000 ca:01 405335                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Package/Stash/XS/XS.so
7f18ae812000-7f18ae815000 r-xp 00000000 ca:01 405232                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Params/Util/Util.so
7f18ae815000-7f18aea14000 ---p 00003000 ca:01 405232                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Params/Util/Util.so
7f18aea14000-7f18aea15000 r--p 00002000 ca:01 405232                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Params/Util/Util.so
7f18aea15000-7f18aea16000 rw-p 00003000 ca:01 405232                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/site_perl/5.18.2/x86_64-linux/auto/Params/Util/Util.so
7f18aea16000-7f18aea1c000 r-xp 00000000 ca:01 403323                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/List/Util/Util.so
7f18aea1c000-7f18aec1b000 ---p 00006000 ca:01 403323                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/List/Util/Util.so
7f18aec1b000-7f18aec1c000 r--p 00005000 ca:01 403323                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/List/Util/Util.so
7f18aec1c000-7f18aec1d000 rw-p 00006000 ca:01 403323                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/List/Util/Util.so
7f18aec1d000-7f18aec25000 r-xp 00000000 ca:01 280275                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Data/Dumper/Dumper.so
7f18aec25000-7f18aee24000 ---p 00008000 ca:01 280275                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Data/Dumper/Dumper.so
7f18aee24000-7f18aee25000 r--p 00007000 ca:01 280275                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Data/Dumper/Dumper.so
7f18aee25000-7f18aee26000 rw-p 00008000 ca:01 280275                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/Data/Dumper/Dumper.so
7f18aee26000-7f18aee2a000 r-xp 00000000 ca:01 403382                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/IO/IO.so
7f18aee2a000-7f18af029000 ---p 00004000 ca:01 403382                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/IO/IO.so
7f18af029000-7f18af02a000 r--p 00003000 ca:01 403382                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/IO/IO.so
7f18af02a000-7f18af02b000 rw-p 00004000 ca:01 403382                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/IO/IO.so
7f18af02b000-7f18af031000 r-xp 00000000 ca:01 280295                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/File/Glob/Glob.so
7f18af031000-7f18af230000 ---p 00006000 ca:01 280295                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/File/Glob/Glob.so
7f18af230000-7f18af231000 r--p 00005000 ca:01 280295                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/File/Glob/Glob.so
7f18af231000-7f18af232000 rw-p 00006000 ca:01 280295                     /home/ubuntu/perl5/perlbrew/perls/perl-5.18.2/lib/5.18.2/x86_64-linux/auto/File/Glob/Glob.so
7f18af232000-7f18af3ef000 r-xp 00000000 ca:01 394635                     /lib/x86_64-linux-gnu/libc-2.17.so
7f18af3ef000-7f18af5ef000 ---p 001bd000 ca:01 394635                     /lib/x86_64-linux-gnu/libc-2.17.so
7f18af5ef000-7f18af5f3000 r--p 001bd000 ca:01 394635                     /lib/x86_64-linux-gnu/libc-2.17.so
7f18af5f3000-7f18af5f5000 rw-p 001c1000 ca:01 394635                     /lib/x86_64-linux-gnu/libc-2.17.so
7f18af5f5000-7f18af5fa000 rw-p 00000000 00:00 0
7f18af5fa000-7f18af604000 r-xp 00000000 ca:01 394639                     /lib/x86_64-linux-gnu/libcrypt-2.17.so
7f18af604000-7f18af803000 ---p 0000a000 ca:01 394639                     /lib/x86_64-linux-gnu/libcrypt-2.17.so
7f18af803000-7f18af804000 r--p 00009000 ca:01 394639                     /lib/x86_64-linux-gnu/libcrypt-2.17.so
7f18af804000-7f18af805000 rw-p 0000a000 ca:01 394639                     /lib/x86_64-linux-gnu/libcrypt-2.17.so
7f18af805000-7f18af833000 rw-p 00000000 00:00 0
7f18af833000-7f18af936000 r-xp 00000000 ca:01 394659                     /lib/x86_64-linux-gnu/libm-2.17.so
7f18af936000-7f18afb35000 ---p 00103000 ca:01 394659                     /lib/x86_64-linux-gnu/libm-2.17.so
7f18afb35000-7f18afb36000 r--p 00102000 ca:01 394659                     /lib/x86_64-linux-gnu/libm-2.17.so
7f18afb36000-7f18afb37000 rw-p 00103000 ca:01 394659                     /lib/x86_64-linux-gnu/libm-2.17.so
7f18afb37000-7f18afb3a000 r-xp 00000000 ca:01 394643                     /lib/x86_64-linux-gnu/libdl-2.17.so
7f18afb3a000-7f18afd39000 ---p 00003000 ca:01 394643                     /lib/x86_64-linux-gnu/libdl-2.17.so
7f18afd39000-7f18afd3a000 r--p 00002000 ca:01 394643                     /lib/x86_64-linux-gnu/libdl-2.17.so
7f18afd3a000-7f18afd3b000 rw-p 00003000 ca:01 394643                     /lib/x86_64-linux-gnu/libdl-2.17.so
7f18afd3b000-7f18afd5e000 r-xp 00000000 ca:01 394625                     /lib/x86_64-linux-gnu/ld-2.17.so
7f18afdc8000-7f18aff51000 r--p 00000000 ca:01 28919                      /usr/lib/locale/locale-archive
7f18aff51000-7f18aff55000 rw-p 00000000 00:00 0
7f18aff5a000-7f18aff5d000 rw-p 00000000 00:00 0
7f18aff5d000-7f18aff5e000 r--p 00022000 ca:01 394625                     /lib/x86_64-linux-gnu/ld-2.17.so
7f18aff5e000-7f18aff60000 rw-p 00023000 ca:01 394625                     /lib/x86_64-linux-gnu/ld-2.17.so
7fffb06dd000-7fffb06fe000 rw-p 00000000 00:00 0                          [stack]
7fffb07cf000-7fffb07d1000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

[garfieldnate]

Released 2.38 with the change reverted, and failures still coming in. The problem has to be in the stringification of big numbers done in teh normalize method of AM.xs. We are also dealing with ints, doubles, and long doubles in that method.

[garfieldnate]

I think it would be useful to find out what the original assumption was for the size of a long double. Now the size is 8, 12, or 16 words on different systems.

[garfieldnate]

New information from release of ad1ff887 (2.39-TRIAL). Here's an example fail print:

normalizing grandtotal
--------
14468472
0
0
0
0
0
0
13
SV = PVNV(0x20862c90) at 0x22eba948
  REFCNT = 3
  FLAGS = (PADMY,NOK,POK,IsCOW,pNOK,pPOK)
  IV = 0
  NV = 7.51246017133991e+40
  PV = 0x22ffc060 "340282366920938463463374607431768211455G"\0
  CUR = 40
  LEN = 72
  COW_REFCNT = 1

and here's an example pass print:

normalizing grandtotal
--------
0
0
0
0
0
0
0
13
SV = PVNV(0x2610b30) at 0x1de1308
  REFCNT = 3
  FLAGS = (PADMY,NOK,POK,pNOK,pPOK)
  IV = 0
  NV = 13
  PV = 0x2689c30 "13"\0
  CUR = 2
  LEN = 16

I seems that the grandtotal variable is messed up before the normalize method. On the failing machines, the last element of the long array used to store the big integer is set to a really high number. The number is not consistent across different machines.

grandtotal is set from gangcount which is set from p which is set from the values of the pointers hash.

[garfieldnate]

Next is 60d9fcb9 (2.40-TRIAL).

The value given for p is completely normal even in fail builds:

68
0
0
0
0
0
0
0

This one gets printed in opposite order of the current grandtotal print. so I guess next I'll print gangcount and grandtotal before the normalize method is called.

[garfieldnate]

Next up is 07551214e 2.41-TRIAL. Only the first 6 entries in gangcount are added to and printed in the gangcount 1 loop on line 1125. Then the prints that happen from lines 1142 to 1151 involve copying all 8 entries of gangcount into grandtotal. But gangcount[7] hasn't been initialized (the other 7 entries have carry_replace to obliterate old values). So perhaps this is just a problem with there being uninitialized garbage at the end of the array, and it was just luck that it still worked on most platforms? Time to find out.

[garfieldnate]

Looks like 1a38c19dc did it! 2.42-TRIAL has no more of these errors. Woohoo!

[garfieldnate]

Hang on a sec, though, I might have spoken too soon. My change fixed it, but I'm not sure my diagnosis of the problem was actually correct. When I look at the code now, it looks like gangcount[7] was correctly initialized and I just overwrote that element with a 0 for no good reason. The use of carry_replace means that gangcount[7] is set instead of just being added to like I thought.

However, it really does end up containing garbage on certain machines. See for example the printout here (look at the first printout of "gangcount 1:"). It's like the right-shift operator returns garbage, which would be weird.

I'm going to do another trial release with more print statements (and without that incorrect set to 0).

[garfieldnate]

No more of these weird reports! d8dc460, or 2.45. Woohoo!