search

Perl / perl5

šŸŖ The Perl programming language
https://dev.perl.org/perl5/
Other
1.88k stars 531 forks source link

perl rounds inode in PP stat #15002

Closed p5pRT closed 6 years ago

p5pRT commented 8 years ago

Migrated from rt.perl.org#126414 (status was 'resolved')

Searchable as RT126414$

p5pRT commented 8 years ago

From @bulk88

Created by @bulk88

While I was thinking about implementing real inode info in Win32's PP stat()[1]\, I discovered that Perl rounds the inode integer.

http​://perl5.git.perl.org/perl.git/commitdiff/8d8cba88681398d40004e97bcb93f7f8b88ae05e

associated ticket https://rt.perl.org/Public/Bug/Display.html?id=84590

An inode has 2 uses AFAIK.

-undelete a file -compare hard if they are the same file

An inode is effectively an opaque pointer (integer) into a FS. If the OS's st_ino is 64b (Win32 inode is always 64b per FS driver API) on 32b IV perl\, it has the potential of being rounded if its > 2^53 and therefore is garbage/uninitialized. In a DB on FS scheme\, bad things could happen if 2 files that aren't links in reality\, "==" in perl as to being the same file.

An inode can be implemented in a couple ways by any FS.

-a 0 based offset into an array of something (offsets or structs) on the disk -an absolute sector (units of 512 bytes) or byte position into the FS partition -random looking hash number -XOR against a secret of any of the 3 above

[1] https://rt.perl.org/Public/Bug/Display.html?id=65052 The last 2 implementations would give very frequent high bits that are > 2^53. Worst case scenario for 0 based inode FS is 2^53 bytes\, which is 9007 TB of storage\, which I dont think anyone would use 32 bit perl on a enterprise SAN/cluster\, but that doesn't address the theoretical hash/xor/checksum based inode FSes.

Everyone agrees storing 64 bit C pointers in 64 bit doubles is forbidden\, so why is perl storing inodes in NVs/doubles? Can something be done about this? Fatally error if its over 2^32 or over 2^53? Store the 64 bit integer as a SVPV in printable ASCII and let the user in PP figure out what to do with it (Math​::Int64 it)?

Perl Info ``` Flags: category=core severity=low Site configuration information for perl 5.23.4: Configured by Owner at Fri Oct 16 16:33:21 2015. Summary of my perl5 (revision 5 version 23 subversion 4) configuration: Derived from: c8b87727e0a3605fae65ba4f0811d649667abbf0 Ancestor: 9b331ac6cfda2819646f52e55c4478f2f4123a3d Platform: osname=MSWin32, osvers=6.1, archname=MSWin32-x86-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -GL -DWIN32 -D_CONSOLE -DNO_STRICT -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS', optimize='-O1 -MD -Zi -DNDEBUG -GL', cppflags='-DWIN32' ccversion='18.00.31101', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234, doublekind=3 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8, longdblkind=0 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -ltcg -libpath:"c:\p523\lib\CORE" -machine:x86 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'" -subsystem:console,"5.01"' libpth=\lib libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl523.lib gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -ltcg -libpath:"c:\p523\lib\CORE" -machine:x86 "/manifestdependency:type='Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publicKeyToken='6595b64144ccf1df' language='*'" -subsystem:console,"5.01"' Locally applied patches: uncommitted-changes c8b87727e0a3605fae65ba4f0811d649667abbf0 @INC for perl 5.23.4: C:/p523/site/lib C:/p523/lib . Environment for perl 5.23.4: HOME (unset) LANG (unset) LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=C:\p523\site\bin;C:\p523\bin;C:\Program Files\ActiveState Komodo Edit 9\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\TortoiseGit\bin;C:\Program Files\Microsoft Windows Performance Toolkit\;C:\strawberry\c\bin;C:\strawberry\perl\site\bin;C:\strawberry\perl\bin;C:\Program Files\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\Microsoft SDKs\TypeScript\1.0\; PERL_BADLANG (unset) SHELL (unset) ```
p5pRT commented 8 years ago

From @bulk88

On Tue Oct 20 18​:46​:47 2015\, bulk88 wrote​:

This is a bug report for perl from bulk88@​hotmail.com\, generated with the help of perlbug 1.40 running under perl 5.23.4.

----------------------------------------------------------------- [Please describe your issue here]

While I was thinking about implementing real inode info in Win32's PP stat()[1]\, I discovered that Perl rounds the inode integer.

http​://perl5.git.perl.org/perl.git/commitdiff/8d8cba88681398d40004e97bcb93f7f8b88ae05e

associated ticket https://rt.perl.org/Public/Bug/Display.html?id=84590

An inode has 2 uses AFAIK.

-undelete a file -compare hard if they are the same file

An inode is effectively an opaque pointer (integer) into a FS. If the OS's st_ino is 64b (Win32 inode is always 64b per FS driver API) on 32b IV perl\, it has the potential of being rounded if its > 2^53 and

Actually\, MS bumped the inode size to 128 bits in Server 2012 OS for their nextgen enterpriseish FS. So for discussion\, assume 128 bit inodes are real and have to be supported (personally Win32 Perl can stick with legacy 64 bit inodes for some years).

https://msdn.microsoft.com/en-us/library/windows/desktop/hh802691%28v=vs.85%29.aspx

See also https://msdn.microsoft.com/en-us/library/windows/desktop/aa363788%28v=vs.85%29.aspx

-- bulk88 ~ bulk88 at hotmail.com

p5pRT commented 8 years ago

From @tonycoz

On Tue Oct 20 18​:46​:47 2015\, bulk88 wrote​:

While I was thinking about implementing real inode info in Win32's PP stat()[1]\, I discovered that Perl rounds the inode integer.

http​://perl5.git.perl.org/perl.git/commitdiff/8d8cba88681398d40004e97bcb93f7f8b88ae05e

associated ticket https://rt.perl.org/Public/Bug/Display.html?id=84590

An inode has 2 uses AFAIK.

-undelete a file -compare hard if they are the same file

An inode is effectively an opaque pointer (integer) into a FS. If the OS's st_ino is 64b (Win32 inode is always 64b per FS driver API) on 32b IV perl\, it has the potential of being rounded if its > 2^53 and therefore is garbage/uninitialized. In a DB on FS scheme\, bad things could happen if 2 files that aren't links in reality\, "==" in perl as to being the same file.

An inode can be implemented in a couple ways by any FS.

-a 0 based offset into an array of something (offsets or structs) on the disk -an absolute sector (units of 512 bytes) or byte position into the FS partition -random looking hash number -XOR against a secret of any of the 3 above

[1] https://rt.perl.org/Public/Bug/Display.html?id=65052 The last 2 implementations would give very frequent high bits that are

2^53. Worst case scenario for 0 based inode FS is 2^53 bytes\, which is 9007 TB of storage\, which I dont think anyone would use 32 bit perl on a enterprise SAN/cluster\, but that doesn't address the theoretical hash/xor/checksum based inode FSes.

Everyone agrees storing 64 bit C pointers in 64 bit doubles is forbidden\, so why is perl storing inodes in NVs/doubles? Can something be done about this? Fatally error if its over 2^32 or over 2^53? Store the 64 bit integer as a SVPV in printable ASCII and let the user in PP figure out what to do with it (Math​::Int64 it)?

I can see us producing some sort of error if the inode number changes value when stored as a NV\, perhaps with an option to disable that error\, since stat() isn't only used to fetch a file's inode number.

As to how to behave when the inode number doesn't fit\, we can look at existing implementations\, from the Solaris stat() man page​:

  The stat()\, fstat()\, and lstat() functions may fail if​:

  EOVERFLOW One of the members is too large to store in the   stat structure pointed to by buf.

Similarly on FreeBSD​:

  [EOVERFLOW] The file size in bytes cannot be represented correctly   in the structure pointed to by sb.

If we follow the lead of Solaris/FreeBSD\, perl's stat() would simply fail with EOVERFLOW (if available) if the inode number doesn't fit.

As to optionally disabling the error\, we could use a variable like ${^WIN32_SLOPPY_STAT}\, perhaps ${^SLOPPY_STAT_INO}\, or something lexically scoped\, to keep the sloppy behaviour restricted to a small amount of code.

Sort of related​: systems with large files can run into this issue for st_size​:

$ ./perl -le 'print +(stat "/home/tony/somefile.txt")[7]' 9.00819925474099e+16 $ ls -l ~/somefile.txt -rw-r--r-- 1 tony tony 90081992547409912 Oct 27 03​:43 /home/tony/somefile.txt

(that's a sparse file on ZFS)

Tony

p5pRT commented 8 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 8 years ago

From @ap

* bulk88 \perlbug\-followup@​perl\.org [2015-10-21 03​:50]​:

An inode is effectively an opaque pointer (integer) into a FS.

Like a zip code. That is also nominally a number but is meaningless to do math on\, and the only meaningful operation is testing equality.

Can something be done about this? Fatally error if its over 2^32 or over 2^53? Store the 64 bit integer as a SVPV in printable ASCII and let the user in PP figure out what to do with it (Math​::Int64 it)?

If you mean converting the inode number 9876543210123456 to the string '9876543210123456' (maybe only on overflow)\, then yes I would favour that (along with updates to the docs telling people to compare inode using eq and not ==). That way it would transparently Just Work\, unlike throwing an error. The only question Iā€™m unclear on is what this would do to XS code.

* Tony Cook via RT \perlbug\-followup@​perl\.org [2015-10-27 04​:50]​:

I can see us producing some sort of error if the inode number changes value when stored as a NV\, perhaps with an option to disable that error\, since stat() isn't only used to fetch a file's inode number.

That would force all code not written for a controlled environment to care about this every time it calls stat. Even more irritating\, you have to declare your *dis*interest in the inode number\, which is by far the more common case in my experience. (Most of the time I stat I want the mtime\, next most commonly probably the size. Iā€™ve needed the inode too but only very occasionally.)

If weā€™re going to go this route it would be nice to only force code that actually looks at the inode to care\, e.g. attaching magic to the scalar that dies on access\, so unaffected code doesnā€™t have to care.

Regards\, -- Aristotle Pagaltzis // \<http​://plasmasturm.org/>

p5pRT commented 8 years ago

From @bulk88

On Tue Oct 27 00​:54​:49 2015\, aristotle wrote​:

* bulk88 \perlbug\-followup@&#8203;perl\.org [2015-10-21 03​:50]​:

An inode is effectively an opaque pointer (integer) into a FS.

Like a zip code. That is also nominally a number but is meaningless to do math on\, and the only meaningful operation is testing equality.

Can something be done about this? Fatally error if its over 2^32 or over 2^53? Store the 64 bit integer as a SVPV in printable ASCII and let the user in PP figure out what to do with it (Math​::Int64 it)?

If you mean converting the inode number 9876543210123456 to the string '9876543210123456' (maybe only on overflow)\, then yes I would favour that (along with updates to the docs telling people to compare inode using eq and not ==). That way it would transparently Just Work\, unlike throwing an error. The only question Iā€™m unclear on is what this would do to XS code.

Although I am hesitant about returning PVs and "eq" instead of "==" for perf reasons\, that might be the only sane way to support 128 bit inodes. If an inode number is really a string\, maybe it should be raw 4/8/16 bytes of binary\, not printable ascii. Integer math on it is illegal and makes no sense.

Lets take a 128bit number\, 0x3EB8E79140631092131F93905E8CB80E\, 32 bytes to print in HEX ASCII. 16 bytes in binary. 38 bytes in decimal ASCII.

Since inode comparison is rare\, I was thinking either warn or die if it is > 2^53 (overflows NV). Even if it were raw binary bytes SVPVs are fatter than SVNVs and SVIVs. Someone with ZFS or whatever might be returning >2^53 inodes all the time that makes PP stat unusable with continuous warnings and dies. Another choice is\, if it overflows\, the inode is simply 0\, an invalid (is inode 0 legal per posix?) constant that any human should know something is wrong.

If weā€™re going to go this route it would be nice to only force code that actually looks at the inode to care\, e.g. attaching magic to the scalar that dies on access\, so unaffected code doesnā€™t have to care.

Attaching magic (upgrade to SVMG\, allocate a MG body\, fill it in\, etc) is expensive unless there is an immortable global SV* that is "fetch forbidden" and set up once per perl process.

-- bulk88 ~ bulk88 at hotmail.com

p5pRT commented 8 years ago

From @abigail

On Tue\, Oct 27\, 2015 at 09​:51​:44AM -0700\, bulk88 via RT wrote​:

Since inode comparison is rare\, I was thinking either warn or die if it is > 2^53 (overflows NV). Even if it were raw binary bytes SVPVs are fatter than SVNVs and SVIVs. Someone with ZFS or whatever might be returning >2^53 inodes all the time that makes PP stat unusable with continuous warnings and dies. Another choice is\, if it overflows\, the inode is simply 0\, an invalid (is inode 0 legal per posix?) constant that any human should know something is wrong.

From my hazy memory\, 0 isn't illegal\, but no file system [1] actually uses 0 for a real inode. 0 is used to signal errors\, or for special cases. A directory entry for instance\, may record inode 0 for a deleted file\, and a readdir() will skip over entries with inode 0.

Abigail -- [1] As soon as you say 'no'\, someone will come up with a counter example.

p5pRT commented 8 years ago

From @ap

* bulk88 via RT \perlbug\-followup@&#8203;perl\.org [2015-10-27 17​:55]​:

Although I am hesitant about returning PVs and "eq" instead of "==" for perf reasons\, that might be the only sane way to support 128 bit inodes. If an inode number is really a string\, maybe it should be raw 4/8/16 bytes of binary\, not printable ascii. Integer math on it is illegal and makes no sense.

Lets take a 128bit number\, 0x3EB8E79140631092131F93905E8CB80E\, 32 bytes to print in HEX ASCII. 16 bytes in binary. 38 bytes in decimal ASCII.

Since inode comparison is rare\,

Yes. It is rare. And since eq does its level best to short-circuit\, the difference will only matter when the inode numbers are equal length and share a long prefix. How likely is that really? Also donā€™t forget that ā€œhave I seen this before?ā€ in practice means ā€œlet me look if this inode number is a key in this hash over hereā€ā€¦ which is string-based rather than == anyway.

I proposed conversion to decimal for two reasons​:

1. You can leave non-overflowing inode numbers as IVs\, so the cost is at   least not paid at stat time (except on overflow). That way callers   who donā€™t care about the inode can avoid paying for it. Inasmuch as   there is noteworthy cost\, surely this is the more important factor.

2. Usability basically. Say you want to print the inode number in some   debug output. Or you want to log it. Having to fiddle a conversion   into the code somewhere to get something more readable than raw bytes   is a pain.

Converting to hex would satisfy the usability consideration but unless you are willing to say ā€œsometimes you get a number and sometimes a hex stringā€ then you must *always* do it\, i.e. the usually-IV optimisation becomes impossible and every stat call everywhere gets penalised.

Basically converting overflowing values to decimal strings seems to be as close as we can probably get this interface to ā€œuser cannot use it wrongā€\, and without being relevantly costly. All other options clearly require users to be much smarter and/or more careful\, and to write more code\, which makes this the obviously better option. Unless Iā€™ve missed some clunker of a problem with it; have I?

I was thinking either warn or die if it is > 2^53 (overflows NV). Even if it were raw binary bytes SVPVs are fatter than SVNVs and SVIVs. Someone with ZFS or whatever might be returning >2^53 inodes all the time that makes PP stat unusable with continuous warnings and dies.

Iā€™m not sure I follow here; first you seem to state a preference for warning/dieing\, then pointing out that on some systems that would suck; but\, Iā€™m not sure what you are concludingā€¦?

Another choice is\, if it overflows\, the inode is simply 0\, an invalid (is inode 0 legal per posix?) constant that any human should know something is wrong.

I guess\, butā€¦ that has the same problem as dying​: it makes stat useless on affected systems. Why do that\, when converting to a string will make it easy to write code that works correctly now and forever?

-- Aristotle Pagaltzis // \<http​://plasmasturm.org/>

p5pRT commented 8 years ago

From @bulk88

On Tue Oct 20 18​:46​:47 2015\, bulk88 wrote​:

This is a bug report for perl from bulk88@​hotmail.com\, generated with the help of perlbug 1.40 running under perl 5.23.4.

----------------------------------------------------------------- [Please describe your issue here]

While I was thinking about implementing real inode info in Win32's PP stat()[1]\, I discovered that Perl rounds the inode integer.

Since the code freezes begun/coming\, I wrote a patch with the most conservative solution (instead of "eq" instead of "==" solutions like PVs of hex strings or raw bytes or PVs of decimal integers) so this ticket can be closed. This patch that I attached is part of a near future win32_stat refactor patch that I split up.

-- bulk88 ~ bulk88 at hotmail.com

p5pRT commented 8 years ago

From @bulk88

0001-perl-126414-add-warning-if-perl-rounds-inode-in-PP-s.patch ```diff From e9dda1b0e3c7c0721d983604d685a6362a39f3f6 Mon Sep 17 00:00:00 2001 From: Daniel Dragan Date: Fri, 5 Feb 2016 02:43:36 -0500 Subject: [PATCH] [perl #126414] add warning if perl rounds inode in PP stat Do the most conservative approach to fixing #126414, warn if rounding occurred. Since on some CCs/platforms integer to FP casting is a internally a function call, only do it once. See also #84590 and commit 8d8cba8868 . --- pod/perldiag.pod | 7 +++++++ pp_sys.c | 13 ++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/pod/perldiag.pod b/pod/perldiag.pod index ae061f9..d22dc66 100644 --- a/pod/perldiag.pod +++ b/pod/perldiag.pod @@ -2733,6 +2733,13 @@ transparently promotes all numbers to a floating point representation internally--subject to loss of precision errors in subsequent operations. +=item Integer overflow in stat inode + +(S overflow) Your system's inode type is larger than perl can represent +on your system's archtecture, and this particular inode value returned by +your system's stat was stored by perl as a floating pointer number with a loss +of precision. + =item Integer overflow in srand (S overflow) The number you have passed to srand is too big to fit diff --git a/pp_sys.c b/pp_sys.c index 15b4d8b..a7d99d1 100644 --- a/pp_sys.c +++ b/pp_sys.c @@ -2965,11 +2965,22 @@ PP(pp_stat) RETURN; } if (max) { +#if ST_INO_SIZE > IVSIZE + NV ino; +#endif EXTEND(SP, max); EXTEND_MORTAL(max); mPUSHi(PL_statcache.st_dev); #if ST_INO_SIZE > IVSIZE - mPUSHn(PL_statcache.st_ino); + ino = (NV)PL_statcache.st_ino; + if( +# if ST_INO_SIGN <= 0 + ino < -NV_OVERFLOWS_INTEGERS_AT || +# endif + ino > NV_OVERFLOWS_INTEGERS_AT) + Perl_ck_warner_d(aTHX_ packWARN(WARN_OVERFLOW), + "Integer overflow in stat inode"); + mPUSHn(ino); #else # if ST_INO_SIGN <= 0 mPUSHi(PL_statcache.st_ino); -- 1.9.5.msysgit.1 ```
p5pRT commented 8 years ago

From @bulk88

On Thu Feb 04 23​:49​:50 2016\, bulk88 wrote​:

Since the code freezes begun/coming\, I wrote a patch with the most conservative solution (instead of "eq" instead of "==" solutions like PVs of hex strings or raw bytes or PVs of decimal integers) so this ticket can be closed. This patch that I attached is part of a near future win32_stat refactor patch that I split up.

Actually\, I'm not sure if this patch should be applied before 5.24. In a separate patch\, I added inode support to Win32 Perl\, and it seems that NTFS inode numbers are a 64 bit bitfield/struct\, not linear 64 bit counter\, and therefore always > 2^53 and every PP stat() is generating an overflow warning on my 32 bit IV perl. Not sure how to proceed now.

Example output from a modified perl

Integer overflow in stat inode hi 00320000 lo 000451e6 fp 14073748835815910.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00390000 lo 000451d8 fp 16044073672790488.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 005b0000 lo 00044a00 fp 25614222880950784.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00440000 lo 00044bf3 fp 19140298416606196.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00240000 lo 00044a42 fp 10133099161864770.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00710000 lo 00044bde fp 31806672368585696.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00500000 lo 00044bf6 fp 22517998137134072.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00260000 lo 00045229 fp 10696049115288104.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002c0000 lo 000452c2 fp 12384898975552194.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00450000 lo 000452cc fp 19421773393318604.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00280000 lo 000452ce fp 11258999068709582.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00260000 lo 000452df fp 10696049115288288.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002c0000 lo 000452e1 fp 12384898975552224.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003f0000 lo 0004551f fp 17732923533055264.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 008a0000 lo 00055510 fp 38843546786419984.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00410000 lo 00048484 fp 18295873486488708.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002e0000 lo 00066d48 fp 12947848929111368.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00520000 lo 000488e0 fp 23080948090570976.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00350000 lo 00021c9c fp 14918173765803164.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00440000 lo 00045511 fp 19140298416608528.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003e0000 lo 0004550c fp 17451448556344588.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00460000 lo 00045509 fp 19703248370029832.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00330000 lo 00055a70 fp 14355223812594288.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00330000 lo 00021ca4 fp 14355223812381860.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003c0000 lo 0004522e fp 16888498602922542.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002f0000 lo 00045193 fp 13229323905683860.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002a0000 lo 000451be fp 11821949022130622.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00230000 lo 000451c3 fp 9851624185156036.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00250000 lo 000451c2 fp 10414574138577346.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00260000 lo 000451c8 fp 10696049115288008.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003b0000 lo 00066d1f fp 16607023626349856.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002b0000 lo 0000af09 fp 12103423998603016.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00220000 lo 00045213 fp 9570149208445460.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002b0000 lo 000452d3 fp 12103423998841556.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 0004535b fp 10977524091999068.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00200000 lo 0004549a fp 9007199255024794.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00570000 lo 000554f2 fp 24488322974176496.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00390000 lo 000554f5 fp 16044073672856820.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00420000 lo 000554f4 fp 18577348463252724.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00480000 lo 000554f7 fp 20266198323516664.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00380000 lo 0004579f fp 15762598696081312.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 007b0000 lo 0001e0f2 fp 34621422135533808.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002b0000 lo 00055530 fp 12103423998907696.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00250000 lo 00055533 fp 10414574138643764.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 00055532 fp 10977524092065074.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003f0000 lo 00047dbb fp 17732923533065660.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00230000 lo 00055556 fp 9851624185222486.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00200000 lo 00055558 fp 9007199255090520.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003c0000 lo 00048508 fp 16888498602935560.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00530000 lo 00048871 fp 23362423067281520.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00350000 lo 0004890f fp 14918173765962000.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00310000 lo 000489e0 fp 13792273859119584.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003e0000 lo 00048d3c fp 17451448556358972.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 003e0000 lo 00059702 fp 17451448556427010.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 009a0000 lo 00059846 fp 43347146413807688.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00340000 lo 00059738 fp 14636698789320504.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00b30000 lo 00059a5b fp 50384020831574616.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002f0000 lo 000546a3 fp 13229323905746596.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00ae0000 lo 0005a016 fp 48976645948022808.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00ab0000 lo 0005a105 fp 48132221017891080.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00280000 lo 00063723 fp 11258999068833572.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00210000 lo 00063735 fp 9288674231858996.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 0006372f fp 10977524092122928.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00200000 lo 0004508e fp 9007199255023758.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002c0000 lo 0004507c fp 12384898975551612.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 00066ceb fp 10977524092136684.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00210000 lo 00066cee fp 9288674231872750.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00220000 lo 00066ced fp 9570149208583404.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00200000 lo 00066cf2 fp 9007199255162098.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00310000 lo 000451ea fp 13792273859105258.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00250000 lo 00047ab4 fp 10414574138587828.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00360000 lo 000452b4 fp 15199648742658740.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00250000 lo 00044c8c fp 10414574138576012.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00230000 lo 00044cb1 fp 9851624185154736.00000 0 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 00044cd2 fp 10977524091997394.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 004d0000 lo 00044bff fp 21673573207002112.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00360000 lo 00044cde fp 15199648742657246.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00270000 lo 00044ce4 fp 10977524091997412.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00370000 lo 00044c21 fp 15481123719367712.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 002e0000 lo 00044ce5 fp 12947848928972004.0000 00 at C​:/p523/src/lib/File/Find.pm line 435. Integer overflow in stat inode hi 00230000 lo 00044cf2 fp 9851624185154802.00000 0 at C​:/p523/src/lib/File/Find.pm line 435.

-- bulk88 ~ bulk88 at hotmail.com

p5pRT commented 8 years ago

From @ap

* bulk88 via RT \perlbug\-followup@&#8203;perl\.org [2016-02-10 02​:30]​:

It seems that NTFS inode numbers are a 64 bit bitfield/struct\, not linear 64 bit counter\, and therefore always > 2^53 and every PP stat() is generating an overflow warning on my 32 bit IV perl.

Which implies that the inode field has always been broken. Not bad!

I must wonder how come nobody ran into this before\, then. An impressive feat of ā€œworks by accidentā€ ā€“ but how?

Presumably by way of the rounding being deterministic. Possibly due to the form of the bit struct aligning with the deterministic rounding in such a way that it mitigates the problem in practical scenarios.

Not sure how to proceed now.

Test that hypothesis. Run a recursive walk over your FSs and record the inodes and paths and check to see if you get false duplicates based on comparing inode values. If not\, then the urgency to solve this goes way down and also the importance of solving it non-disruptively goes way up. (Although it still ought to be solved\, either way.)

Regards\, -- Aristotle Pagaltzis // \<http​://plasmasturm.org/>

p5pRT commented 8 years ago

From @bulk88

On Thu Feb 11 15​:55​:50 2016\, aristotle wrote​:

* bulk88 via RT \perlbug\-followup@&#8203;perl\.org [2016-02-10 02​:30]​:

It seems that NTFS inode numbers are a 64 bit bitfield/struct\, not linear 64 bit counter\, and therefore always > 2^53 and every PP stat() is generating an overflow warning on my 32 bit IV perl.

Which implies that the inode field has always been broken. Not bad!

I must wonder how come nobody ran into this before\, then. An impressive feat of ā€œworks by accidentā€ ā€“ but how?

The windows libc ALWAYS sets st_ino to 0. Blead win32 perl's st_ino is always 0 ( there is an old feature request in https​://rt.perl.org/Ticket/Display.html?id=65052 asking to fix that I attempted to fix). I created an unpublished/yet to publish patch that gets the inode ("FileIndex" or "IndexNumber" https://msdn.microsoft.com/en-us/library/windows/hardware/ff540318%28v=vs.85%29.aspx ) from the MS Win32 API\, since the MS libc API doesn't supply it and presents it on a Perl C level to upper Perl C layers. That is how I got inodes visible on a PP level in win32 perl and was able to test this overflow/rounding patch.

Presumably by way of the rounding being deterministic. Possibly due to the form of the bit struct aligning with the deterministic rounding in such a way that it mitigates the problem in practical scenarios.

The chances of a collision seem low\, but not theoretically impossible (only comparing all 64 bits would be collision free). Since Im not a CS PHD\, I wont try to estimate the chance. Any porters can analyze my raw data that i posted and guess the chance. The highest 16 bit short seems to be a bucket or checksum\, while the low 32 bits seem to be an offset into an array. That raw data is probably from iterating over my perl git source code root dir and some of the offsets seem very close.

Not sure how to proceed now.

Test that hypothesis. Run a recursive walk over your FSs and record the inodes and paths and check to see if you get false duplicates based on comparing inode values. If not\, then the urgency to solve this goes way down and also the importance of solving it non-disruptively goes way up. (Although it still ought to be solved\, either way.)

Iterate over the entire FS\, make a hash with the key being the FP and value 1 or ++. At the end of interating over the entire FS (or as much before 32 bit process mem limit hits)\, go over the hash\, if a key has a value >1\, either I have hard links on Win32 or rounding collision. Is this the correct algorithm?

-- bulk88 ~ bulk88 at hotmail.com

p5pRT commented 6 years ago

From zefram@fysh.org

Fixed in commit 2e8ea15a11326145a7df027b5b2507ff3d7483ba. stat() now returns a string of decimal digits if the inode number isn't representable as a native integer.

-zefram

p5pRT commented 6 years ago

From @shlomif

On Sat\, 11 Nov 2017 07​:50​:52 +0000 Zefram \zefram@&#8203;fysh\.org wrote​:

Fixed in commit 2e8ea15a11326145a7df027b5b2507ff3d7483ba. stat() now returns a string of decimal digits if the inode number isn't representable as a native integer.

-zefram

Thanks\, Zefram!

--

Shlomi Fish http​://www.shlomifish.org/ UNIX Fortune Cookies - http​://www.shlomifish.org/humour/fortunes/

XSLT is the number one cause of programmersā€™ suicides since Visual Basic 1.0.   ā€” http​://www.shlomifish.org/humour/bits/facts/XSLT/

Please reply to list if it's a mailing list post - http​://shlom.in/reply .

p5pRT commented 6 years ago

From @dur-randir

On Fri\, 10 Nov 2017 23​:51​:30 -0800\, zefram@​fysh.org wrote​:

Fixed in commit 2e8ea15a11326145a7df027b5b2507ff3d7483ba. stat() now returns a string of decimal digits if the inode number isn't representable as a native integer.

This commit introduced a new build warning on my machine (64 bit Mac OS X)​:

pp_sys.c​:3014​:32​: warning​: comparison of unsigned expression \< 0 is always false [-Wtautological-compare]   neg = PL_statcache.st_ino \< 0;   ~~~~~~~ ^ ~

p5pRT commented 6 years ago

From @jkeenan

On Sat\, 11 Nov 2017 15​:27​:16 GMT\, randir wrote​:

On Fri\, 10 Nov 2017 23​:51​:30 -0800\, zefram@​fysh.org wrote​:

Fixed in commit 2e8ea15a11326145a7df027b5b2507ff3d7483ba. stat() now returns a string of decimal digits if the inode number isn't representable as a native integer.

This commit introduced a new build warning on my machine (64 bit Mac OS X)​:

pp_sys.c​:3014​:32​: warning​: comparison of unsigned expression \< 0 is always false [-Wtautological-compare] neg = PL_statcache.st_ino \< 0;

Can you attach the output of 'perl -V' from this build?

Thank you very much.

-- James E Keenan (jkeenan@​cpan.org)

p5pRT commented 6 years ago

From @dur-randir

On Sat\, 11 Nov 2017 08​:22​:22 -0800\, jkeenan wrote​:

Can you attach the output of 'perl -V' from this build?

Summary of my perl5 (revision 5 version 27 subversion 6) configuration​:   Commit id​: 7d65f652cb34981f4cb53a56496f5712f740d496   Platform​:   osname=darwin   osvers=13.4.0   archname=darwin-thread-multi-2level   uname='darwin isengard.local 13.4.0 darwin kernel version 13.4.0​: mon jan 11 18​:17​:34 pst 2016; root​:xnu-2422.115.15~1release_x86_64 x86_64 '   config_args='-de -Dusedevel -DDEBUGGING -Duseithreads'   hint=recommended   useposix=true   d_sigaction=define   useithreads=define   usemultiplicity=define   use64bitint=define   use64bitall=define   uselongdouble=undef   usemymalloc=n   default_inc_excludes_dot=define   bincompat5005=undef   Compiler​:   cc='cc'   ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9 -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -DPERL_USE_SAFE_PUTENV'   optimize='-O3 -g'   cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.9 -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'   ccversion=''   gccversion='4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)'   gccosandvers=''   intsize=4   longsize=8   ptrsize=8   doublesize=8   byteorder=12345678   doublekind=3   d_longlong=define   longlongsize=8   d_longdbl=define   longdblsize=16   longdblkind=3   ivtype='long'   ivsize=8   nvtype='double'   nvsize=8   Off_t='off_t'   lseeksize=8   alignbytes=8   prototype=define   Linker and Libraries​:   ld='cc'   ldflags =' -mmacosx-version-min=10.9 -fstack-protector -L/usr/local/lib'   libpth=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/lib /usr/local/lib /usr/lib   libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc   perllibs=-lpthread -ldl -lm -lutil -lc   libc=   so=dylib   useshrplib=false   libperl=libperl.a   gnulibc_version=''   Dynamic Linking​:   dlsrc=dl_dlopen.xs   dlext=bundle   d_dlsymun=undef   ccdlflags=' '   cccdlflags=' '   lddlflags=' -mmacosx-version-min=10.9 -bundle -undefined dynamic_lookup -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:   Compile-time options​:   DEBUGGING   HAS_TIMES   MULTIPLICITY   PERLIO_LAYERS   PERL_COPY_ON_WRITE   PERL_DONT_CREATE_GVSV   PERL_IMPLICIT_CONTEXT   PERL_MALLOC_WRAP   PERL_OP_PARENT   PERL_PRESERVE_IVUV   PERL_TRACK_MEMPOOL   PERL_USE_DEVEL   PERL_USE_SAFE_PUTENV   USE_64_BIT_ALL   USE_64_BIT_INT   USE_ITHREADS   USE_LARGE_FILES   USE_LOCALE   USE_LOCALE_COLLATE   USE_LOCALE_CTYPE   USE_LOCALE_NUMERIC   USE_LOCALE_TIME   USE_PERLIO   USE_PERL_ATOF   USE_REENTRANT_API   Built under darwin   Compiled at Nov 11 2017 18​:30​:42   %ENV​:   PERLBREW_BASHRC_VERSION="0.78"   PERLBREW_HOME="/Users/dur-randir/.perlbrew"   PERLBREW_MANPATH="/Users/dur-randir/perlbrew/perls/perl-5.22.1/man"   PERLBREW_PATH="/Users/dur-randir/perlbrew/bin​:/Users/dur-randir/perlbrew/perls/perl-5.22.1/bin"   PERLBREW_PERL="perl-5.22.1"   PERLBREW_ROOT="/Users/dur-randir/perlbrew"   PERLBREW_VERSION="0.78"   @​INC​:   lib   /usr/local/lib/perl5/site_perl/5.27.6/darwin-thread-multi-2level   /usr/local/lib/perl5/site_perl/5.27.6   /usr/local/lib/perl5/5.27.6/darwin-thread-multi-2level   /usr/local/lib/perl5/5.27.6

p5pRT commented 6 years ago

From zefram@fysh.org

Sergey Aleynikov via RT wrote​:

This commit introduced a new build warning on my machine (64 bit Mac OS X)​:

Thanks. That should be fixed by commit b8897079199ab3165b21ca3aff78c7479842181a.

-zefram

p5pRT commented 6 years ago

From @dur-randir

On Sat\, 11 Nov 2017 15​:27​:55 -0800\, zefram@​fysh.org wrote​:

Thanks. That should be fixed by commit b8897079199ab3165b21ca3aff78c7479842181a.

Yes\, this fixed it\, thanks.

p5pRT commented 6 years ago

@cpansprout - Status changed from 'open' to 'pending release'

p5pRT commented 6 years ago

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release yesterday of Perl 5.28.0\, this and 185 other issues have been resolved.

Perl 5.28.0 may be downloaded via​: https://metacpan.org/release/XSAWYERX/perl-5.28.0

If you find that the problem persists\, feel free to reopen this ticket.

p5pRT commented 6 years ago

@khwilliamson - Status changed from 'pending release' to 'resolved'