Re: Bleadperl v5.17.9-200-g0e0ab62 breaks MLEHMANN/JSON-XS-2.33.tar.gz

Perl / perl5

🐪 The Perl programming language

https://dev.perl.org/perl5/

Other

1.85k stars 527 forks source link

Re: Bleadperl v5.17.9-200-g0e0ab62 breaks MLEHMANN/JSON-XS-2.33.tar.gz #12864

Closed p5pRT closed 10 years ago

p5pRT commented 11 years ago

Migrated from rt.perl.org#117239 (status was 'resolved')

Searchable as RT117239$

p5pRT commented 11 years ago

From @andk

git bisect

commit 0e0ab62106f892a1b7f00ad117493064bf9d72d1 Author: Yves Orton \demerphq@gmail\.com Date: Sun Mar 17 20:19:09 2013 +0100

Harden hashes against hash seed discovery by randomizing hash iteration

sample fail report

http://www.cpantesters.org/cpan/report/48924bc0-9027-11e2-869e-dc2c3b384401

The test failing:

t/19_incr.t (Wstat: 12288 Tests: 697 Failed: 48) Failed tests: 8\, 11\, 14\, 17\, 20\, 29\, 35\, 38\, 41\, 44\, 53 56\, 68\, 71\, 89\, 95\, 101\, 104\, 113\, 122 137\, 140\, 146\, 152\, 158\, 164\, 167\, 170 176\, 179\, 182\, 185\, 188\, 194\, 197\, 200 203\, 206\, 212\, 233\, 260\, 263\, 266\, 269 275\, 284\, 287\, 290 Non-zero exit status: 48

The line it reports in the diagnostics:

https://metacpan.org/source/MLEHMANN/JSON-XS-2.33/t/19_incr.t#L22

When the test is running twice\, then the reported fails are usually different ones.

perl -V

Summary of my perl5 (revision 5 version 17 subversion 10) configuration: Commit id: a7b39f85d7caac0822fdcd78400e131a95f11148 Platform: osname=linux\, osvers=3.2.0-4-amd64\, archname=x86_64-linux-thread-multi-ld uname='linux k83 3.2.0-4-amd64 #1 smp debian 3.2.39-2 x86_64 gnulinux ' config_args='-Dprefix=/home/src/perl/repoperls/installed-perls/perl/v5.17.9-203-ga7b39f8/a2da -Dmyhostname=k83 -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Duseithreads -Duselongdouble -DDEBUGGING=-g' hint=recommended\, useposix=true\, d_sigaction=define useithreads=define\, usemultiplicity=define useperlio=define\, d_sfio=undef\, uselargefiles=define\, usesocks=undef use64bitint=define\, use64bitall=define\, uselongdouble=define usemymalloc=n\, bincompat5005=undef Compiler: cc='cc'\, ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'\, optimize='-O2 -g'\, cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion=''\, gccversion='4.7.2'\, gccosandvers='' intsize=4\, longsize=8\, ptrsize=8\, doublesize=8\, byteorder=12345678 d_longlong=define\, longlongsize=8\, d_longdbl=define\, longdblsize=16 ivtype='long'\, ivsize=8\, nvtype='long double'\, nvsize=16\, Off_t='off_t'\, lseeksize=8 alignbytes=16\, prototype=define Linker and Libraries: ld='cc'\, ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /usr/lib libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=\, so=so\, useshrplib=false\, libperl=libperl.a gnulibc_version='2.13' Dynamic Linking: dlsrc=dl_dlopen.xs\, dlext=so\, d_dlsymun=undef\, ccdlflags='-Wl\,-E' cccdlflags='-fPIC'\, lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_PRESERVE_IVUV PERL_SAWAMPERSAND PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LONG_DOUBLE USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API Built under linux Compiled at Mar 19 2013 00:37:12 %ENV: PERL5LIB="" PERL5OPT="" PERL5_CPANPLUS_IS_RUNNING="3177" PERL5_CPAN_IS_RUNNING="3177" @INC: /home/src/perl/repoperls/installed-perls/perl/v5.17.9-203-ga7b39f8/a2da/lib/site_perl/5.17.10/x86_64-linux-thread-multi-ld /home/src/perl/repoperls/installed-perls/perl/v5.17.9-203-ga7b39f8/a2da/lib/site_perl/5.17.10 /home/src/perl/repoperls/installed-perls/perl/v5.17.9-203-ga7b39f8/a2da/lib/5.17.10/x86_64-linux-thread-multi-ld /home/src/perl/repoperls/installed-perls/perl/v5.17.9-203-ga7b39f8/a2da/lib/5.17.10 .

-- andreas

p5pRT commented 11 years ago

From @andk

(Andreas J. Koenig) (via RT) \perlbug\-followup@perl\.org writes:

# \<URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=117239 >
Harden hashes against hash seed discovery by randomizing hash iteration

Other candidates that start failing around the same time and should get a closer examination:

JEEN/Acme-CPANAuthors-Korean-0.09.tar.gz SHARYANTO/Data-Schema-0.135.tar.gz SMIRNIOS/DBD-SQLAnywhere-2.08.tar.gz TIMB/DBI-1.623.tar.gz ANDK/Devel-Symdump-2.08.tar.gz OPI/IO-Handle-Record-0.14.tar.gz MAKAMAKA/JSON-PPdev-2.27100.tar.gz JEEN/Lingua-KO-TypoCorrector-0.03.tar.gz JROBINSON/Locale-Object-0.79.tar.gz PSCUST/Parse-FSM-1.06.tar.gz MAGGIEXYZ/PDL-Stats-0.6.2.tar.gz ADAMK/Perl-Squish-1.06.tar.gz SATOH/Plack-Middleware-StaticShared-0.05.tar.gz VOJ/RDF-NS-20130208.tar.gz MWS/ResourcePool-1.0106.tar.gz JSIRACUSA/Rose-HTML-Objects-0.617.tar.gz MKUTTER/SOAP-Lite-0.714.tar.gz MKUTTER/SOAP-Transport-TCP-0.715.tar.gz TADAM/Test-Mock-LWP-Dispatch-0.03.tar.gz MSCHWERN/Time-y2038-20100403.tar.gz JEEN/WebService-Aladdin-0.0706.tar.gz

I have not had time to verify that they all start failing on the same commit. At least for DBI I have evidence that it starts failing on v5.17.9-201-g3078e10. But since randomness is involved this finding may be biased.

I have not verified others due to lack of time.

-- andreas

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 06:41:28AM +0100\, Andreas Koenig \andreas\.koenig\.7os6VVqR@franz\.ak\.mind\.de wrote:

Harden hashes against hash seed discovery by randomizing hash iteration

Thanks andreas for pointing this out\, hi Yves.

I am talking a bit out of my ass here (call it a rant if you wish)\, and can only guess whats going on. But in short: this change is wrong.

- Unless I am mistaken\, this is to avoid a DOS attack. DOS attacks are easy to protect against\, and this won't protect me against basically any form of attack\, so it's useless. Far better methods exist that prevent every form of memory resource and/or cpu time starvation. - I'd say this simply breaks perl\, as perl documented the order is fixed\, at least between runs (I would argue even the earlier fix to randomise between runs is a bug). Sure\, the wording is ambiguous\, but that doesn't make it any better. - Perl hash already are very slow (just write yourself some XS functions to interface to gcc's unordered_hash and watch the speed - despite the overhead of XS). - Taking out the inability to write a proper hash on all the perl developers out there is just wrong - Perl is the runtime\, and should get it right\, if possible.

So please reconsider your choice of making perl worse again.

On a tangent\, the current regime of perl development - making perl worse\, breaking CPAN more and more\, adding lots of incompatible changes while at the same time requiring use feature is in the wrong for quite some time now\, and\, frankly\, the only reason I don't fork perl is simply that I am not prepared to maintain yet another software package. Breaking CPAN packages every 6 months just doesn't cut it for me\, and creates way too much needless work on my side. Not even mentioning the train wrecks of design hoisted on us in recent releases (smart matches\, given/when being lexical when everything else is dynamical in perl\, mandatory use strict backfitted\, yet more version string madness etc. etc. - I know designing these things is not easy\, but if it's hard\, don't make hacks and release them every few months. OR\, hey\, even the regex changes which look nice\, but I have yet to see something that is either faster or more readable when using them to implement something in the real world).

I'd much prefer if the current perl5porters would go back to maintaining perl _5_ properly and taking out their dreams on breaking perl and making it "better" on Perl 6.

But hey\, for what I know I am probably in the minority and other people embrace this\, but I am certainly not alone in these sentiments.

So again\, please reconsider\, and think about either writing a proper hash that doesn't suffer from simple attacks (a bit more speed would be appreciated as well)\, or forget about making ineffective partial fixes for problems that make writing perl harder\, and fix nothing.

Yves\, I know you are not to blame\, at least not alone\, but it's really getting too much for me as a user of perl. I honestly wish back the lazy days of perl 5.8\, which had many bugs\, was broken a lot\, but at least respected Perl as a language and CPAN for its code\, instead of treating both as slightly broken anyways.

-- The choice of a Deliantra\, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / schmorp@schmorp.de -=====/_/_//_/\_\,_/ /_/\_\

p5pRT commented 11 years ago

From @demerphq

On 21 March 2013 09:28\, Marc Lehmann \schmorp@schmorp\.de wrote:

On Thu\, Mar 21\, 2013 at 06:41:28AM +0100\, Andreas Koenig \andreas\.koenig\.7os6VVqR@franz\.ak\.mind\.de wrote:
Harden hashes against hash seed discovery by randomizing hash iteration
Thanks andreas for pointing this out\, hi Yves.

I am talking a bit out of my ass here (call it a rant if you wish)\, and can only guess whats going on. But in short: this change is wrong.

Your opinion is noted. However given you appear to be uninformed as to the details it is not worth much.

- Unless I am mistaken\, this is to avoid a DOS attack. DOS attacks are easy to protect against\, and this won't protect me against basically any form of attack\, so it's useless. Far better methods exist that prevent every form of memory resource and/or cpu time starvation.

You are mistaken.

- I'd say this simply breaks perl\, as perl documented the order is fixed\, at least between runs (I would argue even the earlier fix to randomise between runs is a bug). Sure\, the wording is ambiguous\, but that doesn't make it any better.

I am unaware of any such documentation. Please state your sources.

- Perl hash already are very slow (just write yourself some XS functions to interface to gcc's unordered_hash and watch the speed - despite the overhead of XS).

Yes\, and this is a very tiny drop in a large bucket. If you think that the changes I made are going to affect hash performance in any kind of significant way then please provide evidence.

- Taking out the inability to write a proper hash on all the perl developers out there is just wrong - Perl is the runtime\, and should get it right\, if possible.

This doesn't make grammatical sense\, so I don't know what you are trying to say.

So please reconsider your choice of making perl worse again.

I don't think I have made Perl worse\, so currently I have nothing to reconsider.

On a tangent\, the current regime of perl development - making perl worse\, breaking CPAN more and more\, adding lots of incompatible changes while at the same time requiring use feature is in the wrong for quite some time now\, and\, frankly\, the only reason I don't fork perl is simply that I am not prepared to maintain yet another software package. Breaking CPAN packages every 6 months just doesn't cut it for me\, and creates way too much needless work on my side. Not even mentioning the train wrecks of design hoisted on us in recent releases (smart matches\, given/when being lexical when everything else is dynamical in perl\, mandatory use strict backfitted\, yet more version string madness etc. etc. - I know designing these things is not easy\, but if it's hard\, don't make hacks and release them every few months. OR\, hey\, even the regex changes which look nice\, but I have yet to see something that is either faster or more readable when using them to implement something in the real world).

I'd much prefer if the current perl5porters would go back to maintaining perl _5_ properly and taking out their dreams on breaking perl and making it "better" on Perl 6.

But hey\, for what I know I am probably in the minority and other people embrace this\, but I am certainly not alone in these sentiments.

So again\, please reconsider\, and think about either writing a proper hash that doesn't suffer from simple attacks (a bit more speed would be appreciated as well)\, or forget about making ineffective partial fixes for problems that make writing perl harder\, and fix nothing.

Yves\, I know you are not to blame\, at least not alone\, but it's really getting too much for me as a user of perl. I honestly wish back the lazy days of perl 5.8\, which had many bugs\, was broken a lot\, but at least respected Perl as a language and CPAN for its code\, instead of treating both as slightly broken anyways.

Thanks for your feedback.

I am sure the perl5porters who are regular contributors will give your opinions on how they use their time the full attention and consideration that they deserve.

Yves

-- perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT commented 11 years ago

From @lizmat

On Mar 21\, 2013\, at 8:33 AM\, Andreas Koenig \andreas\.koenig\.7os6VVqR@franz\.ak\.mind\.de wrote:

(Andreas J. Koenig) (via RT) \perlbug\-followup@perl\.org writes:

# \<URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=117239 >

Harden hashes against hash seed discovery by randomizing hash iteration

Other candidates that start failing around the same time and should get a closer examination:

JEEN/Acme-CPANAuthors-Korean-0.09.tar.gz SHARYANTO/Data-Schema-0.135.tar.gz SMIRNIOS/DBD-SQLAnywhere-2.08.tar.gz TIMB/DBI-1.623.tar.gz ANDK/Devel-Symdump-2.08.tar.gz OPI/IO-Handle-Record-0.14.tar.gz MAKAMAKA/JSON-PPdev-2.27100.tar.gz JEEN/Lingua-KO-TypoCorrector-0.03.tar.gz JROBINSON/Locale-Object-0.79.tar.gz PSCUST/Parse-FSM-1.06.tar.gz MAGGIEXYZ/PDL-Stats-0.6.2.tar.gz ADAMK/Perl-Squish-1.06.tar.gz SATOH/Plack-Middleware-StaticShared-0.05.tar.gz VOJ/RDF-NS-20130208.tar.gz MWS/ResourcePool-1.0106.tar.gz JSIRACUSA/Rose-HTML-Objects-0.617.tar.gz MKUTTER/SOAP-Lite-0.714.tar.gz MKUTTER/SOAP-Transport-TCP-0.715.tar.gz TADAM/Test-Mock-LWP-Dispatch-0.03.tar.gz MSCHWERN/Time-y2038-20100403.tar.gz JEEN/WebService-Aladdin-0.0706.tar.gz

I have not had time to verify that they all start failing on the same commit. At least for DBI I have evidence that it starts failing on v5.17.9-201-g3078e10. But since randomness is involved this finding may be biased.

I have not verified others due to lack of time.

So\, to get this straight:

my @key= keys %hash; my @value= values %hash;

With this change\, there is no guarantee anymore that $key[N] => $value[N] (in the same process) ?

I'm not 100% sure\, but FWIW I think I have used this property of keys() and values() in production code in the past.

Liz

p5pRT commented 11 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 11 years ago

From @demerphq

On 21 March 2013 10:15\, Elizabeth Mattijsen \liz@dijkmat\.nl wrote:

On Mar 21\, 2013\, at 8:33 AM\, Andreas Koenig \andreas\.koenig\.7os6VVqR@franz\.ak\.mind\.de wrote:

(Andreas J. Koenig) (via RT) \perlbug\-followup@perl\.org writes:

# \<URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=117239 >

Harden hashes against hash seed discovery by randomizing hash iteration

Other candidates that start failing around the same time and should get a closer examination:

JEEN/Acme-CPANAuthors-Korean-0.09.tar.gz SHARYANTO/Data-Schema-0.135.tar.gz SMIRNIOS/DBD-SQLAnywhere-2.08.tar.gz TIMB/DBI-1.623.tar.gz ANDK/Devel-Symdump-2.08.tar.gz OPI/IO-Handle-Record-0.14.tar.gz MAKAMAKA/JSON-PPdev-2.27100.tar.gz JEEN/Lingua-KO-TypoCorrector-0.03.tar.gz JROBINSON/Locale-Object-0.79.tar.gz PSCUST/Parse-FSM-1.06.tar.gz MAGGIEXYZ/PDL-Stats-0.6.2.tar.gz ADAMK/Perl-Squish-1.06.tar.gz SATOH/Plack-Middleware-StaticShared-0.05.tar.gz VOJ/RDF-NS-20130208.tar.gz MWS/ResourcePool-1.0106.tar.gz JSIRACUSA/Rose-HTML-Objects-0.617.tar.gz MKUTTER/SOAP-Lite-0.714.tar.gz MKUTTER/SOAP-Transport-TCP-0.715.tar.gz TADAM/Test-Mock-LWP-Dispatch-0.03.tar.gz MSCHWERN/Time-y2038-20100403.tar.gz JEEN/WebService-Aladdin-0.0706.tar.gz

I have not had time to verify that they all start failing on the same commit. At least for DBI I have evidence that it starts failing on v5.17.9-201-g3078e10. But since randomness is involved this finding may be biased.

I have not verified others due to lack of time.

So\, to get this straight:

my @key= keys %hash; my @value= values %hash;

With this change\, there is no guarantee anymore that $key[N] => $value[N] (in the same process) ?

Hi Liz! This hasn't changed.

I'm not 100% sure\, but FWIW I think I have used this property of keys() and values() in production code in the past.

Yep\, you sure have\, we all have I should think\, at some point or another. :-)

And the behavior has not changed. Any code that is currently correct will continue to work as before.

What has changed is that in earlier perls you could do something like this:

my @keys= keys %hash; my %copy= %hash; my @values= %copy;

and most of the time (but not always) get away with assuming that $keys[$n] corresponds to $values[$n].

In 5.18 you are virtually guaranteed that $keys[$n] will NOT correspond to $values[$n].

So all that has happened is that buggy code is now going to demonstrate the bugs pretty much always instead of very very rarely.

cheers Yves ps: Hope you and Wendy are well!

-- perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT commented 11 years ago

From @demerphq

On 21 March 2013 10:25\, demerphq \demerphq@gmail\.com wrote:

my @keys= keys %hash; my %copy= %hash; my @values= %copy;

That should have been

my @values= values %copy;

Sorry. Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT commented 11 years ago

From @lizmat

On Mar 21\, 2013\, at 10:26 AM\, demerphq \demerphq@gmail\.com wrote:

On 21 March 2013 10:25\, demerphq \demerphq@gmail\.com wrote:

my @keys= keys %hash; my %copy= %hash; my @values= %copy;

That should have been

my @values= values %copy;

Sorry.

No problem\, I realised that already.

And *phew*. :-) Thanks for the explanation.

Liz

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 10:04:24AM +0100\, demerphq \demerphq@gmail\.com wrote:

Your opinion is noted. However given you appear to be uninformed as to the details it is not worth much.

Then please state where I was wrong\, I am always open to learn the facts.

- Unless I am mistaken\, this is to avoid a DOS attack. DOS attacks are easy to protect against\, and this won't protect me against basically any form of attack\, so it's useless. Far better methods exist that prevent every form of memory resource and/or cpu time starvation.

You are mistaken.

You are really claiming that this is one of the best methods to prevent memory resource and/or cpu time starvation? Because that's what the satteemnt you quoted says.

Or maybe you meant I am mistaken in something else? Without telling me what it is you are refering to it is hard to tell.

- I'd say this simply breaks perl\, as perl documented the order is fixed\, at least between runs (I would argue even the earlier fix to randomise between runs is a bug). Sure\, the wording is ambiguous\, but that doesn't make it any better.

I am unaware of any such documentation. Please state your sources.

The perlfunc 5.16 documentation on "keys". Older versions are even more explicit.

all them are ambiguous\, but I already mentioned that.

- Perl hash already are very slow (just write yourself some XS functions to interface to gcc's unordered_hash and watch the speed - despite the overhead of XS).

Yes\, and this is a very tiny drop in a large bucket. If you think that the changes I made are going to affect hash performance in any kind of significant way then please provide evidence.

I don't think that\, and haven't made that claim. I think you can make faster hashes with better properties without the drawbacks.

- Taking out the inability to write a proper hash on all the perl developers out there is just wrong - Perl is the runtime\, and should get it right\, if possible.

This doesn't make grammatical sense\,

I think it's a perfectly fine english sentence. Maybe a bit complicated for you to understand though.

so I don't know what you are trying to say.

I am trying to say that the solution for bad hash tables is better hash tables\, or a sense of proportion\, not breaking perfectly fine existing code.

So please reconsider your choice of making perl worse again.

I don't think I have made Perl worse\, so currently I have nothing to reconsider.

To me\, breaking existing code for questionable or no reason is making it worse. It is obvious that you disagree and think you didn't make it worse\, but at best\, you have to disagree with my perfectly valid opinion on this topic-

I am sure the perl5porters who are regular contributors will give your opinions on how they use their time the full attention and consideration that they deserve.

I am fully aware of them being volunteers\, but so am I (and I am a regular contributor as well)\, please keep that in mind.

The perl5porters are not "better" than me as a major contributor to perl in any way\, and have no higher moral ground. Acting uppity because you are part of a mailinglist and contribute more often than me is proving nothing really.

I can't "the people who maintainer perl5" (as opposed to perl5porters) to do what I want\, and neither do I try\, but sarcaasm and ignorance on your part is wrong too. My concerns are valid\, and while you have all the right in the world to ignore them\, or even ridicule them\, that doesn't make them invalid.

p5pRT commented 11 years ago

From @demerphq

On 21 March 2013 11:02\, Marc Lehmann \schmorp@schmorp\.de wrote:

On Thu\, Mar 21\, 2013 at 10:04:24AM +0100\, demerphq \demerphq@gmail\.com wrote:

Your opinion is noted. However given you appear to be uninformed as to the details it is not worth much.

Then please state where I was wrong\, I am always open to learn the facts.

- Unless I am mistaken\, this is to avoid a DOS attack. DOS attacks are easy to protect against\, and this won't protect me against basically any form of attack\, so it's useless. Far better methods exist that prevent every form of memory resource and/or cpu time starvation.

You are mistaken.

You are really claiming that this is one of the best methods to prevent memory resource and/or cpu time starvation? Because that's what the satteemnt you quoted says.

I am sorry\, I seem to be missing some context here. What statement did I quote? I just checked this thread I did not quote anything.

The purpose of hash iterator randomization is make it harder\, and hopefully outright impossible to discover the hash seed a given process is using.

Once an attacker can determine the hash seed they can launch algorithmic complexity attacks on our hash implementation. However the iterator randomization does not in any way prevent a DOS\, it prevents obtaining the information that makes a DOS possible.

Or maybe you meant I am mistaken in something else? Without telling me what it is you are refering to it is hard to tell.

- I'd say this simply breaks perl\, as perl documented the order is fixed\, at least between runs (I would argue even the earlier fix to randomise between runs is a bug). Sure\, the wording is ambiguous\, but that doesn't make it any better.

I am unaware of any such documentation. Please state your sources.

The perlfunc 5.16 documentation on "keys". Older versions are even more explicit.

all them are ambiguous\, but I already mentioned that.

Well\, I beg to differ. I think you are conflating "does not say anything on the subject" and "says something ambiguous".

The documentation for keys()\, and as far as I know all other documentation relating to hashes says nothing about consistency\, either cross process or in between hashes. It says that the keys are returned in an apparently random order\, that the random order is subject to change in future versions of perl\, and the only guarantee of any form provided is that the keys() function will return items in the same order that each() and values would return them assuming the hash has not been modified (a secondary clause in the documentation for "each" specifies it is safe to delete the key that was just returned by each()).

Furthermore I cannot find any evidence that the wording for this has changed in terms of specificity in the lifetime of perlfunc.pod:

In bleadperl it says:

aeedbbed (Nicholas Clark 2007-12-21 17:58:03 +0000 3109) The keys of a hash are returned in an apparently random order. The actual 3b10bc60 (brian d foy 2010-01-13 16:19:33 +0100 3110) random order is subject to change in future versions of Perl\, but it 504f80c1 (Jarkko Hietaniemi 2003-06-26 05:32:02 +0000 3111) is guaranteed to be the same order as either the C\ or C\ 4546b9e6 (Jarkko Hietaniemi 2003-07-27 20:21:40 +0000 3112) function produces (given that the hash has not been modified). Since c5f61d2f (Peter J. Holzer 2010-12-04 11:40:23 -0800 3113) Perl 5.8.1 the ordering can be different even between different runs of 4546b9e6 (Jarkko Hietaniemi 2003-07-27 20:21:40 +0000 3114) Perl for security reasons (see L\<perlsec/"Algorithmic Complexity d6df3700 (Ronald J. Kimball 2003-07-02 07:43:05 -0400 3115) Attacks">).

In Perl 5.14.2 it says this:

The keys of a hash are returned in an apparently random order. The actual random order is subject to change in future versions of Perl\, but it is guaranteed to be the same order as either the "values" or "each" function produces (given that the hash has not been modified). Since Perl 5.8.1 the ordering is different even between different runs of Perl for security reasons (see "Algorithmic Complexity Attacks" in perlsec).

In perl 5.8.5 it says this:

The keys are returned in an apparently random order. The actual random order is subject to change in future versions of perl\, but it is guaranteed to be the same order as either the "val- ues" or "each" function produces (given that the hash has not been modified). Since Perl 5.8.1 the ordering is different even between different runs of Perl for security reasons (see "Algo- rithmic Complexity Attacks" in perlsec).

Here is what it said in 1998/99:

19799a22 (Gurusamy Sarathy 1999-05-24 07:24:11 +0000 2314) Returns a list consisting of all the keys of the named hash. (In 1d2dff63 (Gurusamy Sarathy 1998-05-29 02:31:44 +0000 2315) scalar context\, returns the number of keys.) The keys are returned in ab192400 (Gurusamy Sarathy 1998-11-28 09:36:40 +0000 2316) an apparently random order. The actual random order is subject to ab192400 (Gurusamy Sarathy 1998-11-28 09:36:40 +0000 2317) change in future versions of perl\, but it is guaranteed to be the same 19799a22 (Gurusamy Sarathy 1999-05-24 07:24:11 +0000 2318) order as either the C\ or C\ function produces (given ab192400 (Gurusamy Sarathy 1998-11-28 09:36:40 +0000 2319) that the hash has not been modified). As a side effect\, it resets ab192400 (Gurusamy Sarathy 1998-11-28 09:36:40 +0000 2320) HASH's iterator.

And in 97:

aa689395 (Perl 5 Porters 1997-02-22 04:41:00 +1200 1710) Returns a normal array consisting of all the keys of the named hash. (In aa689395 (Perl 5 Porters 1997-02-22 04:41:00 +1200 1711) a scalar context\, returns the number of keys.) The keys are returned in aa689395 (Perl 5 Porters 1997-02-22 04:41:00 +1200 1712) an apparently random order\, but it is the same order as either the aa689395 (Perl 5 Porters 1997-02-22 04:41:00 +1200 1713) values() or each() function produces (given that the hash has not been aa689395 (Perl 5 Porters 1997-02-22 04:41:00 +1200 1714) modified). As a side effect\, it resets HASH's iterator.

And in 94 (5.000)

a0d0e21e (Larry Wall 1994-10-17 23:00:00 +0000 1578) Returns a normal array consisting of all the keys of the named a0d0e21e (Larry Wall 1994-10-17 23:00:00 +0000 1579) associative array. (In a scalar context\, returns the number of keys.) a0d0e21e (Larry Wall 1994-10-17 23:00:00 +0000 1580) The keys are returned in an apparently random order\, but it is the same a0d0e21e (Larry Wall 1994-10-17 23:00:00 +0000 1581) order as either the values() or each() function produces (given that a0d0e21e (Larry Wall 1994-10-17 23:00:00 +0000 1582) the associative array has not been modified). Here is yet another way a0d0e21e (Larry Wall 1994-10-17 23:00:00 +0000 1583) to print your environment:

perlfunc.pod didnt exist prior to that commit.

I see nothing ambiguous in the documentation\, nor do I see it becoming more or less specific over time.

- Perl hash already are very slow (just write yourself some XS functions to interface to gcc's unordered_hash and watch the speed - despite the overhead of XS).

Yes\, and this is a very tiny drop in a large bucket. If you think that the changes I made are going to affect hash performance in any kind of significant way then please provide evidence.

I don't think that\, and haven't made that claim. I think you can make faster hashes with better properties without the drawbacks.

So then why did you bring it up in this thread? If you have issues with hash performance and think you can do better then please start a new thread\, preferably by posting patches.

- Taking out the inability to write a proper hash on all the perl developers out there is just wrong - Perl is the runtime\, and should get it right\, if possible.

This doesn't make grammatical sense\,

I think it's a perfectly fine english sentence. Maybe a bit complicated for you to understand though.

I guess you meant "your" when you said "the".

If so then I will just say that I find this comment rude and uniformed. I am not responsible for our current hash implementation\, nor do I think you could do any better. I am pretty sure that the only way one can improve perls hash performance is to make it do less\, and that by doing so you would upset far more people than you would make happy by making it a nanosecond faster.

But please\, go ahead and prove me wrong. Post some patches.

so I don't know what you are trying to say.

I am trying to say that the solution for bad hash tables is better hash tables\, or a sense of proportion\, not breaking perfectly fine existing code.

I find it really hard to take you seriously given how often you make silly statements like "not breaking perfectly fine existing code". If your code broke because of this change it is because you were making unwarranted assumptions about how perl's hash function operated\, and as such your code wasn't perfectly fine. In fact if your code is breaking due to this change then I could trivially contrive code in an older version of perl that would also break your assumptions.

So please reconsider your choice of making perl worse again.

I don't think I have made Perl worse\, so currently I have nothing to reconsider.

To me\, breaking existing code for questionable or no reason is making it worse. It is obvious that you disagree and think you didn't make it worse\, but at best\, you have to disagree with my perfectly valid opinion on this topic-

I have to disagree? What? Did you mean "agree"?

Anyway\, it doesnt matter. I don't break existing code for no reason. A) I have reasons\, B) any code broken was already broken but the author did not know it C) there is a reason the docs say "The actual random order is subject to change in future versions of perl" and make no other guarantees that the limited ones I enumerated above. The reason of course is to allow the hash function to be changed or improved over time.

I am sure the perl5porters who are regular contributors will give your opinions on how they use their time the full attention and consideration that they deserve.

I am fully aware of them being volunteers\, but so am I (and I am a regular contributor as well)\, please keep that in mind.

Perhaps you are\, I never advanced an opinion on the subject.

The perl5porters are not "better" than me as a major contributor to perl in any way\, and have no higher moral ground. Acting uppity because you are part of a mailinglist and contribute more often than me is proving nothing really.

I never made any comments about anyones individuals merits. Nor do I know what I did that was "uppity".

I can't "the people who maintainer perl5" (as opposed to perl5porters) to do what I want\, and neither do I try\, but sarcaasm and ignorance on your part is wrong too. My concerns are valid\, and while you have all the right in the world to ignore them\, or even ridicule them\, that doesn't make them invalid.

Well I have responded to the only cogent concern that I have seen you express\, which is that you think that these changes contradict the documentation. The documentation does not support you in this regard.

Is there some other matter I should respond to?

Oh\, I suppose I forgot. Afaik\, you don't even have to fix this\, just take the patches I did to the JSON::PP tests and apply them.

See: https://rt.cpan.org/Public/Bug/Display.html?id=83421

Thanks for your feedback.

Yves

-- perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 12:28:54PM +0100\, demerphq \demerphq@gmail\.com wrote: +-> > >> > - Unless I am mistaken\, this is to avoid a DOS attack. DOS attacks are easy +-> > >> > to protect against\, and this won't protect me against basically any form +-> > >> > of attack\, so it's useless. Far better methods exist that prevent every +-> > >> > form of memory resource and/or cpu time starvation.	> >>	> >> You are mistaken.	> >	> > You are really claiming that this is one of the best methods to prevent	> > memory resource and/or cpu time starvation? Because that's what the satteemnt	> > you quoted says.	>	> I am sorry\, I seem to be missing some context here. What statement did	> I quote? I just checked this thread I did not quote anything.

+-- These ones (hope you can read my ascii art).

I interpreted it to mean to refer to the last one you quoted.

Now\, if you didn't quote anything\, what would your statement refer to?

The purpose of hash iterator randomization is make it harder\, and hopefully outright impossible to discover the hash seed a given process is using.

That in itself is obviously a no-goal. No\, I don't believe that is the real purpose. It must be something else\, like: "to make it harder to use DOS attacks exploiting knowledge about hash collisions".

Or something like that.

In case you wonder why it is a no-goal\, I can discover these hash seeds with a debugger\, or an XS module\, and I doubt your change will stop me from doing it (to the extent this still exists).

Once an attacker can determine the hash seed they can launch algorithmic complexity attacks on our hash implementation. However the iterator randomization does not in any way prevent a DOS\, it prevents obtaining the information that makes a DOS possible.

I think you are splitting hairs\, but you get it wrong.

The purpose of your change is to make it harder to apply such DOS attacks. Saying that's not the purpose\, but the purpose is just to hide this information is dishonest\, because hiding this information isn't a useful purpose in itself.

So it seems I was completely right after all.

Thank you for spelling it out for me to verify.

The perlfunc 5.16 documentation on "keys". Older versions are even more explicit.

all them are ambiguous\, but I already mentioned that.

Well\, I beg to differ. I think you are conflating "does not say anything on the subject" and "says something ambiguous".

I think it strongly implies it. Beat me if you wish\, but your interpretation is as good as mine\, regardless of the intent behind it.

The documentation for keys()\, and as far as I know all other documentation relating to hashes says nothing about consistency\, either cross process or in between hashes.

Sorry\, but "guaranteed to be the same as either the keys ..." clearly implies some consistency\, at least between those. Also saying the order is the same for the "same hash" is ambiguous\, as "same" has tow meanings\, namely\, "identical" (of same identity) and "similar in some kind" (e.g. same contents).

It says that the keys are returned in an apparently random order\, that the random order is subject to change in future versions of perl\,

No problem with that.

the only guarantee of any form provided is that the keys() function will return items in the same order that each() and values would return them assuming the hash has not been modified (a secondary clause in the documentation for "each" specifies it is safe to delete the key that was just returned by each()).

It also says the ordering is the same for the same hash.

I know how it's meant\, but I also know what is said\, and assuming that "the same hash will result in the same ordering" is not unreasonable.

Note that the earlier randomisation patch already broke the semantics\, as the ordering changed even within the same version (but different program runs).

Furthermore I cannot find any evidence that the wording for this has changed in terms of specificity in the lifetime of perlfunc.pod:

Saying it changes in future versions but is the same for the same hash STRONGLY implies that the ordering is the same for the same version of perl\, which is already not true.

I give you\, as I said\, that the wording is ambiguous\, and that it is actually an older change that broke it\, but this interprettaion is entirely valid\, and I am not the only one who used it.

perlfunc.pod didnt exist prior to that commit.

And it clearly changed\, if you took notice.

I see nothing ambiguous in the documentation\, nor do I see it becoming more or less specific over time.

Saying you don'T se something has no value whatsoever. People tend to chose one interpretation of another.

You'd have to make the stronger claim that no other interprettaion is reasonably possible\, and I don't think you can\, simply because "same"\, in english\, is ambiguous as to whether mean identity or equality.

I don't think that\, and haven't made that claim. I think you can make faster hashes with better properties without the drawbacks.

So then why did you bring it up in this thread? If you have issues with hash performance and think you can do better then please start a new thread\, preferably by posting patches.

I don't think that\, and haven't made that claim. I think you can make ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I said it before\, and Is aid it again: I didn't bring up what you put into my mouth\, and I explain it again what I meant\, and this time\, please either believe what I write\, or argue against it\, but please don't attack strawmen:

I think one can make faster and better hashes which fix this problem without breaking code.

That means the breakage is not necessary.

- Taking out the inability to write a proper hash on all the perl developers out there is just wrong - Perl is the runtime\, and should get it right\, if possible.

This doesn't make grammatical sense\,

I think it's a perfectly fine english sentence. Maybe a bit complicated for you to understand though.

I guess you meant "your" when you said "the".

No\, I specifically meant "the"\, because I didn't want to say "your".

If so then I will just say that I find this comment rude and uniformed.

Since your assumption is wrong\, your conclusion is irrelevant fprtunately.

I am not responsible for our current hash implementation\,

And I didn't claim so. On purpose\, in fact.

nor do I think you could do any better. I am pretty sure that the only way one can improve perls hash performance is to make it do less\, and that by doing so you would upset far more people than you would make happy by making it a nanosecond faster.

But please\, go ahead and prove me wrong. Post some patches.

I already did some benchmarks with unordered hashes. I wonder what they do less than perl hashes\, specially now that almost all conssitency in ordering is gone\, which is usually the only problem.

I am trying to say that the solution for bad hash tables is better hash tables\, or a sense of proportion\, not breaking perfectly fine existing code.

I find it really hard to take you seriously given how often you make silly statements like "not breaking perfectly fine existing code".

If you want me to respect your opinions on what is fine code\, please respect mine.

your code broke because of this change it is because you were making unwarranted assumptions about how perl's hash function operated\,

I don't know if they are unwarranted\, but at least they were documented\, even if that turns out to be ambiguous wording.

as such your code wasn't perfectly fine. In fact if your code is breaking due to this change then I could trivially contrive code in an older version of perl that would also break your assumptions.

Well\, the code in question (which defines my assumptions) works with older versions of perl\, so I call your bluff.

To me\, breaking existing code for questionable or no reason is making it worse. It is obvious that you disagree and think you didn't make it worse\, but at best\, you have to disagree with my perfectly valid opinion on this topic-

I have to disagree? What? Did you mean "agree"?

No\, I mean "disagree".

You seem to be very combative\, and I guess this is why you make so many simple mistakes: I usually really mean what I write (and even then\, I do make mistakes).

I _really_ mean that you disagree\, and I really mean that you have to disagree with me if you have another opinion.

Anyway\, it doesnt matter. I don't break existing code for no reason.

Sure. I am saying they are bad reasons\, and badly executed\, not that it is without reason.

A) I have reasons\,

Never said anything to the contrary.

B) any code broken was already broken but the author did not know it

Empty claim.

C) there is a reason the docs say "The actual random order is subject to change in future versions of perl"

The actual random ordering changes with each make test run\, too\, but the testsuite still passes. That is cleatrly not the bug here.

and make no other guarantees that the limited ones I enumerated above. The reason of course is to allow the hash function to be changed or improved over time.

It says the ordering is the same for the same hash.

Changing the hash function doesn't break the code. Changing the hash function within the same version of perl\, and within the same process\, is what changes it.

At this point I must actually admit that I didn't even check that this is true\, I only assume it.

So if the ordering is really the same for the same hash\, within the same process (using the same perl)\, then I am indeed totally wrong.

Is that the case?

I am fully aware of them being volunteers\, but so am I (and I am a regular contributor as well)\, please keep that in mind.

Perhaps you are\, I never advanced an opinion on the subject.

Well\, I am. I tell you so. What are your reasons do doubt me so much?

The perl5porters are not "better" than me as a major contributor to perl in any way\, and have no higher moral ground. Acting uppity because you are part of a mailinglist and contribute more often than me is proving nothing really.

I never made any comments about anyones individuals merits. Nor do I know what I did that was "uppity".

I interpreted your comment as irony\, as am pretty confident it was meant like it. If you really honetsly say you meant it as-is\, I will\, however\, believe you and admit my reaction to your last comment was wrong.

Well I have responded to the only cogent concern that I have seen you express\, which is that you think that these changes contradict the documentation. The documentation does not support you in this regard.

Well\, it clearly does. It's your interpretation that doesn't support mine\, and now the code.

Is there some other matter I should respond to?

You don't have to respond at all\, but if you do so\, I personally expect you to use more than empty claims\, and actually use evidence (which you have done\, for the most part).

Oh\, I suppose I forgot. Afaik\, you don't even have to fix this\, just take the patches I did to the JSON::PP tests and apply them.

See: https://rt.cpan.org/Public/Bug/Display.html?id=83421

But the bug is in the perl core. Nothing I apply can fix that\, becasue I can't apply fixes to the perl core.

Besides\, it makes no sense to me to say "you don't have to fix that\, just apply the fix". What you mean is "you don't have to make the work to invent a fix\, it has already been done".

The problem is that I think the code is fine\, and even if not (by sheer power of force by you applying a buggy patch and claiming it's a good and reasonable and corretc patch and my code is broken)\, I think it *should* be correct\, because taking away a useful feature that doesn't (in itself) gain you anything

All I urge you\, again\, is to consider doing the "hardening" either better\, in a compatible way (which I think is possible)\, or consider the trade-off between a rather ineffective patch (that doesn't shield against similar DOS attacks) and breaking code that ought to work.

I don't have any strong hope for that\, and think you will opt for the ignore-and-break-it route\, which is a valid way to proceed from your side of things. I did feel the moral obligation to tell you why it is wrong\, and if you still do it\, you at least do it in an informed way.

p5pRT commented 11 years ago

From @tsee

On 03/21/2013 02:00 PM\, schmorp@schmorp.de (via RT) wrote:

I think one can make faster and better hashes which fix this problem without breaking code.

That means the breakage is not necessary.

Alternative solutions\, with implementation to prove their validity\, would be most welcome. Honestly.

You seem to be very combative\, and I guess this is why you make so many simple mistakes: I usually really mean what I write (and even then\, I do make mistakes).

I _really_ mean that you disagree\, and I really mean that you have to disagree with me if you have another opinion.

Umm\, who is splitting hair now?

You don't have to respond at all\, but if you do so\, I personally expect you to use more than empty claims\, and actually use evidence (which you have done\, for the most part).

So why imply otherwise in the first part of your sentence? Shove your passive aggressiveness\, Marc.

But the bug is in the perl core. Nothing I apply can fix that\, becasue I can't apply fixes to the perl core.

So let's rephrase: If you care for JSON::XS to continue working\, then you can work around what *you* consider a bug in perl. Many others sure don't.

Besides\, it makes no sense to me to say "you don't have to fix that\, just apply the fix". What you mean is "you don't have to make the work to invent a fix\, it has already been done".

Hair! In tiny fragments!

--Steffen

p5pRT commented 11 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 11 years ago

From @ribasushi

On Thu\, Mar 21\, 2013 at 06:00:56AM -0700\, schmorp@schmorp.de wrote:

That in itself is obviously a no-goal. No\, I don't believe that is the real purpose. It must be something else\, like: "to make it harder to use DOS attacks exploiting knowledge about hash collisions".

Or something like that.

In case you wonder why it is a no-goal\, I can discover these hash seeds with a debugger\, or an XS module\, and I doubt your change will stop me from doing it (to the extent this still exists).

Marc\, you actually are misunderstanding here. The purpose of the changes is to make it very hard to determine the hash seed by *remotely* observing how the hash key order behaves. Think long running process which exposes the hash key order of parameters submitted via a web request.

There is no access to the running process itself\, no XS\, no debugging tools. Just "black box" observation. The point of the changes is to make such type of observations futile\, nothing more.

Chers

p5pRT commented 11 years ago

From @Leont

On Thu\, Mar 21\, 2013 at 2:00 PM\, schmorp@schmorp.de \perlbug\-followup@perl\.org wrote:

The purpose of hash iterator randomization is make it harder\, and hopefully outright impossible to discover the hash seed a given process is using.

That in itself is obviously a no-goal. No\, I don't believe that is the real purpose. It must be something else\, like: "to make it harder to use DOS attacks exploiting knowledge about hash collisions".

Or something like that.

In case you wonder why it is a no-goal\, I can discover these hash seeds with a debugger\, or an XS module\, and I doubt your change will stop me from doing it (to the extent this still exists).

Well\, if you can insert an XS module or a debugger you've already got control over the process\, no need for a DOS there.

Sorry\, but "guaranteed to be the same as either the keys ..." clearly implies some consistency\, at least between those. Also saying the order is the same for the "same hash" is ambiguous\, as "same" has tow meanings\, namely\, "identical" (of same identity) and "similar in some kind" (e.g. same contents).

IMO same hash clearly means same identity\, not similarly valued (if only because the latter is rather poorly defined).

Saying it changes in future versions but is the same for the same hash STRONGLY implies that the ordering is the same for the same version of perl\, which is already not true.

It suggests this changed wasn't planned years ahead of time\, nothing more nothing less.

Saying you don'T se something has no value whatsoever. People tend to chose one interpretation of another.

You'd have to make the stronger claim that no other interprettaion is reasonably possible\, and I don't think you can\, simply because "same"\, in english\, is ambiguous as to whether mean identity or equality.

Alternative interpretations are unfortunate and sometimes something to deal with\, but they are not authoritative.

If you want me to respect your opinions on what is fine code\, please respect mine.

perl doesn't agree it's fine.

your code broke because of this change it is because you were making unwarranted assumptions about how perl's hash function operated\,

I don't know if they are unwarranted\, but at least they were documented\, even if that turns out to be ambiguous wording.

as such your code wasn't perfectly fine. In fact if your code is breaking due to this change then I could trivially contrive code in an older version of perl that would also break your assumptions.

Well\, the code in question (which defines my assumptions) works with older versions of perl\, so I call your bluff.

You're assuming that similar hashes always return their content in the same order. This is already false in older perls in at least two circumstances:

1) The values don't have the same bucket-size 2) The rehash mechanism has kicked in 3) The hash is tied

Neither case is very likely in most code\, but they can both realistically happen.

But the bug is in the perl core. Nothing I apply can fix that\, becasue I can't apply fixes to the perl core.

Besides\, it makes no sense to me to say "you don't have to fix that\, just apply the fix". What you mean is "you don't have to make the work to invent a fix\, it has already been done".

The problem is that I think the code is fine\, and even if not (by sheer power of force by you applying a buggy patch and claiming it's a good and reasonable and corretc patch and my code is broken)\, I think it *should* be correct\, because taking away a useful feature that doesn't (in itself) gain you anything

All I urge you\, again\, is to consider doing the "hardening" either better\, in a compatible way (which I think is possible)\, or consider the trade-off between a rather ineffective patch (that doesn't shield against similar DOS attacks) and breaking code that ought to work.

I don't have any strong hope for that\, and think you will opt for the ignore-and-break-it route\, which is a valid way to proceed from your side of things. I did feel the moral obligation to tell you why it is wrong\, and if you still do it\, you at least do it in an informed way.

Well\, one of the two has to change\, and it is unlikely to be perl.

Leon

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 06:45:11AM -0700\, Steffen Mueller via RT \perlbug\-followup@perl\.org wrote:

That means the breakage is not necessary.

Alternative solutions\, with implementation to prove their validity\, would be most welcome. Honestly.

Honestly\, I do so much for perl\, and after my previous experiences I don't have the time to argue for ages again to get such a patch through. I am not maintaining perl5\, and I think when one can't do a job properly\, it's likely not worth doing it at all.

You seem to be very combative\, and I guess this is why you make so many simple mistakes: I usually really mean what I write (and even then\, I do make mistakes).

I _really_ mean that you disagree\, and I really mean that you have to disagree with me if you have another opinion.

Umm\, who is splitting hair now?

Obviously not me - would you do me a favour and just read the whole mail\, without picking things out of context and accusing me of splitting hairs?

He made a wrong assumption/accusation\, and all I did was point out that it's not true.

You don't have to respond at all\, but if you do so\, I personally expect you to use more than empty claims\, and actually use evidence (which you have done\, for the most part).

So why imply otherwise in the first part of your sentence?

I am not aware of implying anything otherwise. When it comes to it\, it likely only exists in your mind.

Shove your passive aggressiveness\, Marc.

I honestly don't think I am passive aggressive\, Steffen.

But the fact that you attack me and ignore the actual issue completely tells me you are just trolling here.

I'd take somebody like me\, who actually argues\, any time over somebody like you\, who only insults.

Can you do better? Sure you can.

But the bug is in the perl core. Nothing I apply can fix that\, becasue I can't apply fixes to the perl core.

So let's rephrase: If you care for JSON::XS to continue working\, then you can work around what *you* consider a bug in perl. Many others sure don't.

And many others sure do. Apart from trolling and personal attacks\, do you actually have anything to say on the actual issue? Why are you even responding if you are incapable of contributing anything of value?

Besides\, it makes no sense to me to say "you don't have to fix that\, just apply the fix". What you mean is "you don't have to make the work to invent a fix\, it has already been done".

Hair! In tiny fragments!

That's exactly what I was pointing out. Count yourself a genius for realising it.

In any case\, I won't reply to obvious troll attempts by you anymore.

I have no problem with a bit of drama\, but please\, make it worthwuoe by adding *some* substance.

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 06:54:52AM -0700\, Peter Rabbitson via RT \perlbug\-followup@perl\.org wrote:

Marc\, you actually are misunderstanding here.

No\, I don't.

The purpose of the changes is to make it very hard to determine the hash seed by *remotely* observing how the hash key order behaves.

First\, that's not what yves said (but sure\, he is wrong).

Secondly\, if your ead my mail\, it is painfully obviously thta I know that - I pointed it out myself.

It was yves who disagrees and said it's purpose to to avoid discovering the value\, not me.

Think long running process which exposes the hash key order of parameters submitted via a web request.

I am perfectly aware of that\, thank you very much. Again\, pleae do ourselves a favour and actually read my mail fully\, not in tiny and misleading excerpts.

I even brought up an example of how to exploiut this using a DOS attack originally.

There is no access to the running process itself\, no XS\, no debugging tools. Just "black box" observation. The point of the changes is to make such type of observations futile\, nothing more.

I know. Please read the whole thread before making comments\, thank you.

p5pRT commented 11 years ago

From @demerphq

On 21 March 2013 15:00\, Leon Timmermans \fawaka@gmail\.com wrote:

You're assuming that similar hashes always return their content in the same order. This is already false in older perls in at least two circumstances:

Thats 3. :-)

1) The values don't have the same bucket-size 2) The rehash mechanism has kicked in 3) The hash is tied

Neither case is very likely in most code\, but they can both realistically happen.

Ah but those are the rare examples. The common example is this:

%copy= %hash;

Any items that collide will change order in %copy as compared to hash in every version of perl prior to 3078e109184e83a0c0e99d9eea771d67b90f1e72 (ie a few days ago). With that patch applied there is a very small chance that %copy would have the same key order as %hash. Amazingly tiny small.

This demonstration should work on any perl prior to the new hash randomization:

$ perl -le'my %hash=(13=>1\,19=>1); print join " "\, keys %hash; my %copy= %hash; print join " "\,keys %copy' 19 13 13 19

So in the old days it was not even unlikely that the two "similar" hashes will have different orders\, on the contrary\, simply copying a hash would almost guarantee a different key order.

Yves

-- perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT commented 11 years ago

From @ribasushi

On Thu\, Mar 21\, 2013 at 03:15:27PM +0100\, Marc Lehmann wrote:

There is no access to the running process itself\, no XS\, no debugging tools. Just "black box" observation. The point of the changes is to make such type of observations futile\, nothing more.

I know. Please read the whole thread before making comments\, thank you.

I am confused... If you know there is no access to the running interpreter\, why did you bring up XS/debugging in the first place...?

Not trolling - honest question.

p5pRT commented 11 years ago

From @tux

On Thu\, 21 Mar 2013 15:12:33 +0100\, Marc Lehmann \schmorp@schmorp\.de wrote:

On Thu\, Mar 21\, 2013 at 06:45:11AM -0700\, Steffen Mueller via RT \perlbug\-followup@perl\.org wrote:

That means the breakage is not necessary.

Alternative solutions\, with implementation to prove their validity\, would be most welcome. Honestly.

Honestly\, I do so much for perl\,

Name anything that I cannot live without? Anything I should know about?

$ cat $HOME/.cpan/prefs/MLEHMAN.yml

comment: "use common sense" match: distribution: "^MLEHMAN" disabled: 1 $

Sorry\, you never convinced me your contributions should change that

and after my previous experiences I don't have the time to argue for ages again to get such a patch through.

And what makes you think we *do* have the time to argue?

I am not maintaining perl5\, and I think when one can't do a job properly\, it's likely not worth doing it at all.

It is currently done properly and done well. By (very) competent people. p5p currently manages monthly development releases and on-time production releases that are secure and safe.

-- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.17 porting perl5 on HP-UX\, AIX\, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 07:02:01AM -0700\, Leon Timmermans via RT \perlbug\-followup@perl\.org wrote:

In case you wonder why it is a no-goal\, I can discover these hash seeds with a debugger\, or an XS module\, and I doubt your change will stop me from doing it (to the extent this still exists).

Well\, if you can insert an XS module or a debugger you've already got control over the process\, no need for a DOS there.

Again\, it was yves who brought this up\, not me. I was merely arguing that it's not the purpose of the change\, as do you and others.

IMO same hash clearly means same identity\, not similarly valued (if only because the latter is rather poorly defined).

Your opinion is irrelevant. I am not claiming it's the only valid interpretation\, I am claiming it is a valid interpretation.

Same hahs clearly means "same hash". Ask some people by showing them:

my %a = (a => 1\, b => 2); my %b = (a => 1\, b => 2);

Are you sure that "clearly" %a and %b are not the same? Do you really think that this wording is totally unambiguous.

I don't think so\, but you can surprise me.

Note that you are conveneitnly ignoring the other arguments I made.

Documentation is mostly irrelevant. Even if documentation clearly said it one way or another\, there might be good and valid reasons to break it.

The documentation just explains why relying in this behaviour is not unreasonable. It does not matter what the intended menaing is\, or that there are other interpretations.

I was pointing out that code that relies on this is not broken\, or buggy\, or silly\, or anything else the code was called to be\, but reasonable.

Furthermore\, I claim this is a useful feature\, and breaking it shouldn't be done lightly.

And rantily\, I pointed out that the low regard for bakcwards compatibility and continued breakage is a big drain\, and\, I think\, not necessary.

Saying it changes in future versions but is the same for the same hash STRONGLY implies that the ordering is the same for the same version of perl\, which is already not true.

It suggests this changed wasn't planned years ahead of time\, nothing more nothing less.

The documentation cleaims it's the same for the same hash\, but the ordering might change in future versions.

I think you are talking bullshit semantics here. A normal reader will clearly imply that the algorithm (and ordering) is the same within the same version of perl.

Which it was.

You'd have to make the stronger claim that no other interprettaion is reasonably possible\, and I don't think you can\, simply because "same"\, in english\, is ambiguous as to whether mean identity or equality.

Alternative interpretations are unfortunate and sometimes something to deal with\, but they are not authoritative.

I never claimed they are authoritative. You (as a group) however claimed that it's silly\, wrong\, unreasonable\, invalid whatever.

And that isn't true. It's a reasonable interpretation\, and a useful feature that was implemented for a very long time.

It shouldn't be broken lightly.

If you want me to respect your opinions on what is fine code\, please respect mine.

perl doesn't agree it's fine.

development versions don't\, all released versions do\, by documentation and execution both.

What you mean is that some developer(s) disagree. That is indeed fine\, but definitely not the same thing.

Well\, the code in question (which defines my assumptions) works with older versions of perl\, so I call your bluff.

You're assuming that similar hashes always return their content in the same order.

But I don't\, and haven't said so. I claim the code in question is fine and it is a reasonable tand useful to have it work.

Neither case is very likely in most code\, but they can both realistically happen.

All of these are not relevant to this discussion though.

I don't have any strong hope for that\, and think you will opt for the ignore-and-break-it route\, which is a valid way to proceed from your side of things. I did feel the moral obligation to tell you why it is wrong\, and if you still do it\, you at least do it in an informed way.

Well\, one of the two has to change\,

And why would that be? You hold a gun to my face? You take away maintainership?

and it is unlikely to be perl.

I think so\, too.

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 07:18:13AM -0700\, yves orton via RT \perlbug\-followup@perl\.org wrote:

This demonstration should work on any perl prior to the new hash randomization:

It is also unrelated to the code in question.

So in the old days it was not even unlikely that the two "similar" hashes will have different orders\, on the contrary\, simply copying a hash would almost guarantee a different key order.

A pity\, but not the code (and topic) in question here.

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Fri\, Mar 22\, 2013 at 01:20:40AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote:

There is no access to the running process itself\, no XS\, no debugging tools. Just "black box" observation. The point of the changes is to make such type of observations futile\, nothing more.

I know. Please read the whole thread before making comments\, thank you.

I am confused... If you know there is no access to the running interpreter\, why did you bring up XS/debugging in the first place...?

Because I originally said it's about avoiding DOS attacks (and later refining that to mean cpu/memory starvation)\, and yves said\, no\, it's purpose is to make it hard to discover the seed\, after which I pointed out that this is unlikely the real purpose\, because you can already discover this seed in other ways (for example\, by setting an env variable before starting the program).

Now\, the DOS attack is only an example of an algorithmic complexity attack that this causes\, but my guess is that coming up with something that isn't a DOS attack\, using this technique\, that can't be prohibited by resource limits and monitoring is hard to impossible.

So\, if the difference between the DOS attack I was talking about\, and the algorithmic complexity attack is indeed more than just irrelevant (and I have not seen any indication here)\, then I wonder if the same protections wouldn't help here either.

There are all sorts of attacks one can launch remotely. It is good that perl makes a lot of these hard or impossible\, but perl cannot make all of these hard and impossible.

It is always a trade-off between usefulness of the language and resistance against such an attack.

If such a remote attack would be a significant danger and wouldn't be caught by prudent uses of other mechanisms (resource control...)\, then I might agree that\, regardless of documentation or code\, it is likely good to pay the price of braking code.

But the evidence is lacking\, and this looks more like blind action. I also wonder why the hash itself couldn't be made hardneed - surely there are hashes with (relatively) stable iterating behaviour and collision resistance.

The problem here is likely the over-reliance on power-of-two hashing (which requires good hash algorithms\, which historically has been a problem) on the lack of scalable collision handling.

I hope I answered your question in detail.

Not trolling - honest question.

And surely I had no indications otherwise. I can differentiate between people who contribute nothing except personal attacks and others.

It is unfortunate that Steffen feels he has to poison the atmosphere that way\, but this is clearly Steffen\, and nobody else.

p5pRT commented 11 years ago

From @ribasushi

On Thu\, Mar 21\, 2013 at 04:03:54PM +0100\, Marc Lehmann wrote:

On Fri\, Mar 22\, 2013 at 01:20:40AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote:

There is no access to the running process itself\, no XS\, no debugging tools. Just "black box" observation. The point of the changes is to make such type of observations futile\, nothing more.

I know. Please read the whole thread before making comments\, thank you.

I am confused... If you know there is no access to the running interpreter\, why did you bring up XS/debugging in the first place...?

Because I originally said it's about avoiding DOS attacks (and later refining that to mean cpu/memory starvation)\, and yves said\, no\, it's purpose is to make it hard to discover the seed\, after which I pointed out that this is unlikely the real purpose\, because you can already discover this seed in other ways (for example\, by setting an env variable before starting the program).

Now\, the DOS attack is only an example of an algorithmic complexity attack that this causes\, but my guess is that coming up with something that isn't a DOS attack\, using this technique\, that can't be prohibited by resource limits and monitoring is hard to impossible.

So\, if the difference between the DOS attack I was talking about\, and the algorithmic complexity attack is indeed more than just irrelevant (and I have not seen any indication here)\, then I wonder if the same protections wouldn't help here either.

There are all sorts of attacks one can launch remotely. It is good that perl makes a lot of these hard or impossible\, but perl cannot make all of these hard and impossible.

It is always a trade-off between usefulness of the language and resistance against such an attack.

If such a remote attack would be a significant danger and wouldn't be caught by prudent uses of other mechanisms (resource control...)\, then I might agree that\, regardless of documentation or code\, it is likely good to pay the price of braking code.

But the evidence is lacking\, and this looks more like blind action. I also wonder why the hash itself couldn't be made hardneed - surely there are hashes with (relatively) stable iterating behaviour and collision resistance.

The problem here is likely the over-reliance on power-of-two hashing (which requires good hash algorithms\, which historically has been a problem) on the lack of scalable collision handling.

I hope I answered your question in detail.

You did\, thank you. However I fail to understand how do you find *apparent* hash key order stability for semantically identical [1] hashes a good thing.

That is if a hash order is stable 99.9% of the time\, and then in the last 0.01% all hell breaks loose (because of the many assumptions that the 99.9 == 100) - how is an average programmer supposed to debug that?

Could you elaborate?

[1] Just to make sure I understand - from what I gather this:

`perl -MData::Dumper -e '$Data::Dumper::Indent = 0; print Dumper {(1..20)}; print "\n"'`

producing identical output on all perls between 5.8.2 and 5.16.3 is something you consider a feature. You do not consider it a bug. Am I correct?

Cheers

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 07:24:19AM -0700\, "H. Merijn Brand via RT" \perlbug\-followup@perl\.org wrote:

On Thu\, Mar 21\, 2013 at 06:45:11AM -0700\, Steffen Mueller via RT \perlbug\-followup@perl\.org wrote:

That means the breakage is not necessary.

Alternative solutions\, with implementation to prove their validity\, would be most welcome. Honestly.

Honestly\, I do so much for perl\,

Name anything that I cannot live without?

And what is the relevance for this topic and why would I care?

Anything I should know about?

No\, I honestly don't care if you use my packages.

The reason you stopped using my packages\, as you yourself told me\, is that you wanted to force me to port JSON::XS to some very old vendor compiler that has been superseded\, on a platform that was well-supported by gcc (was it some old HP-UX maybe? I really liked that platform and worked many years on it...).

I said no\, and said it's not unreasonable to ask you to upgrade\, because a working compiler is available for your system.

I have no problem that you don't want to use my module(s) because of that. I have few problems with you threatening me to "warn people" about my software\, and few problems about you doing posts like this one.

Again\, everybody is *fine* to not use my software. This is something that is apparently hard to udnerstand. I don't publish software for self-gratification\, I publish software because I hope it is useful for others\, and it clearly is for a LOT of people - there is no shortage of feedback on that.

But I do consider your behaviour wholly unreasonable\, and if the requirement for you to use my software is that I port it to your pet compiler because you threaten me\, then you just have to live without it.

And the arrogance of trying to threaten me into doing your work for free is breathtaking.

So\, no\, my modules are not for you\, and you clearly don't deserve it - after all\, you are the person suffering most for that decision...

(But I of course have no problems with you using it\, according to their license).

In any case\, folks\, ranting is all fine and good\, but why don't you at last attempt to stay a bit on-topic?

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Fri\, Mar 22\, 2013 at 02:15:53AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote:

I hope I answered your question in detail.

You did\, thank you. However I fail to understand how do you find *apparent* hash key order stability for semantically identical [1] hashes a good thing.

That is if a hash order is stable 99.9% of the time\, and then in the last 0.01% all hell breaks loose (because of the many assumptions that the 99.9 == 100) - how is an average programmer supposed to debug that?

I don't find that usefil\, I find the actual behaviour useful. Old perls arrived at the same order when I built them in the same way each time.

New perls do it only when setting an env variable\, or within one process - fair enough\, not grat\, but managable.

[1] Just to make sure I understand - from what I gather this:

`perl -MData::Dumper -e '$Data::Dumper::Indent = 0; print Dumper {(1..20)}; print "\n"'`

producing identical output on all perls between 5.8.2 and 5.16.3 is

No\, and I made it explicit that I am fine with that (and it's clearly documented\, at leats for me\, since 5.8.1 or so).

I have a problem (or rather\, my module has one) with this code not producing identical output:

use feature 'say';

my %a = (a => 1\, b => 2); my %b = (a => 1\, b => 2);

say keys %a; say keys %b;

Besides\, I didn't know these perls would produce identical output - but they seem to\, wow\, I didn't expect that\, but cool - though I wouldn't rely on that.

You do not consider it a bug.

Are you sure anybody would consider it a bug to get the same output for the same perl snippet on multiple versions of perl?

I would say that's the default expectation\, don't you agree? At least not without good reasons for them to differ.

p5pRT commented 11 years ago

From @tux

On Thu\, 21 Mar 2013 16:17:20 +0100\, Marc Lehmann \schmorp@schmorp\.de wrote:

On Thu\, Mar 21\, 2013 at 07:24:19AM -0700\, "H. Merijn Brand via RT" \perlbug\-followup@perl\.org wrote:

On Thu\, Mar 21\, 2013 at 06:45:11AM -0700\, Steffen Mueller via RT \perlbug\-followup@perl\.org wrote:

That means the breakage is not necessary.

Alternative solutions\, with implementation to prove their validity\, would be most welcome. Honestly.

Honestly\, I do so much for perl\,

Name anything that I cannot live without?

And what is the relevance for this topic and why would I care?

Anything I should know about?

No\, I honestly don't care if you use my packages.

The reason you stopped using my packages\, as you yourself told me\, is that you wanted to force me to port JSON::XS to some very old vendor compiler

No\, I wanted you to conform to C89\, which is what the rest of perl requires. I never expected you to take the time machine to work around failing compilers of the stone-age (xlc on AIX-4 is a good example of a compiler that *clain=ms* to be ANSI compliant but is not)

that has been superseded\, on a platform that was well-supported by gcc (was it some old HP-UX maybe? I really liked that platform and worked many years on it...).

I used HP C-ANSI-C as an *example* to why your code should use ANSI comments instead of c++ style comments. IIRC there were only two or three minor issues that could be fixed easily without breaking anything\, but you refused in blaming me not to know the specs.

I said no\, and said it's not unreasonable to ask you to upgrade\, because a working compiler is available for your system.

I think it is not reasonable to ask to upgrade a compiler that conforms to the minimal requirements for building perl (and still does so up till today: I am about to release perl5.16.3 on that box soon). Especially when there are no updates available. Of course one could install another compiler\, like GNU gcc\, but remember that on that architecture\, it comes with quite a high performance penalty compared to building with the native compiler. I don't think an overall performance hit *ever* warrants the use of // comments in C/XS.

I have no problem that you don't want to use my module(s) because of that. I have few problems with you threatening me to "warn people" about my software\, and few problems about you doing posts like this one.

Again\, everybody is *fine* to not use my software. This is something that is apparently hard to udnerstand. I don't publish software for self-gratification\, I publish software because I hope it is useful for others\, and it clearly is for a LOT of people - there is no shortage of feedback on that.

Then how hard would it be to conform to C89 and let the rest of the people that enjoy perl in?

But I do consider your behaviour wholly unreasonable\, and if the requirement for you to use my software is that I port it to your pet compiler because you threaten me\, then you just have to live without it.

When did I ever threaten you?

And the arrogance of trying to threaten me into doing your work for free is breathtaking.

Hrmph. IIRC my quest came with a (simple) patch. there was NO work for you involved AT ALL.

So\, no\, my modules are not for you\, and you clearly don't deserve it - after all\, you are the person suffering most for that decision...

I'm not suffering at all. Moreover\, I think I am quite amused by the whole situation.

(But I of course have no problems with you using it\, according to their license).

In any case\, folks\, ranting is all fine and good\, but why don't you at last attempt to stay a bit on-topic?

I think I am\, but you seem to misunderstand a lot of other people too.

p5pRT commented 11 years ago

From @demerphq

On 21 March 2013 14:00\, Marc Lehmann \schmorp@schmorp\.de wrote:

On Thu\, Mar 21\, 2013 at 12:28:54PM +0100\, demerphq \demerphq@gmail\.com wrote: +-> > >> > - Unless I am mistaken\, this is to avoid a DOS attack. DOS attacks are easy +-> > >> > to protect against\, and this won't protect me against basically any form +-> > >> > of attack\, so it's useless. Far better methods exist that prevent every +-> > >> > form of memory resource and/or cpu time starvation. > >> > >> You are mistaken. > > > > You are really claiming that this is one of the best methods to prevent > > memory resource and/or cpu time starvation? Because that's what the satteemnt > > you quoted says. > > I am sorry\, I seem to be missing some context here. What statement did > I quote? I just checked this thread I did not quote anything.

+-- These ones (hope you can read my ascii art).

I interpreted it to mean to refer to the last one you quoted.

Now\, if you didn't quote anything\, what would your statement refer to?

Ah\, right\, I didn't consider that "quoting" even tho of course it is.

The purpose of hash iterator randomization is make it harder\, and hopefully outright impossible to discover the hash seed a given process is using.

That in itself is obviously a no-goal. No\, I don't believe that is the real purpose. It must be something else\, like: "to make it harder to use DOS attacks exploiting knowledge about hash collisions".

Or something like that.

In case you wonder why it is a no-goal\, I can discover these hash seeds with a debugger\, or an XS module\, and I doubt your change will stop me from doing it (to the extent this still exists).

Peter Rabbitson already replied to this. Here is the dialog that ensued:

On 21 March 2013 16:03\, Marc Lehmann \schmorp@schmorp\.de wrote:

On Fri\, Mar 22\, 2013 at 01:20:40AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote:

There is no access to the running process itself\, no XS\, no debugging tools. Just "black box" observation. The point of the changes is to make such type of observations futile\, nothing more.

I know. Please read the whole thread before making comments\, thank you.

I am confused... If you know there is no access to the running interpreter\, why did you bring up XS/debugging in the first place...?

Because I originally said it's about avoiding DOS attacks (and later refining that to mean cpu/memory starvation)\, and yves said\, no\, it's purpose is to make it hard to discover the seed\, after which I pointed out that this is unlikely the real purpose\, because you can already discover this seed in other ways (for example\, by setting an env variable before starting the program).

You know that this is not about XS nor having access to a debugger\, nor the ability to set the environment var\, yet bring it up anyway.

So you are trolling.

Once an attacker can determine the hash seed they can launch algorithmic complexity attacks on our hash implementation. However the iterator randomization does not in any way prevent a DOS\, it prevents obtaining the information that makes a DOS possible.

I think you are splitting hairs\, but you get it wrong.

Your attitude makes me want to split hairs with you. You employ a highly disagreeable style of communicating with people and should not be surprised when they do not want to be helpful towards you.

The patch to make hash iterator traversal random is specifically to make it harder for an attacker to determine the hash seed.

The purpose of your change is to make it harder to apply such DOS attacks. Saying that's not the purpose\, but the purpose is just to hide this information is dishonest\, because hiding this information isn't a useful purpose in itself.

I haven't been in any way dishonest. There is nothing false about my statement at all.

The perlfunc 5.16 documentation on "keys". Older versions are even more explicit.

all them are ambiguous\, but I already mentioned that.

Well\, I beg to differ. I think you are conflating "does not say anything on the subject" and "says something ambiguous".

I think it strongly implies it. Beat me if you wish\, but your interpretation is as good as mine\, regardless of the intent behind it.

No\, my interpretation is correct and yours is a figment of your imagination. There is no equivalence there.

The documentation for keys()\, and as far as I know all other documentation relating to hashes says nothing about consistency\, either cross process or in between hashes.

Sorry\, but "guaranteed to be the same as either the keys ..." clearly implies some consistency\, at least between those. Also saying the order is the same for the "same hash" is ambiguous\, as "same" has tow meanings\, namely\, "identical" (of same identity) and "similar in some kind" (e.g. same contents).

No\, no\, no. There is no consistency\, there never has been\, there never will be\, and arguing about it is unproductive.

It says that the keys are returned in an apparently random order\, that the random order is subject to change in future versions of perl\,

No problem with that.

the only guarantee of any form provided is that the keys() function will return items in the same order that each() and values would return them assuming the hash has not been modified (a secondary clause in the documentation for "each" specifies it is safe to delete the key that was just returned by each()).

It also says the ordering is the same for the same hash.

Yes\, as in identity. If you find that

keys(%hash)

returns items in a different order than

values(%hash)

does then please file a bug.

I know how it's meant\, but I also know what is said\, and assuming that "the same hash will result in the same ordering" is not unreasonable.

The docs say "the same hash unmodified"\, so that implies very strongly that it means identity.

Note that the earlier randomisation patch already broke the semantics\, as the ordering changed even within the same version (but different program runs).

No\, it didnt break any semantics. Stop making stuff up.

Furthermore I cannot find any evidence that the wording for this has changed in terms of specificity in the lifetime of perlfunc.pod:

Saying it changes in future versions but is the same for the same hash STRONGLY implies that the ordering is the same for the same version of perl\, which is already not true.

No it doesn't.

I give you\, as I said\, that the wording is ambiguous\, and that it is actually an older change that broke it\, but this interprettaion is entirely valid\, and I am not the only one who used it.

perlfunc.pod didnt exist prior to that commit.

And it clearly changed\, if you took notice.

Not in any way relevant to this discussion\, if you took notice.

I see nothing ambiguous in the documentation\, nor do I see it becoming more or less specific over time.

Saying you don'T se something has no value whatsoever. People tend to chose one interpretation of another.

You'd have to make the stronger claim that no other interprettaion is reasonably possible\, and I don't think you can\, simply because "same"\, in english\, is ambiguous as to whether mean identity or equality.

There is nothing here to argue about\, you are wrong\, and there your interpretation is wrong\, and that is all there is to it.

I don't think that\, and haven't made that claim. I think you can make faster hashes with better properties without the drawbacks.

So then why did you bring it up in this thread? If you have issues with hash performance and think you can do better then please start a new thread\, preferably by posting patches.

I don't think that\, and haven't made that claim. I think you can make ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Are you really so stupid as to claim you didnt say something only a few lines after you say it?

Did you not say:

"I think you can make faster hashes with better properties without the drawbacks."

And was I not replying to you saying it?

I said it before\, and Is aid it again: I didn't bring up what you put into my mouth\, and I explain it again what I meant\, and this time\, please either believe what I write\, or argue against it\, but please don't attack strawmen:

I think one can make faster and better hashes which fix this problem without breaking code.

That means the breakage is not necessary.

So prove it. Patches welcome.

- Taking out the inability to write a proper hash on all the perl developers out there is just wrong - Perl is the runtime\, and should get it right\, if possible.

This doesn't make grammatical sense\,

I think it's a perfectly fine english sentence. Maybe a bit complicated for you to understand though.

I guess you meant "your" when you said "the".

No\, I specifically meant "the"\, because I didn't want to say "your".

If so then I will just say that I find this comment rude and uniformed.

Since your assumption is wrong\, your conclusion is irrelevant fprtunately.

I am not responsible for our current hash implementation\,

And I didn't claim so. On purpose\, in fact.

nor do I think you could do any better. I am pretty sure that the only way one can improve perls hash performance is to make it do less\, and that by doing so you would upset far more people than you would make happy by making it a nanosecond faster.

But please\, go ahead and prove me wrong. Post some patches.

I already did some benchmarks with unordered hashes. I wonder what they do less than perl hashes\, specially now that almost all conssitency in ordering is gone\, which is usually the only problem.

Which just shows you really have no idea at all how Perls hashes work.

Which pretty much ends this discussion.

I am trying to say that the solution for bad hash tables is better hash tables\, or a sense of proportion\, not breaking perfectly fine existing code.

I find it really hard to take you seriously given how often you make silly statements like "not breaking perfectly fine existing code".

If you want me to respect your opinions on what is fine code\, please respect mine.

I don't really give a shit about your opinion actually. Especially as you frame it rudely\, make personal aspersions\, and act like a jerk when you express it.

your code broke because of this change it is because you were making unwarranted assumptions about how perl's hash function operated\,

I don't know if they are unwarranted\, but at least they were documented\, even if that turns out to be ambiguous wording.

As I said before\, there is nothing ambiguous about saying nothing about something. Ambiguity comes when there are more than one option\, here there are zero options. There is no reasonable interpretation of the documentation that supports your position.

as such your code wasn't perfectly fine. In fact if your code is breaking due to this change then I could trivially contrive code in an older version of perl that would also break your assumptions.

Well\, the code in question (which defines my assumptions) works with older versions of perl\, so I call your bluff.

So sure\, I can make it break with a one line change.

To me\, breaking existing code for questionable or no reason is making it worse. It is obvious that you disagree and think you didn't make it worse\, but at best\, you have to disagree with my perfectly valid opinion on this topic-

I have to disagree? What? Did you mean "agree"?

No\, I mean "disagree".

You seem to be very combative\, and I guess this is why you make so many simple mistakes: I usually really mean what I write (and even then\, I do make mistakes).

You are rude and insulting in your manner\, you should not be surprised when people you communicate with are combative.

I _really_ mean that you disagree\, and I really mean that you have to disagree with me if you have another opinion.

Anyway\, it doesnt matter. I don't break existing code for no reason.

Sure. I am saying they are bad reasons\, and badly executed\, not that it is without reason.

A) I have reasons\,

Never said anything to the contrary.

You certainly implied it with "breaking existing code for questionable or no reason is making it worse".

B) any code broken was already broken but the author did not know it

Empty claim.

Ive already demonstrated how your assumptions are broken by a simple hash copy.

C) there is a reason the docs say "The actual random order is subject to change in future versions of perl"

The actual random ordering changes with each make test run\, too\, but the testsuite still passes. That is cleatrly not the bug here.

and make no other guarantees that the limited ones I enumerated above. The reason of course is to allow the hash function to be changed or improved over time.

It says the ordering is the same for the same hash.

And it is. (So long as you dont modify it by inserting new items.)

Changing the hash function doesn't break the code. Changing the hash function within the same version of perl\, and within the same process\, is what changes it.

We don't change the hash function during the process.

At this point I must actually admit that I didn't even check that this is true\, I only assume it.

Well you are wrong.

So if the ordering is really the same for the same hash\, within the same process (using the same perl)\, then I am indeed totally wrong.

Is that the case?

@keys1= keys %hash; @keys2= keys %hash;

will always return identical @keys1 and @keys2 assuming %hash is a "normal" perl hash.

I am fully aware of them being volunteers\, but so am I (and I am a regular contributor as well)\, please keep that in mind.

Perhaps you are\, I never advanced an opinion on the subject.

Well\, I am. I tell you so. What are your reasons do doubt me so much?

Perhaps you are\, I never advanced an opinion on the subject.

But since you bring it up again I will note I checked the log and you are mentioned a whole 6 times.

$ git log --grep "Marc Lehmann" --pretty=oneline a5b310e3129d1a2cd55b8d79445eb65964c997d1 Update Digest-SHA to CPAN version 5.83 d9a4b459f94297889956ac3adc42707365f274c2 Restore building Encode's subextensions for a static build. 50c1ac04356af5f8e8f967db7ed083187aacb550 Pod nit in Encode.pm\, found by Marc Lehmann in RT #36949. 1a08a6ab91582d74e16135fa1c131885305144d0 [perl #22141] patch for Time::HiRes to get rid of -lrt on linux From: Marc Lehmann (via RT) \perlbug\-followup@perl\.org Message-Id: \<rt-22141-56710.3.69543054121962@bugs6 591166816e29d06350e6ec8041e206c1afa73419 In the UTF-8 branch of crypt() the extra \0 byte is required\, found by Marc Lehmann. bf8afc63e607fcc7c1b66ceb8d1c6a9f20862f5b Fix for a segfault\, from Marc Lehmann.

The perl5porters are not "better" than me as a major contributor to perl in any way\, and have no higher moral ground. Acting uppity because you are part of a mailinglist and contribute more often than me is proving nothing really.

I never made any comments about anyones individuals merits. Nor do I know what I did that was "uppity".

I interpreted your comment as irony\, as am pretty confident it was meant like it. If you really honetsly say you meant it as-is\, I will\, however\, believe you and admit my reaction to your last comment was wrong.

I meant every word in that sentence literally.

Well I have responded to the only cogent concern that I have seen you express\, which is that you think that these changes contradict the documentation. The documentation does not support you in this regard.

Well\, it clearly does. It's your interpretation that doesn't support mine\, and now the code.

There is no "clearly does" involved here. The documentation simply does not say what you seem to think it says. If it did it would say it explicitly\, but given how perls hash works (which I do know\, and which you clearly do not)\, I am very very certain that no person that did understand how perls hashes work would ever say what you think the documentation says.

Is there some other matter I should respond to?

You don't have to respond at all\, but if you do so\, I personally expect you to use more than empty claims\, and actually use evidence (which you have done\, for the most part).

Oh\, I suppose I forgot. Afaik\, you don't even have to fix this\, just take the patches I did to the JSON::PP tests and apply them.

See: https://rt.cpan.org/Public/Bug/Display.html?id=83421

But the bug is in the perl core. Nothing I apply can fix that\, becasue I can't apply fixes to the perl core.

Excuse me? No the bug is in your code\, expecting to snapshot a serialized hash and have that be the same as some other serialized hash with the same keys. There has never been any reasonable basis to expect this to always happen.

Besides\, it makes no sense to me to say "you don't have to fix that\, just apply the fix". What you mean is "you don't have to make the work to invent a fix\, it has already been done".

If you prefer that wording then so be it.

The problem is that I think the code is fine\, and even if not (by sheer power of force by you applying a buggy patch and claiming it's a good and reasonable and corretc patch and my code is broken)\, I think it *should* be correct\, because taking away a useful feature that doesn't (in itself) gain you anything

All I urge you\, again\, is to consider doing the "hardening" either better\, in a compatible way (which I think is possible)\, or consider the trade-off between a rather ineffective patch (that doesn't shield against similar DOS attacks) and breaking code that ought to work.

I don't have any strong hope for that\, and think you will opt for the ignore-and-break-it route\, which is a valid way to proceed from your side of things. I did feel the moral obligation to tell you why it is wrong\, and if you still do it\, you at least do it in an informed way.

Post some patches or shut up. I dont care what your opinion is on this subject anymore.

Yves

-- perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 08:36:34AM -0700\, "H. Merijn Brand via RT" \perlbug\-followup@perl\.org wrote:

IIRC there were only two or three minor issues that could be fixed easily without breaking anything\, but you refused in blaming me not to know the specs.

I don't remember it that way\, sorry. It wasn't just minor\, and as people here love to say "where's your patch"? Not that I would likely apply it\, for reasons stated.

I think it is not reasonable to ask to upgrade a compiler that conforms to the minimal requirements for building perl (and still does so up

Right\, asking people to write their C++ or Fortran code in C is not reasonable\, because you say so. Your statement is wrong in general\, and wrong in particular.

But well\, you are entitled to your opinion and not using my module\, but what you do is way out of proportions. But again\, your choice.

Then how hard would it be to conform to C89 and let the rest of the people that enjoy perl in?

The rest of the people happily enjoy it\, last I heard from them.

But I do consider your behaviour wholly unreasonable\, and if the requirement for you to use my software is that I port it to your pet compiler because you threaten me\, then you just have to live without it.

When did I ever threaten you?

In your mail merijin. And now you execute your threat (probably not the first time\, but I am just speculating). In any case\, feel free.

And the arrogance of trying to threaten me into doing your work for free is breathtaking.

Hrmph. IIRC

You don't.

my quest came with a (simple) patch. there was NO work for

It did not.

And making up bullshit like "it was NO work AT ALL" doesn't make it better. It would have been no work for you\, but it would be continued work for me\, because I take maintaining my package seriously.

If it was no work at all\, why didn't you do it? It's free software after all.

The lack of respect you show for the work others have done for you to use\, freely is staggering.

In reality\, you are just whining because I don't support your pet compiler. But it really is my decision on where I set the cut-off point of how antique language standards I support\, and you are entitled to do it better\, or not use my stuff. Complaining like you do is just whining.

What's next\, I need to support perl 5.001?

p5pRT commented 11 years ago

From @ribasushi

On Thu\, Mar 21\, 2013 at 04:52:49PM +0100\, Marc Lehmann wrote:

In reality\, you are just whining because I don't support your pet compiler. But it really is my decision on where I set the cut-off point of how antique language standards I support\, and you are entitled to do it better\, or not use my stuff. Complaining like you do is just whining.

What's next\, I need to support perl 5.001?

That last hypothetical requirement would be beyond unreasonable.

On the other hand this request is beyond reasonable (and very well argumented).

On Thu\, Mar 21\, 2013 at 04:35:48PM +0100\, H.Merijn Brand wrote:

I think it is not reasonable to ask to upgrade a compiler that conforms to the minimal requirements for building perl (and still does so up till today: I am about to release perl5.16.3 on that box soon). Especially when there are no updates available. Of course one could install another compiler\, like GNU gcc\, but remember that on that architecture\, it comes with quite a high performance penalty compared to building with the native compiler. I don't think an overall performance hit *ever* warrants the use of // comments in C/XS.

With all respect for your work and skill Marc\, I would go a tad out of character and call you a sloppy programmer[1]. Any XS (note\, not fortran not C++) code distributed on CPAN is absolutely expected to compile on *every* compiler that is capable of compiling perl5 blead.

Cheers

[1] Note this is not an attack on your character\, this is an attack/characterization of what you distribute on CPAN i.e. your XS work in particular.

p5pRT commented 11 years ago

From @ribasushi

On Thu\, Mar 21\, 2013 at 04:25:06PM +0100\, Marc Lehmann wrote:

I don't find that usefil\, I find the actual behaviour useful. Old perls arrived at the same order when I built them in the same way each time.

New perls do it only when setting an env variable\, or within one process - fair enough\, not grat\, but managable.

[1] Just to make sure I understand - from what I gather this:

`perl -MData::Dumper -e '$Data::Dumper::Indent = 0; print Dumper {(1..20)}; print "\n"'`

producing identical output on all perls between 5.8.2 and 5.16.3 is

No\, and I made it explicit that I am fine with that (and it's clearly documented\, at leats for me\, since 5.8.1 or so).

I have a problem (or rather\, my module has one) with this code not producing identical output:

use feature 'say';

my %a = (a => 1\, b => 2); my %b = (a => 1\, b => 2);

say keys %a; say keys %b;

Besides\, I didn't know these perls would produce identical output - but they seem to\, wow\, I didn't expect that\, but cool - though I wouldn't rely on that.

Strange... to me both behaviors are actually manifestation of the same logic. I somewhat understand how you can distinguish the two\, but I can't agree with your reasoning. I guess it's agree to disagree time ;)

You do not consider it a bug.

Are you sure anybody would consider it a bug to get the same output for the same perl snippet on multiple versions of perl?

For something that has its order declared as "undefined" - yes it is a bug.

I would say that's the default expectation\, don't you agree? At least not without good reasons for them to differ.

In my eye the worst situation in computing is when some undefined behavior appears to be "defined" 99.9% of the time. This is btw evidenced by the massive amount of CPAN borkage when hash randomization was fixed in 5.17

Cheers

p5pRT commented 11 years ago

From @tux

On Thu\, 21 Mar 2013 16:52:49 +0100\, Marc Lehmann \schmorp@schmorp\.de wrote:

On Thu\, Mar 21\, 2013 at 08:36:34AM -0700\, "H. Merijn Brand via RT" \perlbug\-followup@perl\.org wrote:

IIRC there were only two or three minor issues that could be fixed easily without breaking anything\, but you refused in blaming me not to know the specs.

I don't remember it that way\, sorry. It wasn't just minor\, and as people here love to say "where's your patch"? Not that I would likely apply it\, for reasons stated.

If you - in advance - already claim not to apply patches\, there is no use in spending time fixing bugs in your software. I - of course - am convinced that you think your code has no bugs at all\, so fixing isn't needed.

I think it is not reasonable to ask to upgrade a compiler that conforms to the minimal requirements for building perl (and still does so up

Right\, asking people to write their C++ or Fortran code in C is not reasonable\, because you say so. Your statement is wrong in general\, and wrong in particular.

You seem to be blaming anyone else as being "wrong" everywhere on this list. Do you ever agree with anyone? ever? Are you even *able* to admit that someone else is right?

But well\, you are entitled to your opinion and not using my module\, but what you do is way out of proportions. But again\, your choice.

And unless/until you conform to the requirements of perl/XS to C89\, It is very unlikely I will change that choice. Of course you are free to your own opinion of finding that out of proportions. I wonder why\, as you tell me you have loads of happy users. Why does my decision would change that is beyond me.

Then how hard would it be to conform to C89 and let the rest of the people that enjoy perl in?

The rest of the people happily enjoy it\, last I heard from them.

There you go :)

But I do consider your behaviour wholly unreasonable\, and if the requirement for you to use my software is that I port it to your pet compiler because you threaten me\, then you just have to live without it.

When did I ever threaten you?

In your mail merijin.

You don't even take time to correctly write my name.

But please point out any "threat" I posted. By quoting the text (and context if needed). Maybe you - again - misread something simple as threat. I have no reason to threaten you. I'm just explaining why I don't use your code. I won't kill or chase you for it. I can easily live without it.

And now you execute your threat (probably not the first time\, but I am just speculating). In any case\, feel free.

I am puzzled.

And the arrogance of trying to threaten me into doing your work for free is breathtaking.

Hrmph. IIRC

You don't.

Excuse me?

my quest came with a (simple) patch. there was NO work for

It did not.

See https://rt.cpan.org/Public/Bug/Display.html?id=42462 Second post. It might not be in the format/style you prefer\, but it IS a solution to the problem posted. __STDC__ is ANSI and has nothing to do with weird compilers or operating systems.

And making up bullshit like "it was NO work AT ALL" doesn't make it better. It would have been no work for you\, but it would be continued work for me\, because I take maintaining my package seriously.

just to a certain extent

If it was no work at all\, why didn't you do it? It's free software after all.

Because if I would install it *with* a patch\, I'd have to jump through hoops to do it again and again and again on updates: now THAT is work I don't want.

The lack of respect you show for the work others have done for you to use\, freely is staggering.

/me laughs out loudly you start making me feel better with every post

In reality\, you are just whining because I don't support your pet compiler.

The word you admire bounces back: bullshit.

1. I have no pet compiler 2. I don't care about compilers\, but about standard (C89 in particular) in conjunction with my beloved programming language perl

But it really is my decision on where I set the cut-off point

Absolutely! I agree completely! But then state so when you reject sensible patches instead of refusing them as wrong

of how antique language standards I support\, and you are entitled to do it better\, or not use my stuff. Complaining like you do is just whining.

Not really\, but if that is how you see it\, feel free

What's next\, I need to support perl 5.001?

why? it is officially out of support. Even 5.005 is. Support for 5.6.2 would be more than welcome\, but if you choose to take 5.8.7 as minimum\, that'd be fine with me. It is your choice.

/me wonders what perl versions have to do with compiler standards\, but maybe that is on-topic too

p5pRT commented 11 years ago

From schmorp@schmorp.de

Now\, if you didn't quote anything\, what would your statement refer to?

Ah\, right\, I didn't consider that "quoting" even tho of course it is.

And\, was I right in my reply or not?

In case you wonder why it is a no-goal\, I can discover these hash seeds with a debugger\, or an XS module\, and I doubt your change will stop me from doing it (to the extent this still exists).

Peter Rabbitson already replied to this.

Yes\, you should read the resulting thread\, it's quite informative.

On Fri\, Mar 22\, 2013 at 01:20:40AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote: Because I originally said it's about avoiding DOS attacks (and later refining that to mean cpu/memory starvation)\, and yves said\, no\, it's purpose is to make it hard to discover the seed\, after which I pointed out that this is unlikely the real purpose\, because you can already discover this seed in other ways (for example\, by setting an env variable before starting the program).

You know that this is not about XS nor having access to a debugger\, nor the ability to set the environment var\, yet bring it up anyway.

I brought it up originally because I used it as an argument against you claiming the purpose was not a DOS attack\, but avoiding the seed to be discovered.

You conveniently ignore my actual argument\, and that what you said it's real purpose is\, is misleading or disingenious.

So you are trolling.

I am obviously not\, but it seems you ran out of arguments quickly\, ignoring all mine.

Sad.

Your attitude makes me want to split hairs with you.

"I am not responsible\, officer\, he made me kill him!".

Given that you misunderstood what I wrote multiple times and then ignored when I set you right\, I am fairly sure you don't know what my attitude is.

But using that as excuse is lame. I am not playing any games\, I talk about technical issues - possibly in form of a rant.

Attacking me personally won't bring us forward\, it just makes you into a jerk\, and since I don't think you are\, you will only regret this later (as has happened before when you jumped to conclusions...).

Let's go back to the technical issue. Feel free to use clear language\, but ignoring all my technical agruments and attacking my person is a waste of time.

highly disagreeable style of communicating with people and should not be surprised when they do not want to be helpful towards you.

You can disagree with my style\, and I don't *expect* you to be helpful. Just reasonable.

What you now seem to tell me is that you would insist on a bad patch\, just because the person who brought the problems to light used a disagreeable style of communication.

The patch to make hash iterator traversal random is specifically to make it harder for an attacker to determine the hash seed.

That's what I assumed. I made a specific attack example and reminded you of existing mitigations that work on a broeder class of such problems\, and you disagreed.

It would bring su forward if you gave an example of where your patch actually did something useful in a system that expects such resource attacks (not specifically that one).

Or an attack where typical mitigations (resource limits and monitoring) wouldn't work.

The purpose of your change is to make it harder to apply such DOS attacks. Saying that's not the purpose\, but the purpose is just to hide this information is dishonest\, because hiding this information isn't a useful purpose in itself.

I haven't been in any way dishonest. There is nothing false about my statement at all.

Well\, you are good at denial\, but when I ask you direct questions about this and you keep refusing to answer\, that is dishonest\, too.

Really\, all you do is say "it's not true"\, and when asked for details\, you are silent.

I think it strongly implies it. Beat me if you wish\, but your interpretation is as good as mine\, regardless of the intent behind it.

No\, my interpretation is correct and yours is a figment of your imagination. There is no equivalence there.

I just looked up the meaning of "same" at thefreedictionary.com.

First meaning it lists is "identical". Second meaning it lists is "similar in kind\, quality\, quantity\, or degree". (there are two more).

Sorry\, but calling that a "figment of my imagination" is just your attempt at bullying me.

The wording is clearly ambiguous in this case\, and just saying "I am big bozo\, I am right" doesn't make it so.

Sorry\, but "guaranteed to be the same as either the keys ..." clearly implies some consistency\, at least between those. Also saying the order is the same for the "same hash" is ambiguous\, as "same" has tow meanings\, namely\, "identical" (of same identity) and "similar in some kind" (e.g. same contents).

No\, no\, no. There is no consistency\, there never has been\, there never will be\, and arguing about it is unproductive.

Ok\, so ordering I get from keys is not consistent with the ordering I get from values? If that are your plans it is indeed high time to fork perl\, because that isn't just clearly documented\, it is clearly required for a lot of existing and useful code.

Or you disagree with what "same" means? It seems arguing against dictionaries is unproductive\, though\, it leads nowhere (except you being wrong in this case\, but it's still pointless...).

the only guarantee of any form provided is that the keys() function will return items in the same order that each() and values would return them assuming the hash has not been modified (a secondary clause in the documentation for "each" specifies it is safe to delete the key that was just returned by each()).

It also says the ordering is the same for the same hash.

Yes\, as in identity.

And it does that where...?

keys(%hash) returns items in a different order than values(%hash) does then please file a bug.

So suddenly it is consistent. This is all very confusing to me.

I know how it's meant\, but I also know what is said\, and assuming that "the same hash will result in the same ordering" is not unreasonable.

The docs say "the same hash unmodified"\, so that implies very strongly that it means identity.

Well\, not to me\, and the other person I asked (I am too lazy to ask a bigger control group\, sorry).

Note that the earlier randomisation patch already broke the semantics\, as the ordering changed even within the same version (but different program runs).

No\, it didnt break any semantics. Stop making stuff up.

If you didn't break semantics\, then explain why JSON::XS fails it's testsuite?

Saying it changes in future versions but is the same for the same hash STRONGLY implies that the ordering is the same for the same version of perl\, which is already not true.

No it doesn't.

Why mention it at all then? If the ordering is randomly changing\, then why mention it might change in future versions? It frankly makes no sense.

perlfunc.pod didnt exist prior to that commit. And it clearly changed\, if you took notice. Not in any way relevant to this discussion\, if you took notice.

Your point being that it is a documentation bug\, that the existing documentation is ambiguous or what?

I am not claiming your patch breaks documented behaviour (for every interprettation). I am claiming the code in question doesn't rely on undocumented behaviour.

You'd have to make the stronger claim that no other interprettaion is reasonably possible\, and I don't think you can\, simply because "same"\, in english\, is ambiguous as to whether mean identity or equality.

There is nothing here to argue about\, you are wrong\, and there your interpretation is wrong\, and that is all there is to it.

You seem to be unable to explain why it would be wrong\, while I gave ample evidence.

The only reason it seems to be wrong is that the person who wrote it might have intended otherwise\, but there is no evidence for that (was it tomc?)\, and even if\, people wrote programs according to the docs\, and if the documentation allows the behaviour because it is worded differently\, and this behaviour is useful and present for more than a decade (probably two decades\, who knows)\, then you can't just say "you are wrong".

Because all evidence for that is your word\, and your refusal to explain it.

I don't think that\, and haven't made that claim. I think you can make ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Are you really so stupid as to claim you didnt say something only a few lines after you say it?

Please reconsider. It took me a while to find the context\, but as you force me to\, here it is:

- Perl hash already are very slow (just write yourself some XS functions to interface to gcc's unordered_hash and watch the speed - despite the overhead of XS).

Yes\, and this is a very tiny drop in a large bucket. If you think that the changes I made are going to affect hash performance in any kind of significant way then please provide evidence.

I don't think that\, and haven't made that claim.

And this is true. Calling me stupid and claiming I did\, when I didn't\, is crossing the line.

I didn't say it would affect it in any significant way (heck\, I don't know what you mena with significant - there are a lot of slowdowns in perl that I find significant but others don't).

I said they are already slow\, and extra overhead (for sorting\, or the randomisation). And what I meant (as I think I explained later) is that adding these kinds of overhead is likely unnecessary.

Did you not say:

"I think you can make faster hashes with better properties without the drawbacks."

Yeah.

And was I not replying to you saying it?

No.

Your interpretation is probably a valid one and I wouldn't gfault you if you made assumptions based on it\, but after I told you it isn't\, whats the point of arguing further?

I told you twice I didn't say that\, while explaining what I mean with what I actually said.

Sorry\, but your interpretation is wrong\, and I know\, because I am authoritative here. Trying to repeatedly claim I said somethig else after I clarified is dishonest.

mouth\, and I explain it again what I meant\, and this time\, please either believe what I write\, or argue against it\, but please don't attack strawmen:

I think one can make faster and better hashes which fix this problem without breaking code.

That means the breakage is not necessary.

So prove it.

I don't have to\, I gave good evidence for it\, while you gave no evidence for the contrary.

I already did some benchmarks with unordered hashes. I wonder what they do less than perl hashes\, specially now that almost all conssitency in ordering is gone\, which is usually the only problem.

Which just shows you really have no idea at all how Perls hashes work.

Which pretty much ends this discussion.

Again\, it seems when you run out of arguments you just play big boss.

I have to assume that means you just made your claim up.

If you want me to respect your opinions on what is fine code\, please respect mine.

I don't really give a shit about your opinion actually.

That's fine\, but what you are doing is far worse: while I give evidence for my opinions\, you give none for yours\, and discredit mine by calling it silly etc.

Especially as you frame it rudely\,

Where?

make personal aspersions\, and act like a jerk when you express it.

Where?

I would guess you again either ginroe ym questions\, or just claim it's true without evidence.

(sure\, I just say that to atcually get you to bring some evidence\, but I don't think you can...)

Really\, all you do is run into personal attacks after you ran out of arguments (which was very early\, too).

I don't know if they are unwarranted\, but at least they were documented\, even if that turns out to be ambiguous wording.

As I said before\, there is nothing ambiguous about saying nothing about something.

How philosophical. Not the point\, though-

Ambiguity comes when there are more than one option\,

Hey\, I checked a few dictionaries\, and listing "similar" as equal meaning to "identical" is the norm.

Again\, you are just claiming english is wrong and you are right. That's... well\, not very useful.

The claim that the same hash results in the same ordering is ambiguous w.r.t. the meaning of what "same hash" means.

Identity is one meaning\, but the others are there.

here there are zero options.

Only because you ignroe the extra options I stated. As well as you basically ignored all my other arguments.

With none on your own.

Sure that's the kind of maintainer that does good to perl: making decisions because they are right\, even though all the evidence points against it.

There is no reasonable interpretation of the documentation that supports your position.

There certainly is\, I gave it\, and ignoring it doesn't make it go away.

Well\, the code in question (which defines my assumptions) works with older versions of perl\, so I call your bluff.

So sure\, I can make it break with a one line change.

I know you can break code. If you wish I will call you master of breaking code.

That's the problem. I don't want you to break code\, not without good reason. And all reasons you gave is "because you are right\, and I am wrong".

Right\, I understand all that.

You seem to be very combative\, and I guess this is why you make so many simple mistakes: I usually really mean what I write (and even then\, I do make mistakes).

You are rude and insulting in your manner\, you should not be surprised when people you communicate with are combative.

So where was I rude? Where was I insulting?

The problem here is that you misinterpreted what I said\, by changing words\, AFTER I explained to you what I meant..

And now you claim I forced you to. That's just lame\, and you know that.

A) I have reasons\,

Never said anything to the contrary.

You certainly implied it with "breaking existing code for questionable or no reason is making it worse".

No\, I implied what I wrote: the reasons are questionable or nonexistant. And the context you again so conveniently clipped was:

To me\, breaking existing code for questionable or no reason is making it worse.

Quoting things out of context and then building strawmen about what wasn't actually said is dishonest.

B) any code broken was already broken but the author did not know it

Empty claim.

Ive already demonstrated how your assumptions are broken by a simple hash copy.

Well\, you didn't\, as I have explained. The problem here is not "a simple hash copy"\, but the assumptions behind the breakage in JSON::XS and in other modules.

AFAIK\, they don't rely on your example\, or the assumptions you invent.

It says the ordering is the same for the same hash.

And it is. (So long as you dont modify it by inserting new items.)

Experiments show you are wrong.

Changing the hash function doesn't break the code. Changing the hash function within the same version of perl\, and within the same process\, is what changes it.

We don't change the hash function during the process.

Well\, the observable effects are different\, which is the point. You are splitting hais a lot again\, when it's totalyl unnecessar\< because I have made explicit what the problem is\, namely the breakage of JSON::XS.

That the function gets different inputs but the form stays the same is immaterial\, and I am just so sure you know that.

So if the ordering is really the same for the same hash\, within the same process (using the same perl)\, then I am indeed totally wrong.

Is that the case?

@keys1= keys %hash; @keys2= keys %hash;

will always return identical @keys1 and @keys2 assuming %hash is a "normal" perl hash.

I call that a consistent ordering\, you don't.

I think you have a totally unworkable definition of consistent. At least that is clear now\, and explains a few things.

What I mean with consistent is that the ordering is the same\, within limits.

But since you bring it up again I will note I checked the log and you are mentioned a whole 6 times.

The my dick is longer than yours argument is not an argument. I still regularly contributed\, not just by posting on that mailinglist with this e-mail address\, over more than a decade.

Point being\, it means nothing.

I interpreted your comment as irony\, as am pretty confident it was meant like it. If you really honetsly say you meant it as-is\, I will\, however\, believe you and admit my reaction to your last comment was wrong.

I meant every word in that sentence literally.

Ok\, my turn to say sorry for having misinterpreted you. Oh wait\, you insulted me\, you didn't apologize.

Well\, I really am sorry for misinterpreting you then\, I honestly thought it was irony.

At least we can settle that part then?

express\, which is that you think that these changes contradict the documentation. The documentation does not support you in this regard.

Well\, it clearly does. It's your interpretation that doesn't support mine\, and now the code.

There is no "clearly does" involved here.

Well\, I have explained why. All your say is "it's wrong".

I go with the evidence and not the big mouth.

The documentation simply does not say what you seem to think it says.

That's beside the point\, you said my interpretation is unreasonable (and other things)\, and I say it was\, and gave evidence on why.

I am also not the only one who interpreted it that way\, as evidences by the breakage that ensued after both stages of patches in that area.

That doesn't prove the reasonableness\, of course.

If it did it would say it explicitly\,

It says it explicitly\, but unfortunately\, in an ambiguous way.

but given how perls hash works (which I do know\, and which you clearly do not)\,

I actually know very well how hashes worked before your patch. I studied them long ago\, and then regulalry.

You have no evidence that my knowledge is bad\, after all\, my code actually worked\, doesn't it\, and I didnt make a wrong statement\, did I?

What's your reason for making baseless claims at the lack of my knowledge? You just want to be an ass\, or you just want to feel bigger than me? Really\, I am curious.

I am very very certain that no person that did understand how perls hashes work would ever say what you think the documentation says.

Maybe\, but I wonder what relevance that has.

Do you really mean to claim that everybody should disregard the documentation and read thre actual code before being able to write reasonable or correct code?

I think it should be possible to write correct code by documentation and experimenting alone\, but we have to have to disagree on that point.

But the bug is in the perl core. Nothing I apply can fix that\, becasue I can't apply fixes to the perl core.

Excuse me? No the bug is in your code\,

No\, it's in the core. It was introduced recently\, and apparently by your patch (I blindly trust andk here).

expecting to snapshot a serialized hash and have that be the same as some other serialized hash with the same keys.

I expect the same hash to have same ordering within one program run\, yes.

There has never been any reasonable basis to expect this to always happen.

Except the documentation\, which to me\, will always be a reasonable basis. Askign people to understand all the perl core code before reasonably expecting to write perl is not\, uhum\, reasonable for me.

But sure\, you cna disagree here\, but it's a horrible outlook for the future of perl.

Post some patches or shut up.

Not as long as you spread falseness or baseless accusations\, no. You have no right to that. At all\, no matter how powerful you are.

I dont care what your opinion is on this subject anymore.

And I told you that is valid. But you can't hide between not having been told anymore.

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Fri\, Mar 22\, 2013 at 03:01:31AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote:

With all respect for your work and skill Marc\, I would go a tad out of character and call you a sloppy programmer[1]. Any XS (note\, not fortran not C++) code distributed on CPAN is absolutely expected to compile on *every* compiler that is capable of compiling perl5 blead.

Then we have to disagree on that\, Peter. I don't even have access to most of these compilers\, nor do I have the time to do all that work.

Again\, I give my code freely\, and asking any kind of XS code to compile with any compiler is just unreasonable. C++ needs to compile with a C compiler? Linux-specific code needs to compile on OS X? A third-party library that is converted to XS needs to be ported to every platform perl runs on\, even if almost nobody has access to it?

This is not just a matter of disagreeing\, I would go as far as to say this is so far beyond unreasonable that it's not even serious. Are you really serious? Sorry\, but I find that hard to believe.

Or maybe that problem here is the word "expected" - by whom? I certainly don't expect that\, and I know this expectation would be completely unwarranted. Just go to some antique or esoteric platform and compile some of CPAN\, that will heal you of that expectaton quickly.

(You could even start with windows and see how mayn XS modules compile there).

In any case\, without very good reasons otherwise\, this is what you are going to get from me. You are free to not use my code as result

[1] Note this is not an attack on your character\, this is an attack/characterization of what you distribute on CPAN i.e. your XS work in particular.

I'd be honestly interested on what that opinion is based though. Have you really looked at my XS work? My most popular XS modules are EV\, JSON::XS\, Coro\, Convert-UUlib (mostly not by me\, and admittedly bad code)\, Guard and IO::AIO.

I have spent probably the better part of my life to make these portable and run well (even on hard to port to platforms such as Windows\, which has no reasonably bug free perl port itself). I wasted much of the time during my university studies\, and a large share of my professional work consists of maintaining these modules\, often at a loss of compensation.

I have no problem with that. I love coding\, and I publsih my stuff in the hope that it is being useful. But putting extraordinary demands on me\, like porting to very old platforms for a single and very unfriendly user\, is simply too much for me.

I am afraid that if suddenly that is no longer enough for CPAN\, then you can close shop. I certainly can't see how I could invest even more work\, and I think your accusation of that codebase being by a sloppy programmer is unwarranted.

I honestly invite you to look at my modules and then repeat your judgement.

And I will then just have to tell you that I obviously fail the mark for publishing modules on CPAN\, and with me\, most of the XS authors.

p5pRT commented 11 years ago

From vadim.konovalov@alcatel-lucent.com

From: Marc Lehmann I have a problem (or rather\, my module has one) with this code not producing identical output:

use feature 'say';

my %a = (a => 1\, b => 2); my %b = (a => 1\, b => 2);

say keys %a; say keys %b;

for my view\, the 2 hashes at right side of "my" assigments are different\, although these are similarly looking and also contaning some equal content\,

hence I do expect them to have unpredictable order of "keys".

les 2 cents a moi....

p5pRT commented 11 years ago

From @Leont

On Thu\, Mar 21\, 2013 at 6:04 PM\, schmorp@schmorp.de \perlbug\-followup@perl\.org wrote:

# New Ticket Created by schmorp@schmorp.de # Please include the string: [perl #117271] # in the subject line of all future correspondence about this issue. # \<URL: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=117271 >

Now\, if you didn't quote anything\, what would your statement refer to?

Ah\, right\, I didn't consider that "quoting" even tho of course it is.

And\, was I right in my reply or not?

In case you wonder why it is a no-goal\, I can discover these hash seeds with a debugger\, or an XS module\, and I doubt your change will stop me from doing it (to the extent this still exists).

Peter Rabbitson already replied to this.

Yes\, you should read the resulting thread\, it's quite informative.

On Fri\, Mar 22\, 2013 at 01:20:40AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote: Because I originally said it's about avoiding DOS attacks (and later refining that to mean cpu/memory starvation)\, and yves said\, no\, it's purpose is to make it hard to discover the seed\, after which I pointed out that this is unlikely the real purpose\, because you can already discover this seed in other ways (for example\, by setting an env variable before starting the program).

You know that this is not about XS nor having access to a debugger\, nor the ability to set the environment var\, yet bring it up anyway.

I brought it up originally because I used it as an argument against you claiming the purpose was not a DOS attack\, but avoiding the seed to be discovered.

You conveniently ignore my actual argument\, and that what you said it's real purpose is\, is misleading or disingenious.

So you are trolling.

I am obviously not\, but it seems you ran out of arguments quickly\, ignoring all mine.

Sad.

Your attitude makes me want to split hairs with you.

"I am not responsible\, officer\, he made me kill him!".

Given that you misunderstood what I wrote multiple times and then ignored when I set you right\, I am fairly sure you don't know what my attitude is.

But using that as excuse is lame. I am not playing any games\, I talk about technical issues - possibly in form of a rant.

Attacking me personally won't bring us forward\, it just makes you into a jerk\, and since I don't think you are\, you will only regret this later (as has happened before when you jumped to conclusions...).

Let's go back to the technical issue. Feel free to use clear language\, but ignoring all my technical agruments and attacking my person is a waste of time.

highly disagreeable style of communicating with people and should not be surprised when they do not want to be helpful towards you.

You can disagree with my style\, and I don't *expect* you to be helpful. Just reasonable.

What you now seem to tell me is that you would insist on a bad patch\, just because the person who brought the problems to light used a disagreeable style of communication.

The patch to make hash iterator traversal random is specifically to make it harder for an attacker to determine the hash seed.

That's what I assumed. I made a specific attack example and reminded you of existing mitigations that work on a broeder class of such problems\, and you disagreed.

It would bring su forward if you gave an example of where your patch actually did something useful in a system that expects such resource attacks (not specifically that one).

Or an attack where typical mitigations (resource limits and monitoring) wouldn't work.

The purpose of your change is to make it harder to apply such DOS attacks. Saying that's not the purpose\, but the purpose is just to hide this information is dishonest\, because hiding this information isn't a useful purpose in itself.

I haven't been in any way dishonest. There is nothing false about my statement at all.

Well\, you are good at denial\, but when I ask you direct questions about this and you keep refusing to answer\, that is dishonest\, too.

Really\, all you do is say "it's not true"\, and when asked for details\, you are silent.

I think it strongly implies it. Beat me if you wish\, but your interpretation is as good as mine\, regardless of the intent behind it.

No\, my interpretation is correct and yours is a figment of your imagination. There is no equivalence there.

I just looked up the meaning of "same" at thefreedictionary.com.

First meaning it lists is "identical". Second meaning it lists is "similar in kind\, quality\, quantity\, or degree". (there are two more).

Sorry\, but calling that a "figment of my imagination" is just your attempt at bullying me.

The wording is clearly ambiguous in this case\, and just saying "I am big bozo\, I am right" doesn't make it so.

Sorry\, but "guaranteed to be the same as either the keys ..." clearly implies some consistency\, at least between those. Also saying the order is the same for the "same hash" is ambiguous\, as "same" has tow meanings\, namely\, "identical" (of same identity) and "similar in some kind" (e.g. same contents).

No\, no\, no. There is no consistency\, there never has been\, there never will be\, and arguing about it is unproductive.

Ok\, so ordering I get from keys is not consistent with the ordering I get from values? If that are your plans it is indeed high time to fork perl\, because that isn't just clearly documented\, it is clearly required for a lot of existing and useful code.

Or you disagree with what "same" means? It seems arguing against dictionaries is unproductive\, though\, it leads nowhere (except you being wrong in this case\, but it's still pointless...).

the only guarantee of any form provided is that the keys() function will return items in the same order that each() and values would return them assuming the hash has not been modified (a secondary clause in the documentation for "each" specifies it is safe to delete the key that was just returned by each()).

It also says the ordering is the same for the same hash.

Yes\, as in identity.

And it does that where...?

keys(%hash) returns items in a different order than values(%hash) does then please file a bug.

So suddenly it is consistent. This is all very confusing to me.

I know how it's meant\, but I also know what is said\, and assuming that "the same hash will result in the same ordering" is not unreasonable.

The docs say "the same hash unmodified"\, so that implies very strongly that it means identity.

Well\, not to me\, and the other person I asked (I am too lazy to ask a bigger control group\, sorry).

Note that the earlier randomisation patch already broke the semantics\, as the ordering changed even within the same version (but different program runs).

No\, it didnt break any semantics. Stop making stuff up.

If you didn't break semantics\, then explain why JSON::XS fails it's testsuite?

Saying it changes in future versions but is the same for the same hash STRONGLY implies that the ordering is the same for the same version of perl\, which is already not true.

No it doesn't.

Why mention it at all then? If the ordering is randomly changing\, then why mention it might change in future versions? It frankly makes no sense.

perlfunc.pod didnt exist prior to that commit. And it clearly changed\, if you took notice. Not in any way relevant to this discussion\, if you took notice.

Your point being that it is a documentation bug\, that the existing documentation is ambiguous or what?

I am not claiming your patch breaks documented behaviour (for every interprettation). I am claiming the code in question doesn't rely on undocumented behaviour.

You'd have to make the stronger claim that no other interprettaion is reasonably possible\, and I don't think you can\, simply because "same"\, in english\, is ambiguous as to whether mean identity or equality.

There is nothing here to argue about\, you are wrong\, and there your interpretation is wrong\, and that is all there is to it.

You seem to be unable to explain why it would be wrong\, while I gave ample evidence.

The only reason it seems to be wrong is that the person who wrote it might have intended otherwise\, but there is no evidence for that (was it tomc?)\, and even if\, people wrote programs according to the docs\, and if the documentation allows the behaviour because it is worded differently\, and this behaviour is useful and present for more than a decade (probably two decades\, who knows)\, then you can't just say "you are wrong".

Because all evidence for that is your word\, and your refusal to explain it.

I don't think that\, and haven't made that claim. I think you can make ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Are you really so stupid as to claim you didnt say something only a few lines after you say it?

Please reconsider. It took me a while to find the context\, but as you force me to\, here it is:

- Perl hash already are very slow (just write yourself some XS functions to interface to gcc's unordered_hash and watch the speed - despite the overhead of XS).

Yes\, and this is a very tiny drop in a large bucket. If you think that the changes I made are going to affect hash performance in any kind of significant way then please provide evidence.

I don't think that\, and haven't made that claim.

And this is true. Calling me stupid and claiming I did\, when I didn't\, is crossing the line.

I didn't say it would affect it in any significant way (heck\, I don't know what you mena with significant - there are a lot of slowdowns in perl that I find significant but others don't).

I said they are already slow\, and extra overhead (for sorting\, or the randomisation). And what I meant (as I think I explained later) is that adding these kinds of overhead is likely unnecessary.

Did you not say:

"I think you can make faster hashes with better properties without the drawbacks."

Yeah.

And was I not replying to you saying it?

No.

Your interpretation is probably a valid one and I wouldn't gfault you if you made assumptions based on it\, but after I told you it isn't\, whats the point of arguing further?

I told you twice I didn't say that\, while explaining what I mean with what I actually said.

Sorry\, but your interpretation is wrong\, and I know\, because I am authoritative here. Trying to repeatedly claim I said somethig else after I clarified is dishonest.

mouth\, and I explain it again what I meant\, and this time\, please either believe what I write\, or argue against it\, but please don't attack strawmen:

I think one can make faster and better hashes which fix this problem without breaking code.

That means the breakage is not necessary.

So prove it.

I don't have to\, I gave good evidence for it\, while you gave no evidence for the contrary.

I already did some benchmarks with unordered hashes. I wonder what they do less than perl hashes\, specially now that almost all conssitency in ordering is gone\, which is usually the only problem.

Which just shows you really have no idea at all how Perls hashes work.

Which pretty much ends this discussion.

Again\, it seems when you run out of arguments you just play big boss.

I have to assume that means you just made your claim up.

If you want me to respect your opinions on what is fine code\, please respect mine.

I don't really give a shit about your opinion actually.

That's fine\, but what you are doing is far worse: while I give evidence for my opinions\, you give none for yours\, and discredit mine by calling it silly etc.

Especially as you frame it rudely\,

Where?

make personal aspersions\, and act like a jerk when you express it.

Where?

I would guess you again either ginroe ym questions\, or just claim it's true without evidence.

(sure\, I just say that to atcually get you to bring some evidence\, but I don't think you can...)

Really\, all you do is run into personal attacks after you ran out of arguments (which was very early\, too).

I don't know if they are unwarranted\, but at least they were documented\, even if that turns out to be ambiguous wording.

As I said before\, there is nothing ambiguous about saying nothing about something.

How philosophical. Not the point\, though-

Ambiguity comes when there are more than one option\,

Hey\, I checked a few dictionaries\, and listing "similar" as equal meaning to "identical" is the norm.

Again\, you are just claiming english is wrong and you are right. That's... well\, not very useful.

The claim that the same hash results in the same ordering is ambiguous w.r.t. the meaning of what "same hash" means.

Identity is one meaning\, but the others are there.

here there are zero options.

Only because you ignroe the extra options I stated. As well as you basically ignored all my other arguments.

With none on your own.

Sure that's the kind of maintainer that does good to perl: making decisions because they are right\, even though all the evidence points against it.

There is no reasonable interpretation of the documentation that supports your position.

There certainly is\, I gave it\, and ignoring it doesn't make it go away.

Well\, the code in question (which defines my assumptions) works with older versions of perl\, so I call your bluff.

So sure\, I can make it break with a one line change.

I know you can break code. If you wish I will call you master of breaking code.

That's the problem. I don't want you to break code\, not without good reason. And all reasons you gave is "because you are right\, and I am wrong".

Right\, I understand all that.

You seem to be very combative\, and I guess this is why you make so many simple mistakes: I usually really mean what I write (and even then\, I do make mistakes).

You are rude and insulting in your manner\, you should not be surprised when people you communicate with are combative.

So where was I rude? Where was I insulting?

The problem here is that you misinterpreted what I said\, by changing words\, AFTER I explained to you what I meant..

And now you claim I forced you to. That's just lame\, and you know that.

A) I have reasons\,

Never said anything to the contrary.

You certainly implied it with "breaking existing code for questionable or no reason is making it worse".

No\, I implied what I wrote: the reasons are questionable or nonexistant. And the context you again so conveniently clipped was:

To me\, breaking existing code for questionable or no reason is making it worse.

Quoting things out of context and then building strawmen about what wasn't actually said is dishonest.

B) any code broken was already broken but the author did not know it

Empty claim.

Ive already demonstrated how your assumptions are broken by a simple hash copy.

Well\, you didn't\, as I have explained. The problem here is not "a simple hash copy"\, but the assumptions behind the breakage in JSON::XS and in other modules.

AFAIK\, they don't rely on your example\, or the assumptions you invent.

It says the ordering is the same for the same hash.

And it is. (So long as you dont modify it by inserting new items.)

Experiments show you are wrong.

Changing the hash function doesn't break the code. Changing the hash function within the same version of perl\, and within the same process\, is what changes it.

We don't change the hash function during the process.

Well\, the observable effects are different\, which is the point. You are splitting hais a lot again\, when it's totalyl unnecessar\< because I have made explicit what the problem is\, namely the breakage of JSON::XS.

That the function gets different inputs but the form stays the same is immaterial\, and I am just so sure you know that.

So if the ordering is really the same for the same hash\, within the same process (using the same perl)\, then I am indeed totally wrong.

Is that the case?

@keys1= keys %hash; @keys2= keys %hash;

will always return identical @keys1 and @keys2 assuming %hash is a "normal" perl hash.

I call that a consistent ordering\, you don't.

I think you have a totally unworkable definition of consistent. At least that is clear now\, and explains a few things.

What I mean with consistent is that the ordering is the same\, within limits.

But since you bring it up again I will note I checked the log and you are mentioned a whole 6 times.

The my dick is longer than yours argument is not an argument. I still regularly contributed\, not just by posting on that mailinglist with this e-mail address\, over more than a decade.

Point being\, it means nothing.

I interpreted your comment as irony\, as am pretty confident it was meant like it. If you really honetsly say you meant it as-is\, I will\, however\, believe you and admit my reaction to your last comment was wrong.

I meant every word in that sentence literally.

Ok\, my turn to say sorry for having misinterpreted you. Oh wait\, you insulted me\, you didn't apologize.

Well\, I really am sorry for misinterpreting you then\, I honestly thought it was irony.

At least we can settle that part then?

express\, which is that you think that these changes contradict the documentation. The documentation does not support you in this regard.

Well\, it clearly does. It's your interpretation that doesn't support mine\, and now the code.

There is no "clearly does" involved here.

Well\, I have explained why. All your say is "it's wrong".

I go with the evidence and not the big mouth.

The documentation simply does not say what you seem to think it says.

That's beside the point\, you said my interpretation is unreasonable (and other things)\, and I say it was\, and gave evidence on why.

I am also not the only one who interpreted it that way\, as evidences by the breakage that ensued after both stages of patches in that area.

That doesn't prove the reasonableness\, of course.

If it did it would say it explicitly\,

It says it explicitly\, but unfortunately\, in an ambiguous way.

but given how perls hash works (which I do know\, and which you clearly do not)\,

I actually know very well how hashes worked before your patch. I studied them long ago\, and then regulalry.

You have no evidence that my knowledge is bad\, after all\, my code actually worked\, doesn't it\, and I didnt make a wrong statement\, did I?

What's your reason for making baseless claims at the lack of my knowledge? You just want to be an ass\, or you just want to feel bigger than me? Really\, I am curious.

I am very very certain that no person that did understand how perls hashes work would ever say what you think the documentation says.

Maybe\, but I wonder what relevance that has.

Do you really mean to claim that everybody should disregard the documentation and read thre actual code before being able to write reasonable or correct code?

I think it should be possible to write correct code by documentation and experimenting alone\, but we have to have to disagree on that point.

But the bug is in the perl core. Nothing I apply can fix that\, becasue I can't apply fixes to the perl core.

Excuse me? No the bug is in your code\,

No\, it's in the core. It was introduced recently\, and apparently by your patch (I blindly trust andk here).

expecting to snapshot a serialized hash and have that be the same as some other serialized hash with the same keys.

I expect the same hash to have same ordering within one program run\, yes.

There has never been any reasonable basis to expect this to always happen.

Except the documentation\, which to me\, will always be a reasonable basis. Askign people to understand all the perl core code before reasonably expecting to write perl is not\, uhum\, reasonable for me.

But sure\, you cna disagree here\, but it's a horrible outlook for the future of perl.

Post some patches or shut up.

Not as long as you spread falseness or baseless accusations\, no. You have no right to that. At all\, no matter how powerful you are.

I dont care what your opinion is on this subject anymore.

And I told you that is valid. But you can't hide between not having been told anymore.

Marc\, can you please stop doing this.

Obviously you're not seeing why people react so combative to you\, but given how consistently this happens to you (and not to most other people)\, don't you think you're the key player in that dynamic? You consistently manage to get upset people\, and in the end yourself.

Perhaps you could consider delegating these discussions to someone you trust who can communicate with p5p more effectively.

Leon

p5pRT commented 11 years ago

The RT System itself - Status changed from 'new' to 'open'

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Fri\, Mar 22\, 2013 at 03:10:00AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote:

I have a problem (or rather\, my module has one) with this code not producing identical output:

use feature 'say';

my %a = (a => 1\, b => 2); my %b = (a => 1\, b => 2);

say keys %a; say keys %b;

Besides\, I didn't know these perls would produce identical output - but they seem to\, wow\, I didn't expect that\, but cool - though I wouldn't rely on that.

Strange... to me both behaviors are actually manifestation of the same logic.

Witness the documentation for keys:

Since Perl 5.8.1 the ordering can be different even between different runs of Perl for security reasons (see "Algorithmic Complexity Attacks" in perlsec).

Your example is for different runs. It's clearly documented to not be true\, and my code doesn'T rely on it.

I somewhat understand how you can distinguish the two\, but I can't agree with your reasoning. I guess it's agree to disagree time ;)

Yeah\, fine with me.

Are you sure anybody would consider it a bug to get the same output for the same perl snippet on multiple versions of perl?

For something that has its order declared as "undefined" - yes it is a bug.

Could you explain a bit more why you think that getting the same order is a bug?

To me "Identical orders each time" is a valid implementation of "undefined order". But maybe I read too many standard texts to find that weird.

I can understand the opposite case - order being documented to be identical\, but different each time. That would clearly be a bug.

I would say that's the default expectation\, don't you agree? At least not without good reasons for them to differ.

In my eye the worst situation in computing is when some undefined behavior appears to be "defined" 99.9% of the time.

It leads to problems\, yes\, but "accidentally defined" behaviour is not a bug. Maybe misleading\, annoying\, leading to busg\, but not a bug in itself.

But I guess if you call undocumented but deterministic behaviour some kind of "bug" (in quotes)\, you'd have my disagreement\, usually at least.

This is btw evidenced by the massive amount of CPAN borkage when hash randomization was fixed in 5.17

I think the breakage is evidence for people misinterpreting the amibguous documentation and expecting deterministic behaviour.

For example\, when it was reported as a bug that "func $a\, $b" doesn't always evaluate $a before $b\, it was considered a bug\, and fixed\, but (prove m wrong :) I don't think this is guaranteed anywhere. It certainly is expected perl behaviour though.

Similarly\, ::-string quoting is only mentioned in some perldelta\, and not very clearly. But when found out that code relies on it\, it was kept in.

You see\, unlike C\, which is full of undefined behaviour\, perl is much much more deterministic\, and always has been.

That's a feature\, and used by a lot of code. Not a bug.

so\, while you could likewise argue: "these things are undocumented\, we remove them"\, it would\, again\, be needless breakage (well\, who knows\, I can imagine evalution reordering to allow for better optimisations).

I think that might explain more where I am coming from in expecting deterministic behaviour.

p5pRT commented 11 years ago

From @ribasushi

On Thu\, Mar 21\, 2013 at 06:17:00PM +0100\, Marc Lehmann wrote:

On Fri\, Mar 22\, 2013 at 03:01:31AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote:

With all respect for your work and skill Marc\, I would go a tad out of character and call you a sloppy programmer[1]. Any XS (note\, not fortran not C++) code distributed on CPAN is absolutely expected to compile on *every* compiler that is capable of compiling perl5 blead.

Then we have to disagree on that\, Peter. I don't even have access to most of these compilers\, nor do I have the time to do all that work.

Yes\, but others do.

Again\, I give my code freely\, and asking any kind of XS code to compile with any compiler is just unreasonable.

Not any compiler - a compiler that can compile perl5 (the *current* version of perl5\, not some version from 20 years ago).

C++ needs to compile with a C compiler?

We are talking about C here.

Linux-specific code needs to compile on OS X?

There are ifdef's for that :)

A third-party library that is converted to XS needs to be ported to every platform perl runs on\, even if almost nobody has access to it?

This does not apply for most of your XS work on CPAN except for the async-related stuff. It certainly does not apply to JSON::XS to make a poster-child example.

This is not just a matter of disagreeing\, I would go as far as to say this is so far beyond unreasonable that it's not even serious. Are you really serious? Sorry\, but I find that hard to believe.

I am dead serious that when "// comments" is what prevents your code from compiling somewhere - there is extremely strong expectation (bordering on obligation) for you to fix it. Furthermore the same applies to patches provided by people with access to exotic compilers - if you are sent a patch (using the proper channels you prefer)\, and it does not make a mess out of your code - it is sloppy and irresponsible for you not to apply it.

Or maybe that problem here is the word "expected" - by whom? I certainly don't expect that

This is a pity.

and I know this expectation would be completely unwarranted. Just go to some antique or esoteric platform and compile some of CPAN\, that will heal you of that expectaton quickly.

(You could even start with windows and see how mayn XS modules compile there).

This is a well known problem\, yes. Are you publically stating that you are comfortable being part of that particular problem (general portability of CPAN compiled code)\, and do not see value in being part of the solution?

In any case\, without very good reasons otherwise\, this is what you are going to get from me. You are free to not use my code as result

[1] Note this is not an attack on your character\, this is an attack/characterization of what you distribute on CPAN i.e. your XS work in particular.

I'd be honestly interested on what that opinion is based though. Have you really looked at my XS work? My most popular XS modules are EV\, JSON::XS\, Coro\, Convert-UUlib (mostly not by me\, and admittedly bad code)\, Guard and IO::AIO.

I have spent probably the better part of my life to make these portable and run well (even on hard to port to platforms such as Windows\, which has no reasonably bug free perl port itself). I wasted much of the time during my university studies\, and a large share of my professional work consists of maintaining these modules\, often at a loss of compensation.

This is what my "with all due respect to your coding skill" referred to. Yet even if any and all of these offerings were absolutely brilliant and bug free (and I am sure they come pretty close) - I feel I have the right to label your work sloppy on the fact that you are not C89-safe *alone*. This fact alone is enough. Combine that with your refusal to fix the status quo even after supplied patches only strenghtens my case.

I am afraid that if suddenly that is no longer enough for CPAN\, then you can close shop. I certainly can't see how I could invest even more work\, and I think your accusation of that codebase being by a sloppy programmer is unwarranted.

I honestly invite you to look at my modules and then repeat your judgement.

And I will then just have to tell you that I obviously fail the mark for publishing modules on CPAN\, and with me\, most of the XS authors.

Right - and you fail as stated above. And so do many other authors. And I shall ask again - are you publically stating that you are comfortable being part of that particular problem (general portability of CPAN compiled code)\, and do not see value in being part of the solution?

Any answer is valid btw\, after all we are trading opinions here\, nothing more.

Cheers

p5pRT commented 11 years ago

From @ribasushi

On Thu\, Mar 21\, 2013 at 06:27:51PM +0100\, Marc Lehmann wrote:

On Fri\, Mar 22\, 2013 at 03:10:00AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote:

I guess it's agree to disagree time ;)

Yeah\, fine with me.

\

I think that might explain more where I am coming from in expecting deterministic behaviour.

Yes it does\, and it is in line with how I understood your argument the first time. I will continue to disagree\, however\, whether changing accidental undocumented determinism is as much of a problem as you paint it to be.

In any case - good talk ;)

Cheers

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 09:34:03AM -0700\, "H. Merijn Brand via RT" \perlbug\-followup@perl\.org wrote:

I don't remember it that way\, sorry. It wasn't just minor\, and as people here love to say "where's your patch"? Not that I would likely apply it\, for reasons stated.

If you - in advance - already claim not to apply patches\, there is no use in spending time fixing bugs in your software.

That doesn't make sense to me. You can fix bugs in my software without my applying any patches. Your understanding of free softwrae is fundamentally flawed. I couldn't even stop you if I wanted.

Of course\, I do basically always fix bugs in my softwware. What you wanted me to do is port the code to your antique compiler when a working compiler is available.

Really\, comparing apples to oranges like that is dishonest.

C99 code not compiling on a C90 compiler is not a bug just as much as a module that requires 5.8 not compiling with 5.6 is not a bug.

I - of course - am convinced that you think your code has no bugs at

Your conviction is\, however\, based only on the evil strawmen version of me in your mind. A look at the Changes file for JSON::XS should have shown you that this is not so:

- fix a bug in the initial whitespace accumulation. - apparently JSON::XS was used to find some bugs in the JSON_checker testsuite\, so add (the corrected) JSON_checker tests to the testsuite. - fix a bug in decode filters calling ENTER; SAVETMPS; one time too often. - fix a memory overflow bug when indenting. - fix a bug in and optimise canonicalising fastpath (reported by Craig Manley). - the "could not sleep without debugging release". it should basically work now\, with many bugs as no production tests have been run yet.

I clearly accept the possibility of my code having bugs. In fact\, a major reason to publish the code is so people can find bugs.

Redefining bug as "anything that doesn't work with my compiler version" is fun only to some extent.

Right\, asking people to write their C++ or Fortran code in C is not reasonable\, because you say so. Your statement is wrong in general\, and wrong in particular.

You seem to be blaming anyone else as being "wrong" everywhere on this list.

Only if you lack knowledge of the contents of my mails and make them up. Just as with your bug comment above.

Making things up doesn't make them real.

Do you ever agree with anyone? ever?

A lot\, as would be trivial for you to verify.

Are you even *able* to admit that someone else is right?

Clearly I am\, as reading my mails would tell you. I do not admit I am wrong when I am not wrong though\, or when the only arguments I get to change my mind is insults and aspersions (a new word I learned toay\, I think I will not use it often).

Again\, making things up is not making them real.

But well\, you are entitled to your opinion and not using my module\, but what you do is way out of proportions. But again\, your choice.

And unless/until you conform to the requirements of perl/XS to C89

There are no such requirements\, as you clearly know.

Making things up doesn't make them real.

When did I ever threaten you?

In your mail merijin.

You don't even take time to correctly write my name.

Happens to me all the time\, on this list (usually it's "Mark"\, and my name is easier for english speakers than your).

I am sorry that I misspelt your name\, I did actually check\, but I sitll got it wrong. In the future\, I will likely remember and not repeat the mistake.

I do hope you don't suffer too much from that.

But please point out any "threat" I posted.

I alread did.

By quoting the text (and context if needed). Maybe you - again - misread something simple as threat.

Maybe. But it's clear you have a person problem with me\, and try to execute some small kind of vendetta against me.

I have no reason to threaten you.

Of course you hage\, you want me to work for free in porting my stuff to your platform\, while insisting not to upgrade.

I'm just explaining why I don't use your code. I won't kill or chase you for it.

I never thought you would. Not physically at least. However\, I do consider you entering this thread while only saying you don't use my software and ignoring any topic of this thread as chasing me.

I feel completely physically safe from you\, if you mean that. Because I feel you will not harm me physically. I totally trust you in that.

But that's about it. Character assassination by making up stuff publicly is obviously not beyond you.

I can easily live without it.

Not sure.

And the arrogance of trying to threaten me into doing your work for free is breathtaking.

Hrmph. IIRC

You don't.

Excuse me?

IIRC usually means "If I remember correctly"\, to which I replied "You don't."

If you mean something else with IIRC\, then really I have no clue what.

my quest came with a (simple) patch. there was NO work for

It did not.

See https://rt.cpan.org/Public/Bug/Display.html?id=42462 Second post. It might not be in the format/style you prefer\, but it IS a solution to the problem posted.

I don't think it is a solution\, and it is not a patch. It doesn't even have instructions on how to fix the problem\, which still wouldn't be a patch\, but at least some form of instruction\, which could remotely be considered a patch.

__STDC__ is ANSI and has nothing to do with weird compilers or operating systems.

Ah\, I see now. You meant me to add defined(__STDC__) and destroy performance everywhere.

Sorry\, but reverse engineering code that looks like cut&paste is still not a patch. And even as a patch it would break the module\, whose stated purpose is to be fast - an malloc every hash doesn't cut it.

In any case\, better late than never\, but claiming you sent a patch is still bullshit. You didn't.

And making up bullshit like "it was NO work AT ALL" doesn't make it better. It would have been no work for you\, but it would be continued work for me\, because I take maintaining my package seriously.

just to a certain extent

Correct\, the the extent I am willing and capable to. Which is very in both cases. But I have to ste a cut-off\, and if you don't like where I set the cut-off point\, you cna complain\, but thats it.

Oh right\, you can even lie about it\, too.

If it was no work at all\, why didn't you do it? It's free software after

Because if I would install it *with* a patch\, I'd have to jump through hoops to do it again and again and again on updates: now THAT is work I don't want.

Maybe you can understand me then: you are too lazy (and probbaly incapable) of fixing it properly\, I am too lazy to do it for you.

The lack of respect you show for the work others have done for you to use\, freely is staggering.

/me laughs out loudly you start making me feel better with every post

Good for you...

1. I have no pet compiler

You have three of them then\, difference is immaterial. In all three cases there exist easily available upgrades that do not force you to take anything but a speed hit for JSON::XS.

And all your suggestions boil down for everybody else taking a speed hit\, at best\, and breaking code\, at worst.

2. I don't care about compilers\, but about standard (C89 in particular) in conjunction with my beloved programming language perl

Well\, you can't have it. There have been two major revisions since then\, and support for these language fetaures are available for basically two decades now.

I don't stop others from writing code for a 20 year outdated standard\, but forcing me is not going to work.

All you cna do is not use my code. Complaining for something you get for free without any hooks is already beyond arrogant.

But it really is my decision on where I set the cut-off point

Absolutely! I agree completely! But then state so when you reject sensible patches instead of refusing them as wrong

I always do...

Making things up doesn't make them true.

Well\, ok\, maybe not really always\, but barring unintented and rare mistakes from my side\, I basically always do.

Don't confuse "reasonable patches" with "I didn't send a patch". There are people who are capable of sending reasonable patches\, and while you might be one of them\, you haven't done so with me.

of how antique language standards I support\, and you are entitled to do it better\, or not use my stuff. Complaining like you do is just whining.

Not really\, but if that is how you see it\, feel free

I do.

What's next\, I need to support perl 5.001?

why? it is officially out of support. Even 5.005 is. Support for 5.6.2 would be more than welcome\, but if you choose to take 5.8.7 as minimum\, that'd be fine with me. It is your choice.

There are lots of modules that do not work with 5.8 either. Stop whining and upgrade\, or do the work.

/me wonders what perl versions have to do with compiler standards\, but

perl 5.12 is out of support\, but lots of people use it. 5.8 is out of support for five years.

Some people still use it. Some people I hear of still use some of my modules (backporting them even) to perls as old as 5.6.

I do what is reasonable and what I can to support them.

Nothing forces me to it\, and sure enough\, if a person acts as arrogant as you\, I am not inclined to work for them. Why should I.

It's really the same as antique compilers. GCC did have these extensions many many years before C99\, and I have no reaosn to expect that code won't compile with pre-C99 versions of many compilers. The extensions I use are very common\, and I don't use any C99 extensions not in common use.

I have no obligation to you. At all. You really need to understand that. I might still try\, but I rather waste my time on supporting nice people\, which exist in abundance.

maybe that is on-topic too

No\, all your whining and bullshitting is off-topic.

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 10:28:40AM -0700\, Leon Timmermans via RT \perlbug\-followup@perl\.org wrote:

I almost didn't read your mail because of the unnecessary full quote. Please properly format your mails or I will not be able to respond.

Marc\, can you please stop doing this.

Doing what exactly? Point out problems with perl?

No\, I really can't\, or at least it is hard for me. Perl is important for me\, and seeing needless breakage does make me act.

If you don't want me to do that\, you should tell andk not to tell me about breakages\, and never mail me.

Because if that happens\, I will respond.

Obviously you're not seeing why people react so combative to you\,

I can only speculate\, the same as you.

given how consistently this happens to you (and not to most other

I don't know what consistently means\, but I know it happens etxremely rarely.

I do a lot of e-mail - it is my mail commmunications device. I get many dozens of personal mails per day\, and almost never\, if ever\, do I run into problems.

The only consistency I see is the overwhelmingly good responses I get\, with rare occurances like these\, certainly not caused by me.

I understand it can piss of people when you are honest\, but all in all\, that's a good thing\, at least in my experience.

Or maybe I don't - it really is so rare that I can't see any patterns.

people)\, don't you think you're the key player in that dynamic?

Nothing gives me that impression. How would I see it when everything works great most of the time.

Really\, I am in a sea of feel good (cough\, obviously I have bad days\, like everybody else)\, and I am not surprised if sometimes I can't get along with somebody\, that's only naturally.

Seriously\, I get so much feedback on my software\, and it's often constructive\, often naive (you know\, users\, but often they point at shortcomins in the documentation of interesting but failed expectations etc)\, but nothing like this: ignoring all arguments\, just repeatedly saying "you are wrong\, because I say so".

I must say that to me\, it overwhelmingly looks as if this attitude of perl5porters is the problem.

It's the reason I stopped contributing - the atmosphere on this list is so unconstructvie\, with so many people who have nothing to contribute bossing others around.

(I didn't know I reply to perl5porters btw.).

You consistently manage to get upset people\, and in the end yourself.

I much more consistently make people feel thankful for what they got from me\, and contribute useful stuff.

If it really is a problem with me\, you should be able to point out what it is.

Perhaps you could consider delegating these discussions to someone you trust who can communicate with p5p more effectively.

I didn't have bad experiences\, even with p5p\, for quite some time. I remember bad times from the past\, but overall\, not many.

All _I_ ask is for people to be reasonable. At the very least\, if you have no arguments\, then don't post. Just saying "you are wrong\, becasue I say so" simply doesn't cut it for me.

This is different to some lazy chat - I replied to a bug report\, and you can criticise me as much as you want\, but I do expect you to actually have reasons for generating extra work for me.

p5pRT commented 11 years ago

From @doughera88

I think this thread has passed its useful limit and respectfully suggest that everyone just drop it.

Yves fixed an important bug in perl's hash behavior. Unless someone comes up with a better patch that addresses the security issues\, this fix is not going to be reverted.

-- Andy Dougherty doughera@lafayette.edu

p5pRT commented 11 years ago

From @tux

On Thu\, 21 Mar 2013 19:01:03 +0100\, Marc Lehmann \schmorp@schmorp\.de wrote:

But well\, you are entitled to your opinion and not using my module\, but what you do is way out of proportions. But again\, your choice.

And unless/until you conform to the requirements of perl/XS to C89

There are no such requirements\, as you clearly know.

Making things up doesn't make them real.

Just taking this bit\, as the rest of the crap is not worth spilling my time on.

$ perl -ne'39..43 and print' INSTALL Building perl from source requires an ANSI compliant C compiler. A minimum of C89 is required. Some features available in C99 will be probed for and used when found. The perl build process does not rely on anything more than C89.

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Fri\, Mar 22\, 2013 at 04:35:16AM +1100\, Peter Rabbitson \rabbit\-p5p@rabbit\.us wrote:

Again\, I give my code freely\, and asking any kind of XS code to compile with any compiler is just unreasonable.

Not any compiler - a compiler that can compile perl5 (the *current* version of perl5\, not some version from 20 years ago).

All I said took that as assumption - blead\, as you said\, in fact.

C++ needs to compile with a C compiler?

We are talking about C here.

Then say C\, not XS.

Linux-specific code needs to compile on OS X?

There are ifdef's for that :)

Explain how ifdefs help Linux::DVB compile on windows\, and what the gain would be.

Besides\, I really suspect you didn't look at my code. My code is full of #ifdefs for platforms and other things.

But except for a few cases where indeed somebody else maintains code for another platform\, I cannot (and don't want) maintain code for platforms that are extremely different\, very old\, inaccessible to me\, and for which a perfectly reasonable way to get my modules working exists.

A third-party library that is converted to XS needs to be ported to every platform perl runs on\, even if almost nobody has access to it?

This does not apply for most of your XS work on CPAN except for the async-related stuff.

I am not aware of any async-related stuff that uses third-party libraries.

The closest I can imagine is EV (nothing sync) and IO::AIO (async)\, both of which use only libraries I maintain.

The only case where _I_ use third-party code is Convert::UUlib\, and because I was nicely asked to\, I took over maintainerhsip of what I found to be a very bad code base\, and fix a lot of security bugs.

These security bugs benefitted me only theoretically\, I did it because I want to have the module be of high quality\, and to help others.

It certainly does not apply to JSON::XS to make a poster-child example.

Right\, but which code did you actually look at? And of course\, JSON::XS is what I consider a good code base that is very portable. Witness cpantesters\, which I think is a reasonably good tets of portability:

http://static.cpantesters.org/distro/J/JSON-XS.html

This is not just a matter of disagreeing\, I would go as far as to say this is so far beyond unreasonable that it's not even serious. Are you really serious? Sorry\, but I find that hard to believe.

I am dead serious that when "// comments" is what prevents your code from compiling somewhere - there is extremely strong expectation (bordering on obligation) for you to fix it.

Ok\, there we indeed have to agree to disagree. If somebody rugently needs to port my code to a 20 year old compiler\, he's got to do the work.

Besides\, that's not what kept JSON::XS working of merjin's antique boxes.

Furthermore the same applies to patches provided by people with access
to exotic compilers -

Well\, in the JSON::XS case\, I didn't get any such patch.

But you are right\, I am wary of applying such patches\, becasue I cannot guarantee that they work. It's fine in trivial cases that I ca reasonably get right\, but not in others.

if you are sent a patch (using the proper channels you prefer)\, and it does not make a mess out of your code - it is sloppy and irresponsible for you not to apply it.

I agree. But what I want to know is where _I_ was sloppy and/or irresponsible.

Or maybe that problem here is the word "expected" - by whom? I certainly don't expect that

This is a pity.

Taught by reality - you would think that my primary platform (debian gnu/linux amd64) should be a dream with respect to support\, but in reality\, there are a lot of modules that do not compile\, to the extent of my not doing make test\, because chances of it failing are >>10%\, making autmated module installs with testing impossible.

(You could even start with windows and see how mayn XS modules compile

This is a well known problem\, yes. Are you publically stating that you are comfortable being part of that particular problem (general portability of CPAN compiled code)\, and do not see value in being part of the solution?

Peter\, you start to piss me off\, for sure.

I asked you to look at my code. I maintain an antique activestate windows 2000 vm\, and a win7 vm with multiple strawberry perl and activestate perl versions (and cygwin perl on both to boot and a variety of msvc and mingw compilers) and have automated tests run on them to support this platform which means almost nothing to me\, and which is extremely hard to support because of the very low quality of the available perls.

The amount of work I put into supporting these windows platforms in Coro and IO::AIO is staggering.

You can even verify the reports for these platforms on cpan-testers.

And now some you come\, completely clueless about my code and how portable it is\, call it sloppy\, and wondering if I am a problem for bad module availability on windows.

Are you making fun of me? For heavens sake\, go and finally have a look at my code and it's portability\, and then make an informed judgement.

I have spent probably the better part of my life to make these portable and run well (even on hard to port to platforms such as Windows\, which has no reasonably bug free perl port itself). I wasted much of the time during my university studies\, and a large share of my professional work consists of maintaining these modules\, often at a loss of compensation.

This is what my "with all due respect to your coding skill" referred to. Yet even if any and all of these offerings were absolutely brilliant and bug free (and I am sure they come pretty close) - I feel I have the right to label your work sloppy on the fact that you are not C89-safe *alone*.

And which of my modules that you looked at are not "C89-safe"?

Tell you what\, most of them do compile with C89 compilers\, because I do care\, but you clearly didn't even care to have a look and make an informed judgement.

Combine that with your refusal to fix the status quo even after supplied patches only strenghtens my case.

Peter\, you really piss me off now.

Just do your homework. What you claim is a lie\, nothing else. There was no refusal to fix C99 comments that caused a problem.

I must say\, the misinformation campaign by Merjin does pay off for him.

In any case\, my expecttaion for you is to do the research I asked you for.

And I will then just have to tell you that I obviously fail the mark for publishing modules on CPAN\, and with me\, most of the XS authors.

Right - and you fail as stated above.

Wow. But that just means you are asking for too much\, I am sorry. I am not willing to work full time just to satisfy your unreasonable expectations\, certainly not when you base your judgement on fantasies without actually looking at my code.

I shall ask again - are you publically stating that you are comfortable being part of that particular problem (general portability of CPAN compiled code)\, and do not see value in being part of the solution?

Obviously I am not\, but that doesn't help with people like you who can't even be bothered to look at the stuff they criticise. I asked you to look at these modules\, but you didn't even look at JSON::XS. That didn't keep you from making false claims about it.

Any answer is valid btw\, after all we are trading opinions here\, nothing more.

Even asking me that question shows you are not interested in fairness - I asked you to look at my more popular XS modules\, and judge them\, but you didn't\, and instead just repeated some lies from others.

I honestly expected more of you\, but as expectations go...

In any case\, I urge you to actually look at my code before implying I am part of some portability problem.

And maybe\, just maybe\, do a similar XS module\, just one. Maybe you can appreciate the staggering amount of work that goes into making them compile on dozens of platforms.

p5pRT commented 11 years ago

From schmorp@schmorp.de

On Thu\, Mar 21\, 2013 at 11:45:30AM -0700\, "H. Merijn Brand via RT" \perlbug\-followup@perl\.org wrote:

And unless/until you conform to the requirements of perl/XS to C89

There are no such requirements\, as you clearly know.

Making things up doesn't make them real.

Just taking this bit\, as the rest of the crap is not worth spilling my time on.

Your claim was for perl/XS\, not perl. You can take it back or back it up (good luck with that).

In case you are too dense to get it: JSON::XS is not part of perl.

p5pRT commented 11 years ago

From @deven

I'll probably regret responding at all\, but here goes... Yes\, I've read this entire thread already. I'll try to leave out as much as I can...

On Thu\, Mar 21\, 2013 at 1:04 PM\, schmorp@schmorp.de \< perlbug-followup@perl.org> wrote:

I just looked up the meaning of "same" at thefreedictionary.com.

First meaning it lists is "identical". Second meaning it lists is "similar in kind\, quality\, quantity\, or degree". (there are two more).

Just because a word has multiple meanings does not make them synonymous. The context makes it fairly obvious that the "same hash" means the same underlying data structure\, not similar ones. Yes\, your interpretation is plausible given the possible meanings of "same"\, but most programmers would interpret the verbiage in the documentation as a very clear and unambiguous warning not to rely on the hash ordering in general\, except as explicitly supported.

Or you disagree with what "same" means? It seems arguing against

dictionaries is unproductive\, though\, it leads nowhere (except you being wrong in this case\, but it's still pointless...).

Who is arguing against the dictionary? The documentation used "same" in its first meaning of "identical"; the other possible meanings of the word are irrelevant.

keys(%hash)

returns items in a different order than values(%hash) does then please file a bug.

So suddenly it is consistent. This is all very confusing to me.

This is a clearly-supported exception to the general rule that you shouldn't rely on the hash ordering. And I don't think you're confused on this point at all\, you just don't like the answer you've been given.

I know how it's meant\, but I also know what is said\, and assuming that

"the same hash will result in the same ordering" is not unreasonable.

The docs say "the same hash unmodified"\, so that implies very strongly that it means identity.

Well\, not to me\, and the other person I asked (I am too lazy to ask a bigger control group\, sorry).

Let me add to your control group\, then. I would say that the context very strongly implies "identical" even *without* mentioning "unmodified". The "unmodified" is a further warning that you can no longer rely on any consistency in the hash ordering after making ANY change to it\, and that adding a new key to the hash might completely resequence the ordering in any arbitrary way.

Saying it changes in future versions but is the same for the same hash

STRONGLY implies that the ordering is the same for the same version of

perl\, which is already not true.

No it doesn't.

Why mention it at all then? If the ordering is randomly changing\, then why mention it might change in future versions? It frankly makes no sense.

It makes perfect sense. The documentation reserved the right for Perl to change the underlying hash implementation at any time for any reason\, as long as the supported use cases remained supported. It was quite clearly warning not to rely on current behavior.

perlfunc.pod didnt exist prior to that commit. And it clearly changed\, if you took notice. Not in any way relevant to this discussion\, if you took notice.

Your point being that it is a documentation bug\, that the existing documentation is ambiguous or what?

The oldest quoted documentation was somewhat ambiguous\, and it was clarified fairly quickly. What you can and can't rely on in the semantics of hash ordering have been quite clear for 15 years now.

I am not claiming your patch breaks documented behaviour (for every

interprettation). I am claiming the code in question doesn't rely on undocumented behaviour.

Sorry\, but there's only one correct interpretation of "same" in the context of that warning in the documentation. Don't expect to get much traction by "language lawyering" on this point.

Relying on undocumented behavior would actually be in your favor\, since the behavior of the Perl interpreter is presumed correct in the absence of documentation to the contrary. If you're relying on similar hashes to behave the same as identical hashes\, then you are ignoring a clear warning in the documentation and it's only by luck that your code has worked until now.

You'd have to make the stronger claim that no other interprettaion is reasonably possible\, and I don't think you can\, simply because "same"\, in english\, is ambiguous as to whether mean identity or equality.

There is nothing here to argue about\, you are wrong\, and there your interpretation is wrong\, and that is all there is to it.

You seem to be unable to explain why it would be wrong\, while I gave ample evidence.

Despite the fact that "same" can sometimes mean "similar"\, in this context\, it clearly meant "identical". You can make the argument\, but your interpretation is still wrong.

Not much further down\, you say:

Your interpretation is probably a valid one and I wouldn't gfault you if

you made assumptions based on it\, but after I told you it isn't\, whats the point of arguing further?

You might want to take your own advice here and give up arguing about the definition of "same".

Hey\, I checked a few dictionaries\, and listing "similar" as equal meaning

to "identical" is the norm.

Again\, you are just claiming english is wrong and you are right. That's... well\, not very useful.

English is ambiguous. That's the nature of human languages\, because the listener generally has the intelligence to disambiguate between several possible meanings of a word based on the context in which it is used. Yes\, "identical" and "similar" are both possible meanings of "same"\, but in context\, the "identical" meaning was clearly the correct interpretation.

The claim that the same hash results in the same ordering is ambiguous w.r.t. the meaning of what "same hash" means.

Identity is one meaning\, but the others are there.

Are you arguing that there's a bug in the hash implementation or a bug in the documentation?

There is no reasonable interpretation of

the documentation that supports your position.

There certainly is\, I gave it\, and ignoring it doesn't make it go away.

You gave a possible interpretation. But if you ask most programmers to read that entire paragraph and decide which interpretation is more reasonable\, I bet the overwhelming majority would say "identical" is a far more reasonable interpretation than "similar"\, in that context.

You are rude and insulting in your manner\, you should not be surprised when people you communicate with are combative.

So where was I rude? Where was I insulting?

If you can't see it\, that's sad. Bystanders can see it. Yves\, by contrast\, comes across as remarkably patient and reasonable\, despite provocation.

It says the ordering is the same for the same hash.

And it is. (So long as you dont modify it by inserting new items.)

Experiments show you are wrong.

You'll have to prove this assertion to be believed.

Changing the hash function doesn't break the code. Changing the hash

function within the same version of perl\, and within the same process\, is what changes it.

We don't change the hash function during the process.

Well\, the observable effects are different\, which is the point. You are splitting hais a lot again\, when it's totalyl unnecessar\< because I have made explicit what the problem is\, namely the breakage of JSON::XS.

Even if Perl did randomly select a different hash function every time you modify the hash\, that would be perfectly valid and consistent with the documentation. You rely on undefined behavior at your own peril.

So if the ordering is really the same for the same hash\, within the same

process (using the same perl)\, then I am indeed totally wrong.

Is that the case?

@keys1= keys %hash; @keys2= keys %hash;

will always return identical @keys1 and @keys2 assuming %hash is a "normal" perl hash.

I call that a consistent ordering\, you don't.

This is the sort of consistency that the documentation says you may rely on. That doesn't mean that hash ordering will be consistent in all the ways that you want it to be.

I think you have a totally unworkable definition of consistent. At least

that is clear now\, and explains a few things.

What I mean with consistent is that the ordering is the same\, within limits.

And it is. The limits are clear -- if it's the same (identical) hash object\, and you don't modify it\, then a consistent ordering will be returned. Outside those limits (modified hash\, similar but not identical hash)\, all bets are off. It may be consistent\, but it doesn't need to be. You were warned 15 years ago.

The documentation simply does not say what you seem to think it says.

That's beside the point\, you said my interpretation is unreasonable (and other things)\, and I say it was\, and gave evidence on why.

I am also not the only one who interpreted it that way\, as evidences by the breakage that ensued after both stages of patches in that area.

That doesn't prove the reasonableness\, of course.

I don't know I would go so far as to describe your interpretation of "similar" as unreasonable\, but the correct interpretation of "identical" is obviously MORE reasonable.

If it did it would say it explicitly\,

It says it explicitly\, but unfortunately\, in an ambiguous way.

In context\, I don't think it's ambiguous at all. How many people truly find it confusing?

I think it should be possible to write correct code by documentation and experimenting alone\, but we have to have to disagree on that point.

I agree with you on this point. However\, the documentation is quite clear about the limits of relying on hash ordering\, and the documentation takes precedence over experimentation here.

But the bug is in the perl core. Nothing I apply can fix that\, becasue I

can't apply fixes to the perl core.

Excuse me? No the bug is in your code\,

No\, it's in the core. It was introduced recently\, and apparently by your patch (I blindly trust andk here).

Changes to behavior which is explicitly documented as undefined don't qualify as bugs\, even if they break code that relied on that undefined behavior. If you relied on the hash ordering of similar but not identical hashes\, then the bug is in your code\, even if it took this patch to expose it.

expecting to snapshot a serialized hash and have that be the same as some other serialized hash with the same keys.

I expect the same hash to have same ordering within one program run\, yes.

Which definition of "same" are you using here?

Deven

p5pRT commented 11 years ago

From @rjbs

* "schmorp@schmorp.de" \perlbug\-followup@perl\.org [2013-03-21T04:28:54]

- I'd say this simply breaks perl\, as perl documented the order is fixed\, at least between runs (I would argue even the earlier fix to randomise between runs is a bug). Sure\, the wording is ambiguous\, but that doesn't make it any better.

Unlike many other respondents to this thread\, I agree that the wording is imperfect\, and that it is not ludicrous to imagine reading the documentation for "each" and coming away thinking that the order is going to be the same between two hashes with the same contents in one run of perl. I don't think it's the most likely reading\, but of course I have no research on the topic… let's call that a guess\, and throw it out.

So\, there are two readings possible here\, hinging in part on the meaning of the phrase "same hash." Either:

a. a hash with the same contents always orders keys/value in one way or b. a particular hash var always does this\, with no bearing on other hashes

As Yves showed\, it's already the case (in some cases) that two hashes with the same contents are not ordered the same:

my %hash=(13=>1\,19=>1); say join " "\, keys %hash; my %copy= %hash; say join " "\, keys %copy;

If the (a) reading was correct\, the fact that this program emits two different lines would be a bug. If (b)\, not.

As it stands\, it falls to me to decide which reading is canonical. The answer is (b). In my experience\, this has been the understanding on list for as long as I can remember\, and was repeated a good bit when the 5.8.1 ordering changes were made.

This work has been in progress for quite some time\, and has been thoroughly discussed\, and it is going to stay in.

-- rjbs

p5pRT commented 11 years ago

The RT System itself - Status changed from 'new' to 'open'