grep autovifies hash elements

[p5pRT]

Migrated from rt.perl.org#40194 (status was 'rejected')

Searchable as RT40194$

[p5pRT]

From perlbug@schmorp.de

Created by perlbug@schmorp.de

grep "autovifies" hash elements​:

  perl -MData​::Dumper -e 'grep 1\, $hash{key}; warn Dumper \%hash'

shows that $hash{key} is undef when it should not exist at all. The same is true for map and for.

While I can make an educated guess at _why_ this happens (all of map\, grep and for alias $_ to the element)\, it doesn't seem to be documented (the documentation for grep talks only about modifications to $_\, but $_ isn't being modified in the example above).

I think either the documentation should be improved to mention this\, or\, if easy (no idea\, sorry)\, grep/map/for should not autovivify.

Perl Info

``` Flags: category=core severity=low Site configuration information for perl v5.8.8: Configured by Marc Lehmann at Sat May 13 16:26:09 CEST 2006. Summary of my perl5 (revision 5 version 8 subversion 8 patch 28184) configuration: Platform: osname=linux, osvers=2.6.16.9, archname=amd64-linux uname='linux cerebro 2.6.16.9 #1 smp thu apr 27 08:42:39 cest 2006 x86_64 gnulinux ' config_args='-Duselargefiles -Dxxxxuse64bitint -Uuse64bitall -Dusemymalloc=y -Dcc=gcc -Dccflags=-DPERL_DONT_CREATE_GVSV -ggdb -Dcppflags=-DPERL_DONT_CREATE_GVSV -D_GNU_SOURCE -I/opt/include -Doptimize=-O4 -march=opteron -mtune=opteron -funroll-loops -fno-strict-aliasing -Dcccdlflags=-fPIC -Dldflags=-L/opt/perl/lib -L/opt/lib -Dlibs=-ldl -lm -lcrypt -Darchname=amd64-linux -Dprefix=/opt/perl -Dprivlib=/opt/perl/lib/perl5 -Darchlib=/opt/perl/lib/perl5 -Dvendorprefix=/opt/perl -Dvendorlib=/opt/perl/lib/perl5 -Dvendorarch=/opt/perl/lib/perl5 -Dsiteprefix=/opt/perl -Dsitelib=/opt/perl/lib/perl5 -Dsitearch=/opt/perl/lib/perl5 -Dsitebin=/opt/perl/bin -Dman1dir=/opt/perl/man/man1 -Dman3dir=/opt/perl/man/man3 -Dsiteman1dir=/opt/perl/man/man1 -Dsiteman3dir=/opt/perl/man/man3 -Dman1ext=1 -Dman3ext=3 -Dpager=/usr/bin/less -Uafs -Uusesfio -Uusenm -Uuseshrplib -Dd_dosuid -Dusethreads=undef -Duse5005threads=undef -Duseithreads=undef -Dusemultiplicity=undef -Demail=perl-binary@plan9.de -Dcf_email=perl-binary@plan9.de -Dcf_by=Marc Lehmann -Dlocincpth=/opt/perl/include /opt/include -Dmyhostname=localhost -Dmultiarch=undef -Dbin=/opt/perl/bin -des' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=define use64bitall=undef uselongdouble=undef usemymalloc=y, bincompat5005=undef Compiler: cc='gcc', ccflags ='-DPERL_DONT_CREATE_GVSV -ggdb -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/opt/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O4 -march=opteron -mtune=opteron -funroll-loops -fno-strict-aliasing', cppflags='-DPERL_DONT_CREATE_GVSV -D_GNU_SOURCE -I/opt/include -DPERL_DONT_CREATE_GVSV -ggdb -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/opt/include' ccversion='', gccversion='4.0.4 20060422 (prerelease) (Debian 4.0.3-2)', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='gcc', ldflags ='-L/opt/perl/lib -L/opt/lib -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-ldl -lm -lcrypt perllibs=-ldl -lm -lcrypt libc=/lib/libc-2.3.6.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.3.6' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -L/opt/perl/lib -L/opt/lib -L/usr/local/lib' Locally applied patches: MAINT27284 @INC for perl v5.8.8: /root/src/sex /opt/perl/lib/perl5 /opt/perl/lib/perl5 /opt/perl/lib/perl5 /opt/perl/lib/perl5 /opt/perl/lib/perl5 /opt/perl/lib/perl5 /opt/perl/lib/perl5 . Environment for perl v5.8.8: HOME=/root LANG (unset) LANGUAGE (unset) LC_CTYPE=de_DE.UTF-8 LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/root/s2:/root/s:/opt/bin:/opt/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/X11/bin:/usr/games:/root/src/uunet:. PERL5LIB=/root/src/sex PERL5_CPANPLUS_CONFIG=/root/.cpanplus/config PERLDB_OPTS=ornaments=0 PERL_BADLANG (unset) PERL_UNICODE=EAL SHELL=/bin/bash ```

[p5pRT]

From @davidnicol

On 8/17/06\, via RT Marc Lehmann \perlbug\-followup@​perl\.org wrote​:

grep "autovifies" hash elements​:

perl -MData​::Dumper -e 'grep 1\, $hash{key}; warn Dumper \%hash'

shows that $hash{key} is undef when it should not exist at all. The same is true for map and for.

It is well-documented that hash elements are autovivved whenever they are so much as thought about. the exists() function exists to examine a hash entry that might not be there\, and tread softly while doing it.

Consider​: perl -MData​::Dumper -e 'grep exists($hash{$_})\, qw/key/; warn Dumper \%hash'

-- David L Nicol A fundamental lack of underlying cuteness

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From schmorp@schmorp.de

On Fri\, Aug 18\, 2006 at 01​:07​:41PM -0700\, David Nicol via RT \perlbug\-followup@​perl\.org wrote​:

is true for map and for.

It is well-documented that hash elements are autovivved whenever they are so much as thought about.

This is not a very helpful response​: I highly doubt that this is documented\, especially the "thought" part. And if it is documented\, where is that?

Actually\, autovivification is the process of creating a reference where no such thing was before. I used it in a broader sense meaning creating a value where no value existed before during a read access.

I doubt that _reading_ from a hash is documented as being the same as storing undef in it. It does not happen in practise anywhere else\, and I think it is not expected by programmers.

Thats why I think either it should be fixed\, or documented that grep/for/map create hash elements given as parameters\, unlike other parts of the language\, even when you do not modify or even access them.

Note that other forms of referencing such an element does not create it either​:

  my $value = $hahs{key}; # never stores undef in $hash{key}   \$hash{key} # never stores undef in $hash{key}

  my $x = \$hash{key};   print $$x; # never stores undef in hash{key}

So if '"thinking" about a hash elements creates it' were documented anywhere\, it would be wrong\, as it does not happen in the majority of cases.

the exists() function exists to examine a hash entry that might not be there\, and tread softly while doing it.

Please note that "grep exists" is not the same as the example\, nor the same as "grep $_\, ..." nor the same as "for (...)".

"exists" does something else than for/grep/map entirely\, they are not comparable.

Consider​: perl -MData​::Dumper -e 'grep exists($hash{$_})\, qw/key/; warn Dumper \%hash'

I can consider it\, but it has nothing to do whatsoever with the example\, nor with most other uses of grep\, nor with the problem at hand.

--   The choice of a   -----==- _GNU_   ----==-- _ generation Marc Lehmann   ---==---(_)__ __ ____ __ pcg@​goof.com   --==---/ / _ \/ // /\ \/ / http​://schmorp.de/   -=====/_/_//_/\_\,_/ /_/\_\ XX11-RIPE

[p5pRT]

From @davidnicol

On 8/18/06\, Marc Lehmann \schmorp@​schmorp\.de wrote​:

I can consider it\, but it has nothing to do whatsoever with the example\, nor with most other uses of grep\, nor with the problem at hand.

what's the problem at hand? The example didn't seem to do anything but demonstrate the documented behavior\, apparently written bu someone perceiving the documented behavior to be a problem\, perhaps because he did not know about exists(). Exists allows determinig if an element is in a hash or not\, so you can avoid autovivving it when you don't want to.

The problem is unwanted autoviv in hash looking-ups\, yes?

-- David L Nicol A fundamental lack of underlying cuteness

[p5pRT]

From @nwc10

On Fri\, Aug 18\, 2006 at 03​:07​:08PM -0500\, David Nicol wrote​:

On 8/17/06\, via RT Marc Lehmann \perlbug\-followup@​perl\.org wrote​:

grep "autovifies" hash elements​:

perl -MData​::Dumper -e 'grep 1\, $hash{key}; warn Dumper \%hash'

shows that $hash{key} is undef when it should not exist at all. The same is true for map and for.

It is well-documented that hash elements are autovivved whenever they are so much as thought about. the exists() function exists to examine a hash entry that might not be there\, and tread softly while doing it.

Wrong.

$ perl -wle 'sub foo {print defined $_[0]; } foo($hash{key}); print keys %hash'

$

2 blank lines. The key is not autovivified if it's used as a subroutine argument. This is achieved with lots of effort through PVLVs and magic​:

$ perl -MDevel​::Peek -e 'sub foo {Dump $_[0]; } foo($hash{key});' SV = PVLV(0x807f038) at 0x804d16c   REFCNT = 1   FLAGS = (GMG\,SMG)   IV = 0   NV = 0   PV = 0   MAGIC = 0x80865d8   MG_VIRTUAL = &PL_vtbl_defelem   MG_TYPE = PERL_MAGIC_defelem(y)   MG_FLAGS = 0x02   REFCOUNTED   MG_OBJ = 0x804d2c8   SV = PVIV(0x804f82c) at 0x804d2c8   REFCNT = 1   FLAGS = (POK\,pPOK)   IV = 0   PV = 0x8050828 "key"\0   CUR = 3   LEN = 4   TYPE = y   TARGOFF = 0   TARGLEN = 1   TARG = 0x807cf4c   SV = PVHV(0x8052720) at 0x807cf4c   REFCNT = 2   FLAGS = (SHAREKEYS)   IV = 0   NV = 0   ARRAY = 0x0   KEYS = 0   FILL = 0   MAX = 7   RITER = -1   EITER = 0x0

I assume that the same approach could be used by grep for not-yet-existing hash keys.

Nicholas Clark

[p5pRT]

From chromatic@wgz.org

On Friday 18 August 2006 15​:16\, David Nicol wrote​:

The example didn't seem to do anything but demonstrate the documented behavior

Complete nonsense!

Contrary to your claim\, "merely thinking about" a hash key does *not* autovivify it​:

  $ perl -Mstrict -Mwarnings   my %hash = ( bar => 1\, baz => 1 );   print "True\n" if $hash{foo};   print join(' '\, keys %hash)\, "\n";

Marc's code\, while clearly weird (to demonstrate his suspected bug)\, produces bizarrely different behavior that no amount of documentation (besides "This is a bug to difficult to fix\, so avoid it if possible!") can convince me to accept​:

  $ perl -Mstrict -Mwarnings   my %hash = ( bar => 1\, baz => 1 );   print "True\n" if grep { $_ } $hash{foo};   print join(' '\, keys %hash)\, "\n";

Why do you waste p5p's time by responding poorly to bug reports you do not understand?

-- c

[p5pRT]

From schmorp@schmorp.de

On Fri\, Aug 18\, 2006 at 03​:27​:56PM -0700\, Nicholas Clark via RT \perlbug\-followup@​perl\.org wrote​:

argument. This is achieved with lots of effort through PVLVs and magic​:

While I think that would be preferable for consistency reasons\, it also sounds a bit slow (and/or not that easily implementable).

Thats why _I_ would be happy if it were just documented (and just because it took me a while to find the priblem with "grep $_\, $hash{key}"\, which is better written as "$hash{key} || ()" anyways :).

However\, even if documented\, that behaviour feels highly unexpected for perl\, as it doesn't happen in other contexts\, as you pointed out.

$ perl -MDevel​::Peek -e 'sub foo {Dump $_[0]; } foo($hash{key});'

On the other hand\, if this is done in this case\, it would likely make sense to do it in map/grep/for context\, too. Especially if its only done when the key doesn't exist (which is exactly what happens in the above example when $hash{key} exists).

--   The choice of a   -----==- _GNU_   ----==-- _ generation Marc Lehmann   ---==---(_)__ __ ____ __ pcg@​goof.com   --==---/ / _ \/ // /\ \/ / http​://schmorp.de/   -=====/_/_//_/\_\,_/ /_/\_\ XX11-RIPE

[p5pRT]

From schmorp@schmorp.de

On Fri\, Aug 18\, 2006 at 05​:16​:57PM -0500\, David Nicol \davidnicol@​gmail\.com wrote​:

I can consider it\, but it has nothing to do whatsoever with the example\, nor with most other uses of grep\, nor with the problem at hand.

what's the problem at hand? The example didn't seem to do anything but demonstrate the documented behavior\,

*sigh* the point is that it isn't documented.

perceiving the documented behavior to be a problem\, perhaps because he did not know about exists().

As I wrote\, "exists" has nothing to do with it.

Again\, this is not very helpful. If you want to be helpful\, please either point out where this is documented and please point out the relevance of exists to the examples given.

Exists allows determinig if an element is in a hash or not\, so you can avoid autovivving it when you don't want to.

But Perl does not work that way​: accessing a hash element does not "autovivify" (== store undef in the element). I am specifically talking about map\, grep\, and for.

This has _nothing_ to do with autovivification in general.

The problem is unwanted autoviv in hash looking-ups\, yes?

Not at all\, there is no "autoviv" when looking up in hashes in perl\, so thats not the porblem.

The problem is undocumented grep/map/for side effects.

I am pretty sure I wrote this clearly enough\, so if you still don't understand what I wrote about\, could you pinpoint more exactly whats unclear\, I am ready to explain anything in more detail if I only knew what escapes you.

--   The choice of a   -----==- _GNU_   ----==-- _ generation Marc Lehmann   ---==---(_)__ __ ____ __ pcg@​goof.com   --==---/ / _ \/ // /\ \/ / http​://schmorp.de/   -=====/_/_//_/\_\,_/ /_/\_\ XX11-RIPE

[p5pRT]

From rick@bort.ca

On Sat\, Aug 19\, 2006 at 12​:42​:35AM +0200\, Marc Lehmann wrote​:

On Fri\, Aug 18\, 2006 at 03​:27​:56PM -0700\, Nicholas Clark via RT \perlbug\-followup@​perl\.org wrote​:

argument. This is achieved with lots of effort through PVLVs and magic​:

While I think that would be preferable for consistency reasons\, it also sounds a bit slow (and/or not that easily implementable).

I don't know if it's slow but it's not to hard to implement. See patch below.

-- Rick Delaney rick@​bort.ca

Inline Patch

```diff diff -pruN perl-current/op.c perl-current-dev/op.c --- perl-current/op.c 2006-08-09 17:54:18.000000000 -0400 +++ perl-current-dev/op.c 2006-08-18 20:20:55.000000000 -0400 @@ -1331,7 +1331,7 @@ Perl_mod(pTHX_ OP *o, I32 type) case OP_AELEM: case OP_HELEM: ref(cBINOPo->op_first, o->op_type); - if (type == OP_ENTERSUB && + if ((type == OP_ENTERSUB || type == OP_GREPSTART) && !(o->op_private & (OPpLVAL_INTRO | OPpDEREF))) o->op_private |= OPpLVAL_DEFER; if (type == OP_LEAVESUBLV) diff -pruN perl-current/t/op/hashassign.t perl-current-dev/t/op/hashassign.t --- perl-current/t/op/hashassign.t 2006-06-13 15:29:32.000000000 -0400 +++ perl-current-dev/t/op/hashassign.t 2006-08-18 21:43:35.000000000 -0400 @@ -8,7 +8,7 @@ BEGIN { # use strict; -plan tests => 215; +plan tests => 219; my @comma = ("key", "value"); @@ -306,3 +306,25 @@ foreach my $chr (60, 200, 600, 6000, 600 @expect{map "$_", @refs} = @types; ok (eq_hash(\%h, \%expect), 'blessed ref stringification'); } +# test hash entries do not spring into existence +{ + # perl #40194 + my %h; + + %h = (); + grep 1, $h{key}; + is(scalar(keys %h), 0, "grep does not create hash entries"); + + %h = (); + map 1, $h{key}; + is(scalar(keys %h), 0, "map does not create hash entries"); + + %h = (); + 1 for $h{key}; + is(scalar(keys %h), 0, "for does not create hash entries"); + + %h = (); + sub { defined $_[0] }->($h{key}); + is(scalar(keys %h), 0, "subroutine does not create hash entries"); +} + ```

[p5pRT]

From @gbarr

On Aug 18\, 2006\, at 5​:27 PM\, Nicholas Clark wrote​:

On Fri\, Aug 18\, 2006 at 03​:07​:08PM -0500\, David Nicol wrote​: 2 blank lines. The key is not autovivified if it's used as a
subroutine argument. This is achieved with lots of effort through PVLVs and
magic​:

Yes\, that was a bug^Wchange introduced last time someone stumbled on
the hash element in an lvalue context will auto-vivify

I assume that the same approach could be used by grep for not-yet- existing hash keys.

If this is changed I hope it is well documented that it is. I know I
left code at my previous company that used this feature\, and also
code that avoided it by doing grep { ... } reverse @​hash{@​foo}

Graham.

[p5pRT]

From mjtg@cam.ac.uk

Rick Delaney \rick@​bort\.ca wrote

I don't know if it's slow but it's not to hard to implement. See patch below.

For completeness (and checking backward compatibility)\, you should also have test cases for when grep/map/for *do* update their arguments and therefore autovivify.

Mike Guy

[p5pRT]

From @davidnicol

Again\, this is not very helpful. If you want to be helpful\, please either point out where this is documented and please point out the relevance of exists to the examples given.

I'm not the one calling a documented behavior a bug\, although I admit I was loose with "whenever you so much as think about it." That's clearly absurd and was meant to be taken as a joke while communicating that the exact rules under which the key creation occurs are subtle and complex.

My mistake was challenging a German to a contest of demonstrating lack of sense of humor. I should have known better\, as I have done that before and lost then too.

Before we descend into complete name-calling\, I shall quote documentation supporting the documented nature of hash key autovivication in grep/map/for\, even though I have better things to do and so do you.

from perldoc -f map​:

  Note that $_ is an alias to the list value\, so it can be used   to modify the elements of the LIST. While this is useful and   supported\, it can cause bizarre results

from perldoc -f grep​:

  Note that $_ is an alias to the list value\, so it can be used   to modify the elements of the LIST. While this is useful and   supported\, it can cause bizarre results if the elements of LIST   are not variables. Similarly\, grep returns aliases into the   original list\, much as a for loop's index variable aliases the   list elements. That is\, modifying an element of a list   returned by grep (for example\, in a "foreach"\, "map" or another   "grep") actually modifies the element in the original list.   This is usually something to be avoided when writing clear   code.

from perldoc perlsyn​:   the "foreach" loop index variable is an implicit alias for each item in the   list that you're looping over.

from perldoc perlref​:   6. References of the appropriate type can spring into existence if you   dereference them in a context that assumes they exist.

[p5pRT]

From chromatic@wgz.org

On Monday 21 August 2006 12​:05\, David Nicol wrote​:

from perldoc perlref​:           6.  References of the appropriate type can spring into existence           if you dereference them in a context that assumes they exist.

Q​: What's the difference between a reference and undef? A​: One is a reference. The other is undef. That's why they have different names.

I'm sorry it's not a better punchline\, but your interpretation of an otherwise clear part of the documentation so shocked me with its complete nonsensical wrongness that I'm just not feeling that witty.

-- c

[p5pRT]

From @davidnicol

On 8/19/06\, Graham Barr \gbarr@​pobox\.com wrote​:

If this is changed I hope it is well documented that it is. I know I left code at my previous company that used this feature\, and also code that avoided it by doing grep { ... } reverse @​hash{@​foo}

Graham.

The relevance of exists() is that the clearest way to avoid the autoviv\, to me\, is to use exists() to test for the existence of the hash elements before iterating over them.

Why would anybody want to iterate over nonexistent hash entries\, anyway? And what would make that person think that the hash entries would stay nonexistent after they had been iterated over?

Since Graham Barr\, and presumably others\, have used the autoviv in lval context as a feature\, backwards compat. demands that hash key autoviv in lval should be left in. Or at least there should be a deprecation cycle with a warning suggesting an alternate construction.

When would the warning get triggered?   whenever a hash key that has been autovivified due   to use in one of the deprecated contexts gets tested or iterated over?   (nonworkable\, would require autovivving the reference to a magical   placeholder value that exists only to give the warning but is otherwise   undefined)

  whenever a nonexistent hash key is iterated over or included as a   subroutine parameter?   ( clearly identifiable situation that is already special-cased in   Rick Delaney's proposed patch )

The ability to alias to a nonexistent hash key without creating it is definitely a Good Thing\, but it will break existing code in mysterious ways.

[p5pRT]

From @davidnicol

Where is it documented that iterating over nonexistent hash elements does not bring them into existence? Marc Lehmann and chromatic apparently believe that it that is documented somewhere. Where?

[p5pRT]

From q.eibcartereio.=~m-b.{6}-cgimosx@gumdrop.flyinganvil.org

On Mon\, Aug 21\, 2006 at 02​:05​:57PM -0500\, David Nicol wrote​:

Again\, this is not very helpful. If you want to be helpful\, please either point out where this is documented and please point out the relevance of exists to the examples given.

I'm not the one calling a documented behavior a bug\, although I admit I was loose with "whenever you so much as think about it." That's clearly absurd and was meant to be taken as a joke while communicating that the exact rules under which the key creation occurs are subtle and complex.

As a user\, the key creation rules are expected to be simple; I expect $hash{foo} to autovivify if I use $hash{foo} in a lvalue context\, or if I take a reference to $hash{foo} (typically by doing $hash{foo}{bar} but it can occur in other ways.)

However\, foreach is not where I would expect to find that happening\, it cannot be using the list in lvalue context as it can work on constants​:

  foreach ('foo'\, 'bar'\, 'baz') {   $_ = 'foo';   }

outputs Modification of a read-only value attempted at - line 2.

So the aliasing of foreach is not where I expect to have problems. There still is the possibility of reference-caused autovivification​:

  foreach ('foo'\, 'bar'\, 'baz') {   $hash{$_}; # example 1   }

does not create any entries in %hash\, yet

  foreach ('foo'\, 'bar'\, 'baz') {   \$hash{$_}; # example 2   }

will create entries for those keys. The key here is that the aliasing caused by

  foreach ($hash{foo}\, $hash{bar}\, $hash{baz}) {   # do nothing   }

turns this into example 2 instead of example 1\, resulting in %hash containing ( 'foo' => undef\, 'bar' => undef\, 'baz' => undef ). This I think is not intuitive even to those familiar with perl. And despite your claims I do not believe that the documentation makes this point clear; the documentation only talks about side effects caused by modification\, not implicitly by the aliasing itself.

So\, after the proposed change (I didn't see if it made it to blead yet)

  foreach ($hash{foo}\, $hash{bar}\, $hash{baz}) {   \$_;   }

should still cause these hash keys to autovivify.

-- -Ben Carter Human beings\, who are almost unique in having the ability to learn from the experience of others\, are also remarkable for their apparent disinclination to do so. - Douglas Adams\, "Last Chance to See"

[p5pRT]

From chromatic@wgz.org

On Monday 21 August 2006 12​:55\, David Nicol wrote​:

Where is it documented that iterating over nonexistent hash elements does not bring them into existence? Marc Lehmann and chromatic apparently believe that it that is documented somewhere. Where?

perlfaq4\, "Why does passing a subroutine an undefined element in a hash create it?"​:

  Normally\, merely accessing a key's value for a nonexistent key does not   cause that key to be forever there. This is different than awk's   behavior.

You could also\, *run* the code either Marc or I posted several messages ago.

-- c

[p5pRT]

From rick@bort.ca

On Mon\, Aug 21\, 2006 at 05​:32​:03PM +0100\, Mike Guy wrote​:

Rick Delaney \rick@​bort\.ca wrote

I don't know if it's slow but it's not to hard to implement. See patch below.

For completeness (and checking backward compatibility)\, you should also have test cases for when grep/map/for *do* update their arguments and therefore autovivify.

True. The patch is incomplete anyway\, because I forgot the hash slice case.

-- Rick Delaney rick@​bort.ca

[p5pRT]

From @davidnicol

The glossary entries for "lvalue" and "autovivification" in the 3rd ed. blue camel\, taken together\, clearly imply that the hash elements should appear.

Aliasing implies taking a reference.

chromatic is a punk.

[p5pRT]

From chromatic@wgz.org

On Monday 21 August 2006 14​:17\, David Nicol wrote​:

The glossary entries for "lvalue" and "autovivification" in the 3rd ed. blue camel\, taken together\, clearly imply that the hash elements should appear.

In the face of 1) working\, runnable code demonstrating that your ideas are wrong and 2) documentation showing your ideas to be wrong\, are you now relying on 3) repeated proof by assertion and misinterpretation of well-understood documentation (not to mention leaving out the "hard-" in "hard-reference")?

I do not think that is a good way to build a reliable\, documentable\, and explainable computer system.

Aliasing implies taking a reference.

This argument is\, like your previous ones\, also completely full of beans and at an angle severely bent from reality. (Try storing an alias in a scalar\, for example\, without creating a reference.)

Regardless\, I've finished with this discussion\, because it's clearly now just noise between people who know Perl and\, well\, you.

-- c

[p5pRT]

From @nwc10

On Mon\, Aug 21\, 2006 at 04​:55​:21PM -0400\, Rick Delaney wrote​:

On Mon\, Aug 21\, 2006 at 05​:32​:03PM +0100\, Mike Guy wrote​:

Rick Delaney \rick@​bort\.ca wrote

I don't know if it's slow but it's not to hard to implement. See patch below.

For completeness (and checking backward compatibility)\, you should also have test cases for when grep/map/for *do* update their arguments and therefore autovivify.

True. The patch is incomplete anyway\, because I forgot the hash slice case.

So did whomsoever fixed subroutine arguments back in perl5.004​:

$ ./perl -Ilib -MDevel​::Peek -lwe 'sub f {} f(@​a{1\,2}); print scalar keys %a' 2 $ ./perl -Ilib -MDevel​::Peek -lwe 'sub f {} f($a{1}\,$a{2}); print scalar keys %a' 0

I can find the documentation patch (from Mike Guy) but not the code patch.

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/1997-03/msg01956.html

Time to report the hash slice as a bug?

Nicholas Clark

[p5pRT]

From @davidnicol

On 8/21/06\, chromatic \chromatic@​wgz\.org wrote​:

Aliasing implies taking a reference.

This argument is\, like your previous ones\, also completely full of beans and at an angle severely bent from reality. (Try storing an alias in a scalar\, for example\, without creating a reference.)

chromatic\, I think you just agreed with me on that point\, after your acceptably brief and silly stream of abuse. Unless I misunderstand you. By "storing an alias in a scalar" I take you to mean "aliasing one scalar to another."

# perl -le '*foo = $bar = 3; print $foo; $bar = 2; print $foo'

# perl -le '*foo = \($bar = 3); print $foo; $bar = 2; print $foo' 3 2

Here's a serious question\, on-topic for the question of what are the exact semantics of non-autovivifying aliases to hash elements​: How far away from the hash are the PVLVs able to get before they create their elements?

Given the new-style non-autovivifying loop aliases\, when does the key "spring into existence" in the following?

%hash = ( ); foreach ($hash{key}) { # used to autoviv here but won't in the future   defined $_ and print "not defined\, at all."; # does not exist   exists($hash{key}) and print "exists though.\n"; # does in 5.8.7   $reference = \$_; # does it exist yet? }; # exists($hash{key}) should be false here\, by new style $$reference = 1; # must autovivify to accept the assignment # (unless $reference is a reference to $_ rather than to thing $_ was # aliased to​: in 5.8.7\, assigning to $$reference sets $hash{key}.) # exists($hash{key}) but true here

This is an edge case that should be defined and tested.

Furthermore\, if something else sets $hash{key}\, does that invalidate the reference\, or does it still refer to $hash{key}?

Current (5.8.7) behavior is that assigning to $$reference will clobber an assignment to $hash{key} -- that should be preserved.

[p5pRT]

From @davidnicol

So did whomsoever fixed subroutine arguments back in perl5.004​:

$ ./perl -Ilib -MDevel​::Peek -lwe 'sub f {} f(@​a{1\,2}); print scalar keys %a' 2 $ ./perl -Ilib -MDevel​::Peek -lwe 'sub f {} f($a{1}\,$a{2}); print scalar keys %a' 0

For what its worth\, currently in the post-5.004 non-autovivving situation the key starts existing when a reference is taken to it\, rather than later. Should this be construed as a bug too?

%hash = ( ); sub f{   print "keys​: "\, keys(%hash)\, "\n"; # empty here   $reference = \$_[0] }; f($hash{key}); print "keys​: "\, keys(%hash)\, "\n"; # has the key $hash{key} = 2; $$reference = 1; print "key​: $hash{key}\n"; # prints 1

[p5pRT]

From @davidnicol

---------- Forwarded message ---------- From​: Damian Conway \damian@​conway\.org Date​: Aug 22\, 2006 8​:44 PM Subject​: Re​: appeal to authority -- help me\, doctor conway\, you're my only hope ( I exaggerate ) To​: David Nicol \davidnicol@​gmail\.com

Hi David\,

The very last section of Synopsis 9 covers this issue. In Perl 6​:

  Autovivification will only happen if the vivifiable path is used as a   container\, by binding\, assigning\, or capturing into an argument list.   On the other hand\, value extraction does not autovivify.

So containers bound to subroutine parameters *are* autovivified\, and so are any aliases that are explicitly and implicitly bound within a scope (including aliases bound to $_).

On the other hand\, tests like​:

  if ($hash{key1}{key2}{key3}) {...}

or​:

  if (exists $hash{key1}{key2}{key3}) {...}

won't autovivify nested hashes if a higher-level entry doesn't exist. That *is* a Perl 5 behaviour that ought to be fixed IMHO.

Hope this clarifies things.

Damian

PS​: Feel free to share this message with whoever you wish.

-- David L Nicol Dickenson on the flag http​://cronos.advenge.com/pc/EmilyDickenson/SecondBook/p39.html

[p5pRT]

From mjtg@cam.ac.uk

Nicholas Clark \nick@​ccl4\.org wrote

I can find the documentation patch (from Mike Guy) but not the code patch.

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/1997-03/msg01956.html

Time to report the hash slice as a bug?

Rummaging in old files\, I find this from Feb 2000​:

Just to get this into the bug database\, rather than in any expectation of action​:

       foreach \(@​h\{a\, b\}\) \{ \}

autovivifies $h{a} and $h{b}.

Yes\, I understand why it happens.

So I guess (a relative of) this is already in the bug database.

Mike Guy

[p5pRT]

From @smpeters

On Wed\, Aug 23\, 2006 at 07​:25​:28PM +0100\, Mike Guy wrote​:

Nicholas Clark \nick@​ccl4\.org wrote

I can find the documentation patch (from Mike Guy) but not the code patch.

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/1997-03/msg01956.html

Time to report the hash slice as a bug?

Rummaging in old files\, I find this from Feb 2000​:

Just to get this into the bug database\, rather than in any expectation of action​:

       foreach \(@​h\{a\, b\}\) \{ \}

autovivifies $h{a} and $h{b}.

Yes\, I understand why it happens.

So I guess (a relative of) this is already in the bug database.

...and I believe I wrote a few test cases for this behavior as well.

Steve Peters steve@​fisharerojo.org

[p5pRT]

From mjtg@cam.ac.uk

"David Nicol" \davidnicol@​gmail\.com wrote

For what its worth\, currently in the post-5.004 non-autovivving situation the key starts existing when a reference is taken to it\, rather than later. Should this be construed as a bug too?

Yes and no.

The rule should be that a variable is autovivified when it is needed\, and no sooner (rather than some legalistic reading of the incomplete / imprecise documentation). For assignment\, the moment of need is obvious.

For creating a reference\, the need is more obscure. It's because a reference is an "identity" for the variable\, so that something like

  \($h{nonexistent}) == \($h{nonexistent})

is always true. Autovivication provides this identity; conceivably it could also be achieved in some other way.

Mike Guy

[p5pRT]

From @davidnicol

On 8/23/06\, Mike Guy \mjtg@​cam\.ac\.uk wrote​:

Autovivication provides this identity; conceivably it could also be achieved in some other way.

if we add delayed autoviv of hash elements to more places than the one special case we currently have\, consistency would seem to demand that the reference situation would not autoviv\, also that the exists function would take a scalar argument which would generally return true but which would return false when given such a placeholder​:

  perl -le ' sub f{ print(exists(\$_[0])?"yes"​:"no"} f($hash{key})'

would print no.

Anyway\, given Damian Conway's answer regarding how perl 6 will play this shot\, I wonder what we\, the perl 5 maintenance community\, are trying to achieve by introducing (extending?) this additional layer of subtly non-backwards-compatible complexity\, since it isn't going to be forwards-compatible either.

Containers and locations are vivified as soon as they are bound\, to anything\, including the binding required to alias them to $_ within an aliasing loop\, and IMO 40194 and friends can be closed as not bugs\, at least not bugs in perl.

Attached is a patch to perlfaq4.pod.

-- David L Nicol Dickenson on the flag http​://cronos.advenge.com/pc/EmilyDickenson/SecondBook/p39.html

[p5pRT]

From @davidnicol

faqpatch.diff

```diff --- perl-5.9.4/pod/perlfaq4.pod 2006-08-15 07:37:41.000000000 -0500 +++ perl-5.9.4_dln/pod/perlfaq4.pod 2006-08-23 17:09:02.000000000 -0500 @@ -1988,22 +1988,37 @@ my @keys = keys %myhash; # @keys = (0,1,2,3,...) -=head2 Why does passing a subroutine an undefined element in a hash create it? +=head2 Why does using an undefined hash element in a for/map/grep loop create it? -If you say something like: +That's because functions get scalars passed in by reference. If somefunc() modifies C<$_[0]>, +it has to be ready to write it back into the caller's version. The aliasing of $_ +to the value under consideration works similarly: in order to bind $_ to the +hash element, the hash element must be brought into existence. + +=head2 Why does passing a subroutine an undefined element in a hash not create it? + +A change was introduced, by popular demand from beginning Perl programmers +who perceived the situation as a bug, in Per5.004, such that if you write +something like: somefunc($hash{"nonesuch key here"}); -Then that element "autovivifies"; that is, it springs into existence -whether you store something there or not. That's because functions -get scalars passed in by reference. If somefunc() modifies C<$_[0]>, -it has to be ready to write it back into the caller's version. - -This has been fixed as of Perl5.004. +Then that element does not autovivify, that is, spring into existence, +until you store something there or otherwise do something with the +parameter that would cause the undefined element to appear. Normally, merely accessing a key's value for a nonexistent key does I cause that key to be forever there. This is different than -awk's behavior. +awk's behavior. + +If you really need a hash element to autovivify on getting passed to +a function, you may use a hash slice: C. The +hash slicing case was not affected by the 5.004 change. + +The point addressed in this and the previous question are subjects of heated +debate: relying on a particular autovivification behavior when passing hash +entries to subroutines or using them in lists passed to aliasing loop constructs +is currently imprudent. =head2 How can I make the Perl equivalent of a C structure/C++ class/hash or array of hashes or arrays? ```

[p5pRT]

From @rgs

I think it's not appropriate to change this behaviour.

[p5pRT]

@rgs - Status changed from 'open' to 'rejected'