Does ‘my sub f; eval 'sub f{}'’ make sense?

[p5pRT]

Migrated from rt.perl.org#123367 (status was 'resolved')

Searchable as RT123367$

[p5pRT]

From @cpansprout

I thought I had submitted this already\, but I couldn’t find it in RT​:

I know I was the one that implemented it and made eval 'sub f{}' work with ‘my sub f’ in scope\, because it seemed that one might expect it to work. But on second thought I’m not sure it does make sense.

‘sub f {}’ is not actually evaluated at run time. It is a compile-time declaration. For a lexical ‘my’ symbol\, it creates\, at compile time\, a protosub that later gets cloned on scope entry.

In ‘my sub f; eval 'sub f{}'’\, the sub declaration should create a protosub that gets cloned the *next* time the enclosing scope is entered. Right? It least that’s how it should work if we want it consistent. But this is what currently happens​:

$ ./perl -XIlib -e 'use feature "​:all"; for(1\,0) { my sub f; eval "sub f{warn 42}" if $_; f }' 42 at (eval 1) line 1. Undefined subroutine &f called at -e line 1.

Now\, I’m not sure that this eval-after-my even makes that much sense\, and lexical subs are experimental\, so that is an argument for changing it.

I *do* think this code makes sense\, but it doesn’t work​:

$ ./perl -Ilib -e 'use feature "​:all"; my sub f; BEGIN { eval "sub f{warn 42}"} f' The lexical_subs feature is experimental at -e line 1. Undefined subroutine &f called at -e line 1.

BEGIN time (when subs are being defined) seems a logical place to decide which sub to install.

--

Father Chrysostomos

[p5pRT]

From @ap

* Father Chrysostomos \perlbug\-followup@​perl\.org [2014-12-04 19​:45]​:

On second thought I’m not sure it does make sense.

You asked me to look at this a while ago. After repeat reading\, with some lengthy pauses included\, I still remain half-confused about what the problem is or what you think is the way it ought to work.

All I can say is that to my (possibly naïve) mind\, `my sub f {...}` ought to work just as if it said `my $f = sub {...};`\, with subsequent invocations of `f(...)` being exactly equivalent to `$f->(...)`. And of course a `sub f {...}` within scope of `my sub f` would likewise amount to `$f = sub {...};` – maybe even down to the lack of redefine warning.

This is the only way of defining this feature that I can come up with to produce behaviour with a coherence and self-consistence that I will find it graspable and predictable even in strange corner cases.

By that line of thinking\, then\,

$ ./perl -XIlib -e 'use feature "​:all"; for(1\,0) { my sub f; eval "sub f{warn 42}" if $_; f }' 42 at (eval 1) line 1. Undefined subroutine &f called at -e line 1.

… this is exactly what one would expect. And by the same token\,

$ ./perl -Ilib -e 'use feature "​:all"; my sub f; BEGIN { eval "sub f{warn 42}"} f' The lexical_subs feature is experimental at -e line 1. Undefined subroutine &f called at -e line 1.

… this *ought* to work.

Is the strict equivalence of these forms in my mind the reason I might be thinking of this differently than you do? The following appears to suggest it​:

‘sub f {}’ is not actually evaluated at run time. It is a compile-time declaration. For a lexical ‘my’ symbol\, it creates\, at compile time\, a protosub that later gets cloned on scope entry.

I’m not steeped enough in the implementation to talk about it in terms of protosub etc\, but again\, based on the strict equivalence\, `my sub f` basically ought to behave essentially like `my $f` does WRT the compile time/runtime effects split.

Is this opinion useful to you?

Regards\, -- Aristotle Pagaltzis // \<http​://plasmasturm.org/>

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @cpansprout

On Wed Jan 07 22​:08​:03 2015\, aristotle wrote​:

* Father Chrysostomos \perlbug\-followup@&#8203;perl\.org [2014-12-04 19​:45]​:

On second thought I’m not sure it does make sense.

You asked me to look at this a while ago. After repeat reading\, with some lengthy pauses included\, I still remain half-confused about what the problem is or what you think is the way it ought to work.

All I can say is that to my (possibly naïve) mind\, `my sub f {...}` ought to work just as if it said `my $f = sub {...};`\, with subsequent invocations of `f(...)` being exactly equivalent to `$f->(...)`. And of course a `sub f {...}` within scope of `my sub f` would likewise amount to `$f = sub {...};` – maybe even down to the lack of redefine warning.

This is the only way of defining this feature that I can come up with to produce behaviour with a coherence and self-consistence that I will find it graspable and predictable even in strange corner cases.

With regular subs\, sub foo{} is mostly equivalent to​:

  BEGIN { *foo = sub{} }

Similarly\, state sub foo{} is like​:

  state sub foo;   BEGIN { \&foo = sub{} }

My-subs also have a compile-time action\, in that the declaration creates a protosub that will be cloned at run time\, just as sub{...} does.

By that line of thinking\, then\,

$ ./perl -XIlib -e 'use feature "​:all"; for(1\,0) { my sub f; eval "sub f{warn 42}" if $_; f }' 42 at (eval 1) line 1. Undefined subroutine &f called at -e line 1.

… this is exactly what one would expect. And by the same token\,

$ ./perl -Ilib -e 'use feature "​:all"; my sub f; BEGIN { eval "sub f{warn 42}"} f' The lexical_subs feature is experimental at -e line 1. Undefined subroutine &f called at -e line 1.

… this *ought* to work.

Is the strict equivalence of these forms in my mind the reason I might be thinking of this differently than you do? The following appears to suggest it​:

‘sub f {}’ is not actually evaluated at run time. It is a compile- time declaration. For a lexical ‘my’ symbol\, it creates\, at compile time\, a protosub that later gets cloned on scope entry.

I’m not steeped enough in the implementation

Neither am I. I recently had a look at the code in newMYSUB\, and I don’t understand it. This is coming from the person who *wrote* it. So that suggests that the number of people who understand that code is zero\, which is a scary thought.

to talk about it in terms of protosub etc\, but again\, based on the strict equivalence\, `my sub f` basically ought to behave essentially like `my $f` does WRT the compile time/runtime effects split.

Is this opinion useful to you?

I don’t really know the answer to that. :-)

Taking this example that you say ought to work​:

$ ./perl -Ilib -e 'use feature "​:all"; my sub f; BEGIN { eval "sub f{warn 42}"} f' The lexical_subs feature is experimental at -e line 1. Undefined subroutine &f called at -e line 1.

Is that equivalent to

  my sub f;   BEGIN { eval '\&f = sub { warn 42 }' }

?

If so\, then we will have aliased the sub at compile time before the lexical scope is entered. Since all lexical variables are implicitly reset at scope entry (or exit\, depending on how you look at it)\, at some point sub f has to revert back to being undefined.

Or\, if

  my sub f { warn 41 }   BEGIN { sub { warn 42 } }

is equivalent to

  my sub f;   \&f = sub { warn 41 };   BEGIN { \&f = sub { warn 42 } }

then the BEGIN block is a no-op\, since the run-time 41 assignment would take precedence.

Similarly\, the preceding example would be equivalent to having \&f = \ outside the BEGIN block.

So the assignment model doesn’t work here.

Brad Gilbert wrote​:

FWIW\, in Perl6 it also works​:

$ perl6 \-e ' my &f //= sub \(\)\{???\}; f; EVAL q\[&f = sub \(\)\{say "eval"\}\]; f'
Stub code executed  in sub  at \-e​:1

eval
$ perl6 \-e ' my &f //= sub \(\)\{???\}; f; BEGIN EVAL q\[&f = sub \(\)\{say "eval"\}\]; f '
eval
eval

I notice you are using //=\, which avoids all the problems we are dealing with here.

What if you replace that with ‘my sub f {???}’?

Using \&f=... assignment in Perl 5 I can make it work very similarly to your Perl 6 example. It is the declarative syntax I am asking about.

--

Father Chrysostomos

[p5pRT]

From @cpansprout

On Wed Jan 07 22​:08​:03 2015\, aristotle wrote​:

* Father Chrysostomos \perlbug\-followup@&#8203;perl\.org [2014-12-04 19​:45]​:

On second thought I’m not sure it does make sense.

You asked me to look at this a while ago. After repeat reading\, with some lengthy pauses included\, I still remain half-confused about what the problem is or what you think is the way it ought to work.

All I can say is that to my (possibly naïve) mind\, `my sub f {...}` ought to work just as if it said `my $f = sub {...};`\, with subsequent invocations of `f(...)` being exactly equivalent to `$f->(...)`. And of course a `sub f {...}` within scope of `my sub f` would likewise amount to `$f = sub {...};` – maybe even down to the lack of redefine warning.

This is the only way of defining this feature that I can come up with to produce behaviour with a coherence and self-consistence that I will find it graspable and predictable even in strange corner cases.

Here is another way of looking at it. This code​:

my $f = sub {...}

involves three distinct steps.

1. A protosub is created at compile time (you can observe this with   attributes). 2. The sub is cloned. 3. The clone is assigned to $f.

With 'my sub f {...}' it is clear that the scoping is identical to 'my $f = sub {...}'.

It is when the declaration precedes the definition that we can no longer us the 'my $f =' model to explain it.

my sub f; f(); sub f {...};

In this case step 1 happens at compile time\, as before\, but on line 3. Steps 2 and three happen on line 1\, allowing the f() call on line 2 to work.

This later definition of the sub works even if it is inside an inner sub​:

my sub f; f(); sub { sub f {...} }

my sub f; f(); sub BEGIN { sub f {...} }

It is when we have string eval that things get confusing.

my sub f; eval 'sub f {...}';

In that code\, steps 2 and 3 happen first\, and then all three steps happen at the same time inside the eval. (That the internal code that handles sub definitions should have to handle cloning as well really complicates things.) This is an exception to the general rule\, which is why I question whether it was a good idea. Usually steps 2 and 3 happen only when the enclosing block is entered.

When you have a BEGIN block with an eval inside it\, then things get a little screwy.

my sub f; BEGIN { eval 'sub f {...}'; }

Because of the way lexical-sub-definition-in-eval has been special-cased to work\, you get all three steps happening at once at compile time\, and then steps 2 and 3 repeat at run time.

Now this may all sound like internal jargon\, but it is all observable behaviour. Steps 2 and 3 always happen together with my-subs\, but step 1 happens separately. One can observe step 1 through attributes (i.e.\, based on when MODIFY_CODE_ATTRIBUTES is called.) One can observe steps 2 and 3 (actually a single step) by the effects of calling the sub.

--

Father Chrysostomos

[p5pRT]

From @cpansprout

On Fri Jan 09 18​:14​:33 2015\, sprout wrote​:

On Wed Jan 07 22​:08​:03 2015\, aristotle wrote​:

* Father Chrysostomos \perlbug\-followup@&#8203;perl\.org [2014-12-04 19​:45]​:

On second thought I’m not sure it does make sense.

You asked me to look at this a while ago. After repeat reading\, with some lengthy pauses included\, I still remain half-confused about what the problem is or what you think is the way it ought to work.

All I can say is that to my (possibly naïve) mind\, `my sub f {...}` ought to work just as if it said `my $f = sub {...};`\, with subsequent invocations of `f(...)` being exactly equivalent to `$f->(...)`. And of course a `sub f {...}` within scope of `my sub f` would likewise amount to `$f = sub {...};` – maybe even down to the lack of redefine warning.

This is the only way of defining this feature that I can come up with to produce behaviour with a coherence and self-consistence that I will find it graspable and predictable even in strange corner cases.

Here is another way of looking at it. This code​:

my $f = sub {...}

involves three distinct steps.

1. A protosub is created at compile time (you can observe this with attributes). 2. The sub is cloned. 3. The clone is assigned to $f.

With 'my sub f {...}' it is clear that the scoping is identical to 'my $f = sub {...}'.

It is when the declaration precedes the definition that we can no longer us the 'my $f =' model to explain it.

my sub f; f(); sub f {...};

In this case step 1 happens at compile time\, as before\, but on line 3. Steps 2 and three happen on line 1\, allowing the f() call on line 2 to work.

This later definition of the sub works even if it is inside an inner sub​:

my sub f; f(); sub { sub f {...} }

my sub f; f(); sub BEGIN { sub f {...} }

It is when we have string eval that things get confusing.

my sub f; eval 'sub f {...}';

In that code\, steps 2 and 3 happen first\, and then all three steps happen at the same time inside the eval. (That the internal code that handles sub definitions should have to handle cloning as well really complicates things.) This is an exception to the general rule\, which is why I question whether it was a good idea. Usually steps 2 and 3 happen only when the enclosing block is entered.

When you have a BEGIN block with an eval inside it\, then things get a little screwy.

my sub f; BEGIN { eval 'sub f {...}'; }

Because of the way lexical-sub-definition-in-eval has been special- cased to work\, you get all three steps happening at once at compile time\, and then steps 2 and 3 repeat at run time.

Now this may all sound like internal jargon\, but it is all observable behaviour. Steps 2 and 3 always happen together with my-subs\, but step 1 happens separately. One can observe step 1 through attributes (i.e.\, based on when MODIFY_CODE_ATTRIBUTES is called.) One can observe steps 2 and 3 (actually a single step) by the effects of calling the sub.

Oh\, I think I know what I was missing. When the sub is defined in a string eval and declared outside the eval (this is the special case referred to above)\, the sub is immediately cloned and the clone is installed in the pad. Otherwise\, the protosub is set aside for the sake of future cloning. Maybe we need *both* to happen in string eval\, and everybody’s expectations will be met. Or maybe not everybody’s.

  for (1\,2) {   my sub f { print "42\n" };   f();   eval 'sub f { print "43\n" }';   f();   }

That would print​:

42 43 43 43

But it would allow BEGIN-time string eval to set things up properly.

--

Father Chrysostomos

[p5pRT]

From @cpansprout

On Wed Jan 07 22​:08​:03 2015\, aristotle wrote​:

* Father Chrysostomos \perlbug\-followup@&#8203;perl\.org [2014-12-04 19​:45]​:

$ ./perl -Ilib -e 'use feature "​:all"; my sub f; BEGIN { eval "sub f{warn 42}"} f' The lexical_subs feature is experimental at -e line 1. Undefined subroutine &f called at -e line 1.

… this *ought* to work.

And it started working with commit v5.21.6-197-g0f94cb1\, but I do not know why. (That commit was not supposed to change behaviour at all.) It needs a test.

--

Father Chrysostomos

[p5pRT]

From @cpansprout

On Wed May 18 22​:22​:33 2016\, sprout wrote​:

On Wed Jan 07 22​:08​:03 2015\, aristotle wrote​:

* Father Chrysostomos \perlbug\-followup@&#8203;perl\.org [2014-12-04 19​:45]​:

$ ./perl -Ilib -e 'use feature "​:all"; my sub f; BEGIN { eval "sub f{warn 42}"} f' The lexical_subs feature is experimental at -e line 1. Undefined subroutine &f called at -e line 1.

… this *ought* to work.

And it started working with commit v5.21.6-197-g0f94cb1\, but I do not know why. (That commit was not supposed to change behaviour at all.) It needs a test.

Much of what I said in this ticket was based on a misunderstanding of how newMYSUB determines how to install the sub.¹ Whether a newly-defined lexical sub is set aside for cloning later or cloned and installed in the lexical scope immediately is determined by whether the scope in which it is declared² is currently active (the sub at that scope level is running).

So\, that

  my sub f;   BEGIN { eval "sub f {...}" }

did not work was not due to the model followed\, but simply due to a bug in the code. I do not think it is worth poring through the code to find out exactly why it used to fail. But it probably had something to do with the more complex structure of SV + magic not being handled correctly somewhere. v5.21.6-197-g0f94cb1 simplified the data structure and apparently fixed the mishandling thereof at the same time.

All this is to say that this bug can be closed and that the current model\, which I suggested changing\, is fine. I added a test in 38fe8a04ecd.

¹I know\, I even *wrote* the code. But by the time this ticket came up\, I had forgotten it. Also\, just *look* at the code in newMYSUB. That’s enough to hurt anyone’s head. Maybe I should plead insanity\, but not my own in sanity; rather\, insanity in Perl’s run-time-vs-compile-time scoping model.

²Note the distinction between declaration and definition. The former refers to ‘my sub’\, which establishes the lexical scope. The latter refers to the body\, which defines what the sub does.

--

Father Chrysostomos

[p5pRT]

@cpansprout - Status changed from 'open' to 'resolved'