SyntaxError when free variable name is also an exception target

[amauryfa]

BPO	4617
Nosy	@gvanrossum, @warsaw, @rhettinger, @terryjreedy, @pjeby, @amauryfa, @benjaminp, @ezio-melotti, @florentx
Files	delete_deref.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/amauryfa' closed_at = created_at = labels = ['interpreter-core', 'type-feature'] title = 'SyntaxError when free variable name is also an exception target' updated_at = user = 'https://github.com/amauryfa' ``` bugs.python.org fields: ```python activity = actor = 'barry' assignee = 'amaury.forgeotdarc' closed = True closed_date = closer = 'amaury.forgeotdarc' components = ['Interpreter Core'] creation = creator = 'amaury.forgeotdarc' dependencies = [] files = ['12318'] hgrepos = [] issue_num = 4617 keywords = ['patch'] message_count = 20.0 messages = ['77536', '77691', '77693', '77696', '77703', '77704', '78434', '79228', '99247', '99855', '99866', '99880', '99911', '99915', '99918', '99950', '113312', '113335', '113340', '116048'] nosy_count = 11.0 nosy_names = ['gvanrossum', 'jhylton', 'barry', 'rhettinger', 'terry.reedy', 'pje', 'amaury.forgeotdarc', 'benjamin.peterson', 'ezio.melotti', 'cmcqueen1975', 'flox'] pr_nums = [] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue4617' versions = ['Python 3.2'] ```

[amauryfa]

This issue comes from bpo-4613. The following code raises a SyntaxError("can not delete variable 'e' referenced in nested scope"):

def f():
    e = None
    def g():
        e

try:
    pass
except Exception as e:
    pass # SyntaxError here???

The reason is because of http://www.python.org/dev/peps/pep-3110/#semantic-changes, a "del e" statement is inserted.

The above code is correct, and should work. I suggest that the limitation: "can not delete variable referenced in nested scope" could be removed.

After all, the "variable referenced" has no value before it is set, accessing it raises either NameError("free variable referenced before assignment in enclosing scope") or UnboundLocalError("local variable referenced before assignment")

The Attached patch adds a DELETE_DEREF opcode, that removes the value of a cell variable, and put it in a "before assignment" state.

Some compiler experts should review it. Few regressions are possible, since the new opcode is emitted where a SyntaxError was previously raised. The patch could also be applied to 2.7, even if it is less critical there.

Tests are to come, but I'd like other's suggestions.

[terryjreedy]

-1 as I understand the proposal. Your code is bugged and should fail as soon as possible.

If I understand correctly, you agree that the SyntaxError is correct as the language is currently defined, but you want the definition changed. It is not clear if you only want implicit deletes at the end of except clauses to work or if you only want explicit deletes to work.

If the latter, you want

def f():
  e = 1
  del e
  def g(): print(e)
  return g

to compile. I would not. Your reason "After all, the "variable referenced" has no value before it is set," (duh, right) makes no sense to me in this context. g must have a valid value of e to run. So you seem to be suggesting that detection of buggy code should be delayed.

[rhettinger]

Not sure the "del e" idea was a good solution to the garbage collection problem. Amaury's code looks correct to me. Maybe the existing e variable should be overwritten and the left intact (as it used to be) or perhaps it should be made both temporary and invisible like the induction variable in a list comprehension.

Phillip, any thoughts?

[amauryfa]

Terry, my motivation is that the sample code above runs correctly with python 2.6, but python 3.0 cannot even compile it. The sample looks valid python code, and should run. Yes, the same 'e' is used both as a nested variable and as an exception target, but this should not matter with our dynamic language.

First I thought to turn the implicit "del e" into something else (and change PEP-3110), but then I saw that the error "can not delete variable referenced in nested scope" is actually a limitation of the interpreter that is easy to remove.

[afcdb1a8-bc5d-4852-a2a5-71b4ae6361ae]

I could argue either way on this one; it's true that deleting a nested-scope variable is sometimes desirable, but it also seems to me like closing over an except: variable is a Potentially Bad Idea.

In neither case, however, do I think it's appropriate to drop the temporary nature of the variable. I could perhaps get behind resetting the variable to None instead of deleting it, but of course the PEP would need changing. There's also a question of whether we should do the same thing with "with ... as" variables.

(Btw, I'm not sure why this one's assigned to me; ISTM I might have proposed the current except/as GC semantics, but I'm not familiar with the actual implementation in 2.6 or 3.0)

[rhettinger]

Guido, any thoughts?

[benjaminp]

I think being able to delete free variables is reasonable and brings more consistency as well as solving corner cases like this.

[gvanrossum]

I don't think this has much to do with try/except. That it works in 2.6 but not in 3.0 isn't a big deal; the semantics of variables used in except clauses has changed dramatically.

It has to do with deletion of a variable that's held in a cell for reference by an inner function, like this:

def outer():
  x = 0
  def inner(): return x
  del x  # SyntaxError

I suspect (but do not know for sure) that the reason this is considered a SyntaxError is that the implementer of cells punted on the 'del' implementation and inserted a SyntaxError instead. (You can tell it's a pass-two SyntaxError because it doesn't mention a line number.)

I think it's fine to fix this in 2.7 and 3.1, but I don't see it as a priority given that this has always been this way (and despite that it now affects try/except). It will probably require a new opcode.

I don't see a reason to declare this a release blocker just because the try/except code is affected, and I don't think try/except needs to be changed to avoid this.

[f1e1d87a-4340-4801-8770-9e5b7119ec5a]

There's also this one which caught me out:

def outer():
  x = 0
  y = (x for i in range(10))
  del x  # SyntaxError

[03bde425-37ce-4291-88bd-d6cecc46a30e]

It's an interesting bug. Maybe the compiler shouldn't allow you to use such a variable as a free variable in a nested function?

On Thu, Feb 11, 2010 at 9:09 PM, Craig McQueen \report@bugs.python.org\ wrote:

Craig McQueen \python@craig.mcqueen.id.au\ added the comment:

There's also this one which caught me out:

def outer():  x = 0  y = (x for i in range(10))  del x  # SyntaxError

---------- nosy: +cmcqueen1975

---

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue4617\

---

---

Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/jeremy%40alum.mit.edu

[gvanrossum]

All examples so far (*) have to do with our inability to have properly nested blocks. If we did, I'd make the except clause a block, and I'd issue a syntax warning or error if a nested block shadowed a variable referenced outside it. Ditto for generator expressions and comprehensions.

As long as we don't have nested blocks, I think it's okay to see the limitation on (implicit or explicit) "del" of a cell variable as a compiler deficiency and fix that deficiency.

---

(*) However there's also this example:

>>> def f():
...  try: 1/0
...  except Exception as a:
...   def g(): return a
...   return g
... 
SyntaxError: can not delete variable 'a' referenced in nested scope
>>>

[03bde425-37ce-4291-88bd-d6cecc46a30e]

On Mon, Feb 22, 2010 at 6:10 PM, Guido van Rossum \report@bugs.python.org\ wrote:

Guido van Rossum \guido@python.org\ added the comment:

All examples so far (*) have to do with our inability to have properly nested blocks. If we did, I'd make the except clause a block, and I'd issue a syntax warning or error if a nested block shadowed a variable referenced outside it. Ditto for generator expressions and comprehensions.

There's no reason we couldn't revise the language spec to explain that except clauses and comprehensions are block statements, i.e. statements that introduce a new block. For the except case, there would be some weird effects.

y = 10
try:
  ...
except SomeError as err:
  y = 12
print y  # prints 10

In the example above, y would be a local variable in the scope of the except handler that shadows the local variable in the block that contains the try/except. It might be confusing that you couldn't assign to a local variable in the except handler without using a nonlocal statement.

As long as we don't have nested blocks, I think it's okay to see the limitation on (implicit or explicit) "del" of a cell variable as a compiler deficiency and fix that deficiency.

The general request here is to remove all the SyntaxErrors about deleting cell variables, right? Instead, you'd get a NameError at runtime saying that the variable is currently undefined. You'd want that change regardless of whether we change the language as described above.

hoping-for-some-bdfl-pronouncements-ly y'rs, Jeremy

---

(*) However there's also this example:

>>> def f(): ...  try: 1/0 ...  except Exception as a: ...   def g(): return a ...   return g ... SyntaxError: can not delete variable 'a' referenced in nested scope >>>

----------

---

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue4617\

---

[gvanrossum]

On Mon, Feb 22, 2010 at 6:51 PM, Jeremy Hylton \report@bugs.python.org\ wrote:

There's no reason we couldn't revise the language spec to explain that except clauses and comprehensions are block statements, i.e. statements that introduce a new block.

However (even apart from the below example) it would be tough to implement cleanly in CPython.

For the except case, there would be some weird effects.

y = 10 try:  ... except SomeError as err:  y = 12 print y  # prints 10

In the example above, y would be a local variable in the scope of the except handler that shadows the local variable in the block that contains the try/except.  It might be confusing that you couldn't assign to a local variable in the except handler without using a nonlocal statement.

Yeah, there are all sorts of problems with less-conspicuous nested scopes like this, for a language that defaults to local assignment like Python. Hence the horrible hacks.

> As long as we don't have nested blocks, I think it's okay to see the limitation on (implicit or explicit) "del" of a cell variable as a compiler deficiency and fix that deficiency.

The general request here is to remove all the SyntaxErrors about deleting cell variables, right?  Instead, you'd get a NameError at runtime saying that the variable is currently undefined.  You'd want that change regardless of whether we change the language as described above.

Yeah, if we could kill those SyntaxErrors we can leave the rest as is.

[amauryfa]

The above patch adds a new opcode (DELETE_DEREF), does the Moratorium apply here?

[gvanrossum]

I don't think so. It's very marginal.

--Guido (on Android)

On Feb 23, 2010 8:52 AM, "Amaury Forgeot d'Arc" \report@bugs.python.org\ wrote:

Amaury Forgeot d'Arc \amauryfa@gmail.com\ added the comment:

The above patch adds a new opcode (DELETE_DEREF), does the Moratorium apply here?

----------

---

Python tracker \report@bugs.python.org\ \<http://bugs.python...

[03bde425-37ce-4291-88bd-d6cecc46a30e]

The patch looks pretty good.

I'd factor out the common error-checking code (common between LOAD_DEREF and DELETE_DEREF) into a helper function.

It would also be good to add some test cases.

Jeremy

On Tue, Feb 23, 2010 at 9:38 AM, Guido van Rossum \report@bugs.python.org\ wrote:

Guido van Rossum \guido@python.org\ added the comment:

I don't think so. It's very marginal.

--Guido (on Android)

On Feb 23, 2010 8:52 AM, "Amaury Forgeot d'Arc" \report@bugs.python.org\ wrote:

Amaury Forgeot d'Arc \amauryfa@gmail.com\ added the comment:

The above patch adds a new opcode (DELETE_DEREF), does the Moratorium apply here?

----------

---

Python tracker \report@bugs.python.org\ \<http://bugs.python...

---------- Added file: http://bugs.python.org/file16341/unnamed

---

Python tracker \report@bugs.python.org\ \http://bugs.python.org/issue4617\

---

[eda57068-96ad-4b33-8431-9c528f59a6a6]

This bug is waiting for unit tests and a small patch cleanup. See previous message: http://bugs.python.org/issue4617#msg99950

[terryjreedy]

I have changed my mind on this issue. Since

e = 1
del e
def g(): print(e)
g()

compiles and raises a run-time name error, so should the same code embedded within a function. In either case, the premature deletion is a logic error, not a syntax error.

However, changing the language definition, even to fix what is considered a design bug, is a feature request. For both 2.7 and 3.1, section 6.5. "The del statement", says "It is illegal to delete a name from the local namespace if it occurs as a free variable in a nested block."

So this seems too late for 2.7. On the other hand, Guido has allowed it for 3.2 in spite of the moratorium, but I think it should go in the initial release.

[gvanrossum]

Yeah, please fix in 3.2, don't fix in 2.7.

[amauryfa]

Fixed in r84685, with tests and doc updates.