exec docs should note that the no argument form in a local scope is really the two argument form

[2052e752-d105-408e-86e9-af7f2b818b01]

BPO	24800
Nosy	@bitdancer, @eryksun
Superseder	bpo-23087: Exec variable not found error

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-feature', '3.8', '3.9', '3.10', 'docs'] title = 'exec docs should note that the no argument form in a local scope is really the two argument form' updated_at = user = 'https://bugs.python.org/PeterEastman' ``` bugs.python.org fields: ```python activity = actor = 'eryksun' assignee = 'docs@python' closed = False closed_date = None closer = None components = ['Documentation'] creation = creator = 'Peter Eastman' dependencies = [] files = [] hgrepos = [] issue_num = 24800 keywords = [] message_count = 9.0 messages = ['248064', '248068', '248075', '248076', '248077', '248079', '248091', '248097', '388247'] nosy_count = 4.0 nosy_names = ['r.david.murray', 'docs@python', 'eryksun', 'Peter Eastman'] pr_nums = [] priority = 'normal' resolution = None stage = 'needs patch' status = 'open' superseder = '23087' type = 'enhancement' url = 'https://bugs.python.org/issue24800' versions = ['Python 3.8', 'Python 3.9', 'Python 3.10'] ```

[2052e752-d105-408e-86e9-af7f2b818b01]

The following script demonstrates a bug in the exec() function in Python 3.4. (It works correctly in 2.7).

script = """
print(a)
print([a for i in range(5)])
"""
exec(script, globals(), {"a":5})

It produces the following output:

5
Traceback (most recent call last):
  File "test.py", line 5, in <module>
    exec(script, globals(), {"a":5})
  File "<string>", line 3, in <module>
  File "<string>", line 3, in <listcomp>
NameError: name 'a' is not defined

The variable "a" is getting passed to the script, as expected: printing it out works correctly. But if you use it in a comprehension, the interpreter claims it does not exist.

[bitdancer]

exec is subtle. See the explanation linked from bpo-23087, which while not *exactly* on point explains the underlying problem (a comprehension is a new scope, and exec can't reach an intermediate scope the way a compiled function can).

As far as the difference from 2.7 goes, the scoping rules for comprehensions changed in python3: the variable you are concerned with is now part of the local scope.

[2052e752-d105-408e-86e9-af7f2b818b01]

I don't believe that explanation is correct. You can just as easily get the same problem without explicitly passing a map to exec(). For example:

def f():
    script = """
print(a)
print([a for i in range(5)])
    """
    a = 5
    exec(script)

f()

The documentation for exec() states, "In all cases, if the optional parts are omitted, the code is executed in the current scope." Therefore the code above should be exactly equivalent to the following:

def f():
    a = 5
    print(a)
    print([a for i in range(5)])

f()

But the latter works and the former doesn't. Contrary to the documentation, the code is clearly not being executed in the same scope.

[bitdancer]

Yes it is. The comprehension is a *new scope, within the outer scope of the exec, and it *cannot see the variables in the outer scope of the exec. You have the same problem if you try to use a comprehension in that way in a class statement at the class level. An exec is explicitly *not* equivalent to a function body. It is equivalent to operating in a class body, if you give it two namespaces, and in a global context if you give it one. This is documented.

Please don't reopen the issue.

[2052e752-d105-408e-86e9-af7f2b818b01]

Then fix the documentation. This behavior directly contradicts the documentation of the exec() function. The question is not what scope the comprehension runs in, it's what scope the script runs in. See my third example. A comprehension in the f() function has no problem seeing local variables defined in that function. If the script were running into the same scope as that function, then comprehensions inside the script would also see those variables. They don't, clearly demonstrating that the script does *not* run in the same scope, and contradicting the documentation.

[bitdancer]

OK, it looks like what the documentation of exec is missing is the fact that calling exec with no arguments in a non-global is equivalent to calling it with *two arguments. That is, your "exec(script)" statement is equivalent to "exec(script, globals(), locals())". This is implicit but very much *not explicit in the current documentation, and should be made explicit.

To be sure I'm explaining this fully: the documentation of exec says "If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition".

>>> class Foo:
...   a = 10
...   [a for x in range(5)]
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in Foo
  File "<stdin>", line 3, in <listcomp>
NameError: name 'a' is not defined

[eryksun]

If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.

Probably there needs to be more clarification of the compilation context. Class definitions support lexical closures, whereas source code passed to exec is compiled at the time of the call, independent of the lexical context.

In the following example, the code objects for both the class body and the comprehension can access the free variable "a". In CPython, the class body references the free variable via the LOAD_CLASSDEREF op, and the comprehension uses the LOAD_DEREF op.

    def f():
       a = 5
       class C:
           print(a)
           print([a for i in range(5)])

    >>> f()
    5
    [5, 5, 5, 5, 5]

    >>> dis.dis(f.__code__.co_consts[2])
      3           0 LOAD_NAME                0 (__name__)
                  3 STORE_NAME               1 (__module__)
                  6 LOAD_CONST               0 ('f.<locals>.C')
                  9 STORE_NAME               2 (__qualname__)

  4          12 LOAD_NAME                3 (print)
             15 LOAD_CLASSDEREF          0 (a)
             18 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             21 POP_TOP

  5          22 LOAD_NAME                3 (print)
             25 LOAD_CLOSURE             0 (a)
             28 BUILD_TUPLE              1
             31 LOAD_CONST               1 (<code object <listcomp> ...>)
             34 LOAD_CONST               2 ('f.<locals>.C.<listcomp>')
             37 MAKE_CLOSURE             0
             40 LOAD_NAME                4 (range)
             43 LOAD_CONST               3 (5)
             46 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             49 GET_ITER
             50 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             53 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             56 POP_TOP
             57 LOAD_CONST               4 (None)
             60 RETURN_VALUE

    >>> dis.dis(f.__code__.co_consts[2].co_consts[1])
      5           0 BUILD_LIST               0
                  3 LOAD_FAST                0 (.0)
            >>    6 FOR_ITER                12 (to 21)
                  9 STORE_FAST               1 (i)
                 12 LOAD_DEREF               0 (a)
                 15 LIST_APPEND              2
                 18 JUMP_ABSOLUTE            6
            >>   21 RETURN_VALUE

[bitdancer]

OK, yes, so "a class body at global scope" or something like that :)

LOAD_CLASSDEREF is another whole level of complication to the scoping weirdness for classes; see bpo-19979 and bpo-24129.

[eryksun]

So there are a couple things to clarify here. When the documentation says "if the optional parts are omitted, the code is executed in the current scope", I think it should explicitly state that this is equivalent to calling exec(object, globals(), locals()). This should help to disabuse the reader of any assumption that the compiled code will extend the nested scoping (i.e. lexical closures) of the calling context.

When it says that if "exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition", I think this can be misleading. exec() compiles top-level code. It extends module-like execution, allowing globals and locals to differ and defaulting to the current scope. This sharply contrasts to code that's compiled for a class statement in the same context.

[sweeneyde]

I just closed https://github.com/python/cpython/issues/92681 as duplicate of this.

When the documentation says "if the optional parts are omitted, the code is executed in the current scope", I think it should explicitly state that this is equivalent to calling exec(object, globals(), locals()).

That sounds reasonable to me as well. I'd imagine that change would be a bit less controversial, so it could even happen independently of negotiating the "class definition" phrasing, if need be.

[iritkatriel]

cc @carljm who is changing this now with PEP 709.

[carljm]

I think PEP 709 doesn't really change the core of the discussion above (regarding how to more clearly document the behavior of exec), it just means that list/dict/set comprehensions are no longer useful repros to demonstrate the finer scoping distinctions. Replace the comprehensions with actual nested functions and the above repros still demonstrate the same behaviors of exec.

[ncoghlan]

(I am currently reviewing issues potentially addressed by the PEP 667 changes in Python 3.13)

While there have been several clarifications to the exec and eval docs as part of that, the key sentence in the exec docs at issue here remains the same as previously discussed: "In all cases, if the optional parts are omitted, the code is executed in the current scope."

The current wording in the eval docs is clearer:

The expression argument is parsed and evaluated as a Python expression (technically speaking, a condition list) using the globals and locals mappings as global and local namespace. ... other docs notes ... If the locals mapping is omitted it defaults to the globals dictionary. If both mappings are omitted, the expression is executed with the globals and locals* in the environment where eval() is called.

So for exec, rather than rewording just that sentence, I would suggest rewording that entire paragraph:

If globals and locals are given, they are used for the global and local variables, respectively. globals must specifically be a builtin dict instance (not a subclass), while locals may be any mapping object. If both namespaces are omitted, globals() and locals() from the calling frame are used implicitly (note: inside a function, this means the given code executes as if it were part of a nested class definition, not as if it were part of the function). If only globals is provided, it is also used as locals. If only locals is provided, globals is implicitly set to globals() from the calling frame.

(the last case is a currently undocumented situation arising from adding keyword argument support to exec and eval in Python 3.13)

To reduce duplication, I would also suggest amending the eval docs to refer down to exec for the explanation of the globals and locals parameters rather than repeating the same explanation in two places.

(although it's worth noting that 3.14 may be dropping the dict restriction for the globals arg: #121306)