yield expression inside generator expression does nothing

[dd5ce2ff-95a1-4e28-ace2-6d841dd59913]

BPO	10544
Nosy	@gvanrossum, @birkenfeld, @rhettinger, @ncoghlan, @abalkin, @benjaminp, @glyph, @serhiy-storchaka, @1st1, @ajdavis, @DimitrisJim
PRs	python/cpython#4564 python/cpython#4579 python/cpython#4676
Files	yield-in-comprehensions.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['interpreter-core', 'type-bug', '3.7'] title = 'yield expression inside generator expression does nothing' updated_at = user = 'https://bugs.python.org/InyeolLee' ``` bugs.python.org fields: ```python activity = actor = 'serhiy.storchaka' assignee = 'none' closed = True closed_date = closer = 'serhiy.storchaka' components = ['Interpreter Core'] creation = creator = 'Inyeol.Lee' dependencies = [] files = ['47296'] hgrepos = [] issue_num = 10544 keywords = ['patch'] message_count = 66.0 messages = ['122475', '122512', '122645', '122658', '122705', '122759', '122760', '122762', '122765', '123641', '123649', '241410', '286273', '286274', '286280', '286281', '286282', '286283', '286407', '286408', '286409', '306784', '306802', '306804', '306805', '306806', '306807', '306808', '306810', '306812', '306829', '306831', '306832', '306833', '306834', '306835', '306836', '306837', '306838', '306840', '306841', '306842', '306843', '306844', '306845', '306846', '306847', '306939', '306940', '306963', '306964', '306983', '306995', '307013', '307033', '307035', '307039', '307051', '307067', '307091', '307095', '307359', '307361', '307426', '307449', '311593'] nosy_count = 15.0 nosy_names = ['gvanrossum', 'georg.brandl', 'rhettinger', 'ncoghlan', 'belopolsky', 'benjamin.peterson', 'erickt', 'glyph', 'Inyeol.Lee', 'serhiy.storchaka', 'yselivanov', 'esc24', 'danielsh', 'emptysquare', 'Jim Fasarakis-Hilliard'] pr_nums = ['4564', '4579', '4676'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'behavior' url = 'https://bugs.python.org/issue10544' versions = ['Python 3.7'] ```

[serhiy-storchaka]

Here is a sample of the implementation of the Nick's idea.

[ncoghlan]

Serhiy's PR now implements the "Prohibit yield & yield from in generator expressions and comprehensions" approach discussed on python-dev (currently as a hard SyntaxError, but it could be amended to be a warning instead without too much difficulty).

The PR does highlight an interesting subtlety though: the easiest way to implement this still allows yield expressions in the outermost iterator, since that gets compiled in a different scope from the rest of the comprehension body (it's evaluated eagerly and passed in to the implicit nested function).

I'm thinking we probably don't want to expose that detail to end users, and will instead want to include a second check that prohibits yield expressions in the outermost iterator as well. However, I'm not entirely sure how we could implement such a check, short of adding a new "yield expression prohibited" counter in the AST generation and/or symbol analysis pass.

[gvanrossum]

No.

On Nov 25, 2017 18:01, "Nick Coghlan" \report@bugs.python.org\ wrote:

Nick Coghlan \ncoghlan@gmail.com\ added the comment:

Serhiy's PR now implements the "Prohibit yield & yield from in generator expressions and comprehensions" approach discussed on python-dev (currently as a hard SyntaxError, but it could be amended to be a warning instead without too much difficulty).

The PR does highlight an interesting subtlety though: the easiest way to implement this still allows yield expressions in the outermost iterator, since that gets compiled in a different scope from the rest of the comprehension body (it's evaluated eagerly and passed in to the implicit nested function).

I'm thinking we probably don't want to expose that detail to end users, and will instead want to include a second check that prohibits yield expressions in the outermost iterator as well. However, I'm not entirely sure how we could implement such a check, short of adding a new "yield expression prohibited" counter in the AST generation and/or symbol analysis pass.

----------

---

Python tracker \report@bugs.python.org\ \https://bugs.python.org/issue10544\

---

[gvanrossum]

To clarify, the outermost iterator is marked here:

[func(x, y) for x in <outermost iterator> for y in <something else>]

and it is evaluated before the comprehension proper is started. This is just part of how the language works (it's a similar rule as "in an assignment the RHS is evaluated before it is assigned to the LHS").

I see no benefit in banning yield in .

On Sat, Nov 25, 2017 at 10:22 PM, Guido van Rossum \report@bugs.python.org\ wrote:

Guido van Rossum \guido@python.org\ added the comment:

No.

On Nov 25, 2017 18:01, "Nick Coghlan" \report@bugs.python.org\ wrote:

> > Nick Coghlan \ncoghlan@gmail.com\ added the comment: > > Serhiy's PR now implements the "Prohibit yield & yield from in generator > expressions and comprehensions" approach discussed on python-dev (currently > as a hard SyntaxError, but it could be amended to be a warning instead > without too much difficulty). > > The PR does highlight an interesting subtlety though: the easiest way to > implement this still allows yield expressions in the outermost iterator, > since that gets compiled in a different scope from the rest of the > comprehension body (it's evaluated eagerly and passed in to the implicit > nested function). > > I'm thinking we probably don't want to expose that detail to end users, > and will instead want to include a second check that prohibits yield > expressions in the outermost iterator as well. However, I'm not entirely > sure how we could implement such a check, short of adding a new "yield > expression prohibited" counter in the AST generation and/or symbol analysis > pass. > > ---------- > > > Python tracker \report@bugs.python.org\ > \https://bugs.python.org/issue10544\ > >

----------

---

Python tracker \report@bugs.python.org\ \https://bugs.python.org/issue10544\

---

[ncoghlan]

Cool, if you're OK with that behaviour, it actually makes this a lot easier, since it means:

1. Serhiy's patch is already sufficient for the final hard compatibility break
2. It can be readily adapted to emit either DeprecationWarning or SyntaxWarning for 3.7

[gvanrossum]

Great. It should be a DeprecationWarning, since we're planning to disallow it completely, right? IIRC SyntaxWarning is for syntax that we can't deprecate.

[ncoghlan]

Guido, should we write this change up as a PEP, or are you happy to just cover it as a section in the What's New document for 3.7?

[serhiy-storchaka]

Note that the DeprecationWarning exception is replaced with a SyntaxError to get a more accurate error report.

$ cat yield-gen.py 
def f():
    return ((yield x) for x in range(3))

$ ./python -Wd yield-gen.py
yield-gen.py:2: DeprecationWarning: 'yield' inside generator expression
  return ((yield x) for x in range(3))

$ ./python -We yield-gen.py
  File "yield-gen.py", line 2
    return ((yield x) for x in range(3))
           ^
SyntaxError: 'yield' inside generator expression

Without this replacement the result would be:

$ ./python -We yield-gen.py
DeprecationWarning: 'yield' inside generator expression

[gvanrossum]

After the tiresome debate I am happy to see this just as a "what's new" entry rather than soliciting more debate with a PEP. (However there may be some existing PEP that suggests it should work? That PEP should be amended with a note that this is being deprecated. But I don't know if there is such a PEP.)

[ncoghlan]

As far as I'm aware, there's nothing that specifically promises these constructs will do anything in Py3 at all - the existing behaviour is just an accident of implementation arising from the way nested scopes and yield expressions interact in general.

Tinkering with "await" in comprehensions and generator expressions would be different, since PEP-530 actually does make promises about how we expect that to work.

[gvanrossum]

OK, great.

[ncoghlan]

New changeset 73a7e9b10b2ec9636e3c6396cf7b3695f8ed1856 by Nick Coghlan (Serhiy Storchaka) in branch 'master': bpo-10544: Deprecate "yield" in comprehensions and generator expressions. (GH-4579) https://github.com/python/cpython/commit/73a7e9b10b2ec9636e3c6396cf7b3695f8ed1856

[ncoghlan]

With Serhiy's patch merged, I'm marking this as resolved. Thanks all!

https://bugs.python.org/issue32189 is the follow-up issue to turn the warning into an unconditional SyntaxError in 3.8.

[serhiy-storchaka]

PR 4676 backports warnings to 2.7 in Py3k mode.

[serhiy-storchaka]

New changeset 65d1887170fb278c10a836e9e4319cae4707f524 by Serhiy Storchaka in branch '2.7': [2.7] bpo-10544: Deprecate "yield" in comprehensions and generator expressions in Py3k mode. (GH-4579) (bpo-4676) https://github.com/python/cpython/commit/65d1887170fb278c10a836e9e4319cae4707f524

[serhiy-storchaka]

New changeset 07ca9afaa8768b44baf816b4998d209ed3e0088f by Serhiy Storchaka in branch 'master': bpo-10544: Disallow "yield" in comprehensions and generator expressions. (GH-4564) https://github.com/python/cpython/commit/07ca9afaa8768b44baf816b4998d209ed3e0088f