PyIter_Next clears a StopException, where tp_iternext does not

[mattip]

Consider this code:

async def gen():
    yield 123

def test_last_yield(g):
    ai = g.__aiter__()
    an = ai.__anext__()
    try:
        next(an) 
    except StopIteration as ex:
        return ex.args
    else:
        return None 

ret = test_last_yield(gen())
assert ret[0] == 123

There are two ways to convert next(an) to C: either PyIter_Next(an) or an.tp_iternext(). There is a subtle difference: PyIter_Next will return NULL, signaling that the iteration is complete, and clear the StopIteration exception, erasing the ex.args. Calling tp_iternext will not clear the exception, so the yield value will be available through the ex.args. This came up in cython since it tends not to use the slot functions on non-CPython (in this case PyPy). As far as I can tell, there is no PyIter_* function to only call tp_iternext without clearing the exception.

[gvanrossum]

(This one sounds like deja vu to me -- I wonder if we've done a similar inventory in the past.)

Another problem with PyIter_Next is that its NULL return is ambiguous -- it can mean an actual error occurred, or the end was reached. This particular API seems to be poorly designed, and we ought to replace it with something else that looks more complex but avoids the issues.

[vstinner]

See also https://github.com/capi-workgroup/decisions/issues/21

[serhiy-storchaka]

Note also that setting a StopIteration in tp_nextitem is optional. To signal the end of iteration it should either set a StopIteration and return NULL, or return NULL without setting any error.