Debug failing tests under pypy with stackless backend

[rgalanakis]

Stackless Python 2.7 works fine, as does vanilla 2.7 with gevent, but we get terrible behavior under PyPy with stackless/continulets. We need to figure out what is up.

[aukaost]

I'll have a look.

[aukaost]

https://github.com/surfly/gevent/pull/434

[rgalanakis]

Have you tried that out? I am not sure if the readme for that branch is still up to date, seems very experimental. Or was everything working better (tests passing, no-jit crash, etc) by the time it was merged in? Things don't install correctly, at least, but I didn't follow that branch's instructions yet. Update here if you get a chance to try it, I'll do likewise.

[aukaost]

I'll look into it during the upcoming Reykjavik Python group sprint.

[aukaost]

All tests pass using PyPy 2.3 + gevent (@f315c1eb70)!

[rgalanakis]

Wow! Awesome! Close this if someone else can verify :) Do you want to look into getting a release cut for pypi?

[aukaost]

Yup, I'll have a look. I'll also add a note to the README regarding PyPy.

[chuim]

Cool stuff! I might be able to get back to participating soon... :-) On May 31, 2014 8:54 AM, "aukaost" notifications@github.com wrote:

Yup, I'll have a look. I'll also add a note to the README regarding PyPy.

— Reply to this email directly or view it on GitHub https://github.com/rgalanakis/goless/issues/2#issuecomment-44747567.

[rgalanakis]

Confirmed working with PyPy 2.2.1 and head of surfly/gevent repo in GitHub.

[rgalanakis]

Wait, misunderstood this ticket. Yes, PyPy works with gevent, but stackless bug remains. Leaving this open, opening a new ticket for choosing gevent by default under pypy.

[ctismer]

The stackless module was never completely correct. I'm wondering if it would be better to consider a special backend on top of the continulet?

http://pypy.readthedocs.org/en/latest/stackless.html

That would remove all the extra overhead that stackless.py tries to do right on PyPy to build certain things that you do not need at all. Not sure - do you use the stackless channels, or didn't you build your own, anyway?

The advantage of a continulet-based implementation would be not only a very simple model, but this scales since it can co-exist with other concepts built on top of continulets.

Actually, if there was a chance to change history, I would like to re-write Stackless to have continulets in CPython as the building blocks, and the current stackless is just a special case.

[rgalanakis]

We use stackless channels as the underlying channel implementation (see goless/goless/backends.py). Same with gevent channels. The goless channels just provide slightly different semantics (they are similar to some special channel types in stacklesslib).

Creating a new backend that works on top of continulets is a possibility. I have a feeling, though, that it would end up very similar to stackless.py. Keep in mind that stackless.py works in the normal cases, it is just the extreme failures and stress testing due to our test suite that teases out some problems. Though perhaps writing a new backend would allow us to uncover the issue, and fix it in stackless.py.

Finally it's also worth nothing that the problem may be in the PyPy continulets, or the runtime itself, since we are segfaulting, not just causing a Python error (IIRC).

[ctismer]

Ah right, sorry, I found it again. Was a bit mislead, and my eye vision is bad.

Then, true, you would end up not so far from stackless.py. Maybe you would consider another version with less cruft of stackless compatibility on channels and your own scheduler, closer to GoLang's scheduler at some point, avoiding a scaling issue... tossing ideas... Yes, you might consider at some point to clone stackless.py, which is already continulet-based, use a different name and tailor this to goless.

But on the other hand, I guess you are already close to the optimum ;-)

Interesting that there is a segfault! Probably worth telling Armin.

[MichaelAz]

The equivalent vanilla stackless code works fine:

import stackless

chan = stackless.channel()
data2send = list(range(5))
for data in data2send:
    t = stackless.tasklet(chan.send)(data)
    t.run()

chan.close()

But, just by replacing the channel with any goless channel (SyncChannel, AsyncChannel, and various sizes of BufferedChannel) causes an error. Using a SyncChannel causes an assert in stackless.py to fail, while using an AsyncChannel (or any buffered channel, for that matter) outright segfaults.

I've been able to prevent the segfault and reduce the number of failing tests (both on stackless.py and vanilla stackless). All remaining failures in tests are (I suspect) related to yielding from the main tasklet. The gevent tests still pass, of course.

You can see the work at 4faeb53b15c2836e3c98466e9baa26128791fd0d and e737a986189c94993335d178500bfd0402bd0139.

I think the question is, now, what to do with the "yield from main tasklet" case. Should it just act as a pass, "switching" to itself? I think that makes sense but breaks current behavior. What do you guys think?

[rgalanakis]

I was wondering about the "yield from main tasklet" case and just realized how simple the answer is. We should just copy Go's behavior :) It will panic/exit, so we should raise an exception. I am pretty sure stackless and gevent do this; goless will (right now) catch the error and silence the yield, which in hindsight isn't a good idea. Or maybe it is and we should break with Go on this point. Would like to get more input here.