Closed dkelsey closed 6 months ago
I've had success using httpx with more complicated trio applications, so I'm thinking locust must do something very strange. Probably not the same thing as gunicorn though (re your question: gunicorn fails because it removes the primitives we need to wait for files).
EDIT: nevermind, locust does use gevent. see https://github.com/locustio/locust/blob/baf007f1c9f243a550621ecc99720d835513e220/docs/changelog.rst#L789-L790
There's no satisfying solution IMO because if trio reaches into gevent's API to pull out the original kqueue, then I think we're blocking the greenlet? Which is obviously not what the user wants IMO.
Maybe we should just fall back to gevent's patched version of something? I'm not too familiar with the underlying mechanics of trio to say what. Hopefully someone else has an answer!
EDIT 2: Heh, looks like they did some special casing on their end re: gevent w/ trio! See https://github.com/locustio/locust/commit/aedc3b919ff3983ea3af678ad4861279668412d1
sounds like something to bring up on their end in that case. Either to re-fix their workaround, or to see if they know how we should work around the patch on our end. Is this a thing that's broken in one of our recent releases, given that we're getting several reports now? Or new version of gevent/other?
Is it possible to do backend='trio'
?
@CoolCat467 is this something you are asking me to try?
I added backent'trio'
as a param to httpx.AsyncClient
I get the same errror.
I ran both scripts in debug (pyCharm).
It lands on this part of trio/_core/_run.py
...
elif TYPE_CHECKING or hasattr(select, "kqueue"):
from ._generated_io_kqueue import *
from ._io_kqueue import (
EventResult as EventResult,
KqueueIOManager as TheIOManager,
_KqueueStatistics as IOStatistics,
)
TYPE_CHECKING
is False
hasattr
inspects select-cpython-311-darwin.so
and finds kqueue
exists.<module 'select' from '/Users/dkelsey/.pyenv/versions/3.11.4/lib/python3.11/lib-dynload/select.cpython-311-darwin.so'>
I place a break point in _run.py
and ran the locust script in debug.
The same check occurs:
<module 'select' from '/Users/dkelsey/.pyenv/versions/3.11.4/lib/python3.11/lib-dynload/select.cpython-311-darwin.so'>
kqueue
kqueue
go?It was monkey patched away?
Note:
I followed this to get debugging working with locust/gevent in pyCharm:
https://github.com/locustio/locust/issues/613#issuecomment-891404020
Also added this to the locust script at the end (and imported it):
if __name__ == "__main__":
run_single_user(MyUser)
After all this... I re-read the comments from @A5rocks and @jakkdl I now understand what you wrote.
So how to get httpx.AsyncClient
or trio
to work in locust
?
A hacky solution would be to edit /Users/dkelsey/.pyenv/versions/HTTPX/bin/locust
to try import trio
before anything else, which will cache trio
. Obviously this is a no-go for various reasons.
The best solution here (IMO) is to use non-async httpx, and use gevent if you need "async" stuff. Unfortunately it looks like locust is built around this.
As for fixing this in trio, I'm starting to lean more towards the broken-solution to this where we reach into gevent and pull out kqueue
(there's a way to do that) but this probably requires more thought.
Actually I thought about this a bit more, and select.kqueue = gevent.monkey.get_original("select", "kqueue")
before you import trio (with the necessary imports, of course) might work in your locustfile.py
? I'm on mobile so I can't check whether you can assign to module contents like that.
Obviously this is a bad solution but less so than my previous one where you had to patch locust!
Found this in the locust docs here: testing other systems
It is important that the protocol libraries you use can be monkey-patched by gevent.
A hacky solution would be to edit
/Users/dkelsey/.pyenv/versions/HTTPX/bin/locust
to tryimport trio
before anything else, which will cachetrio
. Obviously this is a no-go for various reasons.The best solution here (IMO) is to use non-async httpx, and use gevent if you need "async" stuff. Unfortunately it looks like locust is built around this.
As for fixing this in trio, I'm starting to lean more towards the broken-solution to this where we reach into gevent and pull out
kqueue
(there's a way to do that) but this probably requires more thought.
My intention is to send multiple multiplexed requests for resources through a single connection.
I'll reiterate that I think you should open an issue in locust and see if they can re-fix their workaround. Getting kqueue
support into gevent
or general gevent
support in trio
both sound like quite tough tasks.
Thanks @jakkdl those are good recommendations.
I found a way to achieve what I want useing pycurl in locust, writing a client and user class following the locust docs on testingnother services.
I'm satisfied. Thanks again.
Description:
testTrio.py
) that useshttpx.AsyncClient
withtrio
. This works correctly when I run it.locustfile.py
that also useshttpx.AsyncClient
. It fails withNotImplementedError: unsupported platform
.Working Script
testTrio.py
Output
Non-working
locustfile.py
Output
Environment:
OS: Mac M1 Python: 3.11.4 (venv) Locust:
httpx:
trio:
Line
2800
oftrio/_core/_run.py
Questions