Closed yuvipanda closed 6 years ago
So far, we've:
A big problem we ran into was that we were constantly hit by redirect loops. Me and @minrk dug through this (yay teamwork? I made some comments and went to sleep and he had solved it when I woke up!), and https://github.com/jupyterhub/jupyterhub/pull/1221 seems to get rid of them all!!!
When I ran a 2000 user test, about 150 failed to start. Some of them were because of this curl error:
[E 2017-07-14 20:09:46.147 JupyterHub ioloop:638] Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7f8cfbdc1ea0>, <tornado.concurrent.Future object at 0x7f8c30a87048>)
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tornado/ioloop.py", line 605, in _run_callback
ret = callback()
File "/usr/local/lib/python3.5/dist-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tornado/ioloop.py", line 626, in _discard_future_result
future.result()
File "/usr/local/lib/python3.5/dist-packages/jupyterhub/app.py", line 1382, in update_last_activity
routes = yield self.proxy.get_all_routes()
File "/usr/local/lib/python3.5/dist-packages/jupyterhub/proxy.py", line 556, in get_all_routes
resp = yield self.api_request('', client=client)
tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20553 milliseconds with 163693 bytes received
This could patially be fixed by tuning pycurl. However, it failed a 20s timeout, which is pretty generous. I think a k8s ingress based Proxy implementation would help here, and might be the next bottleneck to solve.
The spawn times increased linearly with time, eventually going over the 300s timeout we have for image starts. We should take a profiler to the hub process (not sure how to profile tornado code tho haha) and figure out what's the thing that takes linear amounts of time.
Ran another 5000, and am now convinced that next step is a proxy implementation that does not need to make a network request for each get_all_routes()
I suspect that slowness of get_all_routes
may be a bit misleading, because lots of actions are using the same IOLoop and even httpclient waiting for Spawners to start, etc. My guess is that it's all tied up in the busy-ness of the Hub IOLoop. Plus, get_all_routes
only happens once every five minutes, so it seems a bit odd for it to be a major contributing factor.
When I run get_all_routes on its own when things are otherwise idle, it takes 50ms to fetch 20k routes, so it's unlikely that the fetch is actually what's taking that time. My guess is that it's something like:
Under load, presumably CHP would take longer to respond to the request, but 20s seems like a lot compared to 50ms.
I think it's important to separate what we're testing as concurrent:
In particular, JupyterHub is based on the idea that login/spawn is a relatively infrequent event compared to actually talking to their server, so while stress testing all users concurrently spawning is useful as a worst-case scenario, it doesn't give an answer to the question "how many users can I support" because it is not a realistic load of concurrent users.
I'd love a load test that separated the number of people trying to spawn from the number of people concurrently trying to use their running notebook servers (opening & saving, running code, etc .). e.g. being able to ask the questions:
+1 on load testing a matrix of factors!
We should also identify what 'performance' means. I think it's:
Anything else, @minrk? If we nail these down as things we wanna measure, then we can form the test matrix properly...
This is awesome @yuvipanda
I think ideal way for us to stress test is that we can define a function that'll draw a 'current active users' graph - spikes will get more useres logging in, slumps will have users dropping out, and steady states will do nothing. This will let us generate varying kinds of load by just tweaking the shape of this graph (which you can do interactively).
That looks like a good starting point.
Some 'performance' metrics:
Hub:
Singleuser (spawned and ready to go):
So I did more stuff around this in the last couple of days.
I also filed https://github.com/jupyterhub/jupyterhub/issues/1253 - that's the most CPU intensive hotspot we've found. Plus there are lots of well tested libraries for this we should use.
Filed https://github.com/kubernetes-client/python-base/issues/23 for issues with kubernetes api-client.
Update: It's most likely a PyPy bug: https://bitbucket.org/pypy/pypy/issues/2615/this-looks-like-a-thread-safety-bug
We can offload TLS to a kubectl proxy
instance that's running as a sidecar container, providing a http endpoint that's bound to localhost. I've filed https://github.com/kubernetes-incubator/client-python/issues/303 on figuring out how to do this.
Me and @minrk tracked down the SQLAlchemy issues to https://github.com/jupyterhub/jupyterhub/pull/1257 - and it's improved performance considerably!
hashlib is next step!
More notes is:
Move all sqlalchemy operations to a single external thread (not a threadpool probably)
This is probably the hardest one, because we have to make sure that we bundle db operations within a single synchronous function that we pass to a thread, whereas right now we only have to ensure that we commit in between yields.
Additional point - make sure that each tornado tic isn't doing too much work. We want it to be as evenly spread out as possible - so we don't get tocs that are too long followed by tocs that are too small. Am implementing some jitter into the exponential backoff algorithm for this now...
We fixed a ton of stuff doing this! YAY!
Sorry but how do we run the stress test. I don't think I see any tutorial yet. Correct me if I'm wrong.
I want to be able to simulate 5000 concurrent users, doing some amount of notebook activity (cell execution) and getting results back in some reasonable time frame.
This will require that we test & tune:
Ideally, we want Kubernetes itself to be our bottleneck, and our components introduce no additional delays.
This issue will document and track efforts to achieve this, as well as to define what 'this' is.