Open pablogsal opened 2 years ago
This is great! @markshannon will write something about the kinds of numbers we'd like to see.
I think there are a few things that would be useful.
@markshannon For the perf breakdown you means something like % of every opcode?
Here's a progress report from @pablogsal:
I talked with legal and other related departments and this aspect is already sorted out. We are still pending a talk with CTO, but I don't expect anything relevant from them.
We already have identified parts of the application that are big enough and critical enough to provide good benchmark candidates.
Currently we are working on what's going to be the most annoying part: there are many dependencies of this application that don't work on the current main branch due to the following:
Lack of wheels for 3.11.
Cython not working for 3.11.
Changes to the C-API
Also, there are some internal dependencies in Bloomberg packages that we need to ensure that also work on main.
We are currently investigating what's the best way to obtain something that we can benchmark without having to fix all of these or if we can substitute those dependencies for other pure python ones
When this is sorted, we want to make an investigation into the stability of the benchmarks to ensure we are not going to have lot of noise. The application is big enough that although there is no big I/O on the middle, is not trivial to know how stable the benchmarks are.
Once all this is sorted, we will hook this up to pyperformance for easy testing of new versions.
As you can see, we are progressing, but the biggest pain is going to be figuring out that this works for 3.11, which is a bit of packaging hell, but I think we will get there.
@markshannon For the perf breakdown you means something like % of every opcode?
No. Just the usual perf output: %time in each C function, etc.
Hi,
I spotted this project and I have a thought - it was said that there is a need for a "production" like application. I wonder - maybe it would be worth also using https://github.com/locustio/locust framework as a reference point.
This is quite a popular framework for doing performance tests - an alternative for JMeter / Gatling / k6.
It has master/worker concept and uses requests/urllib3 | gevents | ZMQ. So I guess "normal" usage of the basic libs. Apart from that there would be some "basic" code usage of lists etc.
People might add their own code on top of it - like here is a bunch of plugins - https://github.com/SvenskaSpel/locust-plugins My team has own code to send stats to InfluxDB or process data on websockets.
I guess any optimization in the usage of libraries underneath like requests/SSL will bring gains that would be visible as the possibility to generate more Virtual Users per worker/CPU. Normally when too many Virtual Users are generated on a single CPU - it will get exhausted and the context switch will affect response time giving unrealistic values. This is the point when more workers have to be added to the pool.
This could work hand in hand with some very basic Flask application that would just serve as a testing point and return " hello world" - although you could embed some "real code" underneath to have a more realistic scenario of application usage. I can imagine it could save/get data using Mongo or call simple SQL with some other DB. The logs would interact with disk giving you and I/O stats.
Depending on the scenario you could observe - CPU usage on Locust side - with same load and optimization usage should drop. At the same time optimization on the server side will give probably a slightly higher RPS and/or faster response time to the user - it is also possible that CPU usage will change.
I'm not sure how memory allocation would change - I guess driver will be the number of concurrent sessions.
I'm not sure if this is something that you are looking for.
It looks like Locust is mainly intended for app developers to be able to create flexible testing scenarios for their web apps (I presume by creating synthetic loads). That's not exactly what we're looking for here -- we're looking for apps that are already developed for which we can measure their performance under a load already defined. So if you have a particular app that you don't mind sharing and you have developed a Locust-based synthetic load for it, we would love to be able to compare how your app performs under Python 3.10 and 3.11. But if you have some other way of stress-testing your app we would be just as happy -- as long as your app can run under 3.10 and 3.11 (or, hopefully, what's in main, but I don't want to press my luck :-).
It looks like Locust is mainly intended for app developers to be able to create flexible testing scenarios for their web apps (I presume by creating synthetic loads). That's not exactly what we're looking for here -- we're looking for apps that are already developed for which we can measure their performance under a load already defined. So if you have a particular app that you don't mind sharing and you have developed a Locust-based synthetic load for it, we would love to be able to compare how your app performs under Python 3.10 and 3.11. But if you have some other way of stress-testing your app we would be just as happy -- as long as your app can run under 3.10 and 3.11 (or, hopefully, what's in main, but I don't want to press my luck :-).
We have a FastAPI microservice (py3.10) in production for which we've developed tests in locust to repeat the requests we get in production. It uses two redis clusters, queries solr using aiohttp, and finally queries a model on a (different) Tensorflow Serving VM via grpcio. All of these tasks are async & it also has a few list, dict, & NumPy ops. I think I should be able to run it with py3.11 too. I don't think I can share the code verbatim, but I can perform benchmarks on isolated VMs & share a few critical sections. Would you be interested in benchmarks of such an app?
I would be super interested if you could do those benchmarks! In particular comparing 3.10 and 3.11 in a real world app would be huge. I am prepared to be disappointed, because usually real-world performance is not just about CPU usage of the language the main app is written in... But nevertheless I would love to hear from you.
I would be super interested if you could do those benchmarks! In particular comparing 3.10 and 3.11 in a real world app would be huge. I am prepared to be disappointed, because usually real-world performance is not just about CPU usage of the language the main app is written in... But nevertheless I would love to hear from you.
All times are in milliseconds.
Python 3.10.7: Mean: 33 Median: 23-24 (mostly 23) 90%: 41 95%: 57-62 99%: 200
Python 3.11.0rc2 Mean: 32 Median: 21-22 (mostly 22) 90%: 39 95%: 54-58 99%: 190
Memory usage was almost the same, but CPU usage looked slightly lower. But I can't be more specific without a graph.
Nonetheless, this is an impressive result considering that this service is very database intensive (Around 4-10 redis gets, 1-6 redis sets, 1-5 solr queries, & 1 TF serving call). But it also has a lot of list & dict comprehensions as well as multiple dicts with 700K & 1M elements, which I think is responsible for the performance improvement.
I'll see if I can share the source code by obfuscating some parts. It might be better if I replace the database calls completely with sleep
so that you could run it on your side.
Fun fact: I had rewritten this service in rust (with actix-web & async for all database calls) & I got 17-18ms median latency with it. I think this makes the 3.11 improvement more impressive if we consider the python overhead compared to rust.
Thanks for the quick results! It sounds like 3-5% faster, which for a real-world app is not easy.
Might be interesting to gather 3.11 stats, if you're comfortable compiling Python from source. This basically dumps tons of internal counters that allow us to see how well the interpreter is handling your program (without giving us access to the source of the program itself).
Basically, configure --with-pystats
, build, make an empty /tmp/py_stats
directory, run your program, and run Tools/scripts/summarize_stats.py
from the main repo using the same Python. It will dump a few thousand lines of markdown to stdout, which you can drop into a comment here. To see what type of information is gathered, see this example.
Might be interesting to gather 3.11 stats, if you're comfortable compiling Python from source. This basically dumps tons of internal counters that allow us to see how well the interpreter is handling your program (without giving us access to the source of the program itself).
Basically, configure
--with-pystats
, build, make an empty/tmp/py_stats
directory, run your program, and runTools/scripts/summarize_stats.py
from the main repo using the same Python. It will dump a few thousand lines of markdown to stdout, which you can drop into a comment here. To see what type of information is gathered, see this example.
Sure, I'll try it tomorrow.
Might be interesting to gather 3.11 stats, if you're comfortable compiling Python from source. This basically dumps tons of internal counters that allow us to see how well the interpreter is handling your program (without giving us access to the source of the program itself).
Hmm, I will try to ask my company's (LINE/NAVER) machine-learning team(I can sure that they have enough workload for the APAC area, especially for Korea/Japan) if they can share bytecode execution information.
Might be interesting to gather 3.11 stats, if you're comfortable compiling Python from source. This basically dumps tons of internal counters that allow us to see how well the interpreter is handling your program (without giving us access to the source of the program itself).
Basically, configure
--with-pystats
, build, make an empty/tmp/py_stats
directory, run your program, and runTools/scripts/summarize_stats.py
from the main repo using the same Python. It will dump a few thousand lines of markdown to stdout, which you can drop into a comment here. To see what type of information is gathered, see this example.
I built python by modifying this Dockerfile which is used in the official py3.11.0 docker builds. I just added --enable-pystats
(--with-pystats
doesn't work) to the configure step.
Here is the output after 15k requests:
Stats gathered on: 2022-10-31
We should gather these results somewhere. Presumably in the stats/
folder
After some talk with @gvanrossum we discuss the possibility of gathering benchmarks from big production-grade applications. I talked with my employer (Bloomberg) and we have a couple of very large, performance-critical Python applications we could use as a performance target (as another source of data). I discussed the possibility of getting some engineer hours from the teams in charge of these applications for helping to build some kind of automated system that allows running some benchmark based on production data with different versions of CPython (so we can compare different commits in the
main
branch or for different proposals).I open this issue to gather first what kind of "requirements" or data are we interested in so we can talk internally on how to prepare and build this. For instance, are we just looking at "time it takes per request" or "time per opcode" or something like that.
As a note, the code of the application itself cannot be made public for obvious reasons but (after checking with the legal department) we can possibly discuss the general nature of the applications if there are questions.