JoeyHendricks
commented
3 years ago I am very happy to hear you like QuickPotato and thank you for the compliments this project has been my pandemic hobby project!
Yes, the performance breakpoint was not yet "feature complete" mainly because previous feedback was more like "I don't want to be intrusive in my production code for testing purposes." Out of that, the non-intrusive construct was born because it was not intrusive and would allow for easier use for creating "micro" performance tests for your code.
So I started mainly to see the performance breakpoint as a more advanced feature to better control profiling and use it for custom stuff. I never removed it because it goes against the vision of the framework and was planning of improving it soon.
The vision of QuickPotato is to make profiling more flexible. easier in use for automation purposes and allow developers to write focussed performance tests for their code so performance assessment happens earlier in the development life cycle. I have found these concepts missing in similar frameworks so that is why this project exists.
Do be warned this decorator does add an overhead to your function so for monitoring purposes in production you might be better of using an APM as they provide "somewhat" the same insight against lesser overhead. But if you don't wanna use any fancy APM tooling (They cost a lot of money if you want a decent one.) you can also just flick the decorator on and off using the parameters on specific intervals as a sort of "advanced logging".
After reading what you want to achieve with the "performance breakpoint" I made the following changes to the master branch (This is always my unstable edge version):
- I have changed the way how you have to define and create a database no longer has this be an explicit copy of the test case name but you can now define it yourself. (This was indeed very confusing thank you for pointing that out!)
- I have made it possible to push in a custom test case name into the decorator
- Now you can also generate a custom test id yourself to control the test identification yourself.
All of this would look like this in regular old Python code:
# Example code is not on GitHub this is a simple CRUD app to manually test QuickPotato.
from DummyApp.main import ToDoList
from QuickPotato.profiling.intrusive import performance_breakpoint
from QuickPotato.configuration.management import options
from QuickPotato.statistical.visualizations import FlameGraph, CsvFile, HeatMap, BarChart
# Settings and stuff
options.connection_url = "mysql+pymysql://root:password@localhost"
options.enable_intrusive_profiling = True # <-- Make sure that profiling is enabled
# Building the class that we will use in the test
instance = ToDoList()
# Parameters used by the break point
rand_test_case_name = "I am a test case"
rand_database_name = " I am a database name"
rand_test_id = "12345678910"
# Decorating the function
@performance_breakpoint(test_case_name=rand_test_case_name, database_name=rand_database_name, test_id=rand_test_id)
def dummy_function():
instance.add_action_to_board(payload=[{"Action": "Do Something", "Person": "Joey Hendricks", "DueDate": "NOW"}])
# Running the test
for _ in range(0, 10):
dummy_function()
# Exporting some visuals
FlameGraph(test_case_name=rand_test_case_name, database_name=rand_database_name , test_id=rand_test_id ).export("C:\\temp\\")
CsvFile(test_case_name=rand_test_case_name, database_name=rand_database_name , test_id=rand_test_id ).export("C:\\temp\\")
HeatMap(test_case_name=rand_test_case_name, database_name=rand_database_name , test_ids=[rand_test_id ]).export("C:\\temp\\")
BarChart(test_case_name=rand_test_case_name, database_name=rand_database_name , test_ids=[rand_test_id ]).export("C:\\temp\\")What bothers me a bit with these changes to the decorator is that this does include a lot of extra parameters which looks a bit ugly and could be better solved by sharing the parameters as a dictionary ;D . What do you think?
As is customary these changes have also created some other bugs and because of that I have already checked most of the framework code base for problems but some probably still exist in the statistical analysis parts of the project. So I will patch up those parts somewhere this week as soon as I have some free time probably this weekend or somewhere next week.
On a side note, as QuickPotato is a testing tool so by default it is packed with analytical capabilities allowing you to compare your dragster runs for regression after code changes and set SLA targets that a dragster run needs to meet to be able to pass your performance test. (Soon I will be adding my own regression algorithm that would make these regression tests more reliable as they now only work through a T-test which works great but can be better.)
These results can then be stored over time by increasing or disabling the data retention policy and can be analyzed after a while to verify if the speed of your product is improving or degrading. (This is information is very powerful and fairly unique to come across in a project.)
Thank you for going through the project I am very happy somebody finds it useful :) cheers from the hilly South of Holland! I have now enabled GitHub discussions but I like issues as they can also double as a bit of documentation for the project :).
Below are the visuals that the example code has generated:
afparsons
This is a nice library. I had some time to play with it this afternoon (we previously exchanged a few messages on Reddit) and I am impressed.
If you don't mind, I am going to use this GitHub issue to maintain a dialogue with you since it appears that you have not enabled GitHub Discussions. If I develop any concrete feature requests or specific questions or bug reports, I will spin those out as separate and discrete GitHub issues.
I intend to use QuickPotato to find slow parts of my Dagster pipelines.
Dagster describes itself in the following way:
Essentially, Dagster is a competitor/spiritual successor to Apache Airflow. It is used as follows:
Solid.Pipeline, which is analogous to an Apache Airflow DAG ("directed acyclic graph").Resource,Mode,Schedules,Sensor,IOManager, etc.; it really is a well thought out system, but for now, those aren't relevant to my questions.Pipelinecan be executed automatically viaSchedule(think cron job), triggered by aSensor("run this pipeline when a new file appears in the S3 bucket"), run via the Dagster CLI, or run through the Dagit UI.I was most attracted to this library because of the intrusive profiling this library offers. The other alternatives I looked at (py-spy, pyflame, etc.) require providing a specific Python script as an argument to the profiler. That approach simply doesn't work when integrating with Dagster! Being able to wrap a function and to know that the function will be profiled whenever it is called (and enabled=True) is really awesome. Congratulations :grinning:
I have provided the context above just so you understand my use cases.
I've watched your NeotysPAC demonstration. Please correct me if I am wrong, but it seems like your initial idea was to build the intrusive profiler, and since then you have subsequently added in the unit test workflows (for example, the
pt.measure_method_performance(...)construct). At this point, it seems like theperformance_breakpointdecorator is not feature complete.Below are my thoughts after experimenting with QuickPotato for about an hour. Please note that I have not yet looked through the codebase extensively and I might hold some misconceptions.
I would like to be able to provide more configuration arguments to
performance_breakpoint. It looks like it inherits theperformance_test.test_case_namefrom scope, but being able to pass this in, or dynamically set it, or generate it using some sort of rule (lambda?) would be great. Likewise with a database name/connection.It took me a while to figure out why QuickPotato wasn't writing to my PostgreSQL server (docker container). I finally saw that the database name is set by the test_case_name. This was not intuitive for me. It would be helpful if you could document how QuickPotato organizes its recordings, i.e. that each test case gets its own database table.
QuickPotato/database/operations.py
return f"{self.URL}/{database_name}"