profiling my plan

[francoisverbeek]

Hi guys, thanks again for open-sourcing testplan. Having written a testplan that starts a few processes and use the TCP server etc, I notice some of my tests are slow. I tried to use the builtin profiler in intellij or cProfile to find out about where and what and I notice that the testplan is actually executed as a separate thread, which escapes the default profilers. For profiling purposes, I'd like to be able to escape that and have a mode where I can run single-threaded. I couldn't find anything in the doc in that respect. Do you have a canonical way to do that? Would you be interested by a pull request?

[francoisverbeek]

This seems to work for me, will try on a bigger test plan and confirm.

[ryan-collingham]

Hi Francois,

Thanks for raising. Yes I agree that it would be useful to be able to profile a testplan. As you note, currently the execution is always run in a separate thread - this is done so that the main thread can quickly jump in and handle any signal interrupts, there are a number of cases where the python runtime will not immediately handle a signal (e.g. while join()-ing another thread). But there are downsides as you note for profiling (also I think many common debuggers do not handle threads well).

Fundamentally though, Testplan is (now) a multithreaded application so I think the most ideal solution would be to find a profiler that can cope with threads, rather than providing a special single-threaded mode for profiling.

Do you think it would be useful to add profiling options at the MultiTest scope, or would a testplan-wide profile be more useful?

[francoisverbeek]

Hi Ryan,

I've briefly looked at threading-aware profilers and was not overly impressed by what I saw, to be fair. cProfile also has the advantage of being always available, and to be well known and well tooled. My aim would be more to profile user test code than testplan itself, although I think including things like drivers start and stop would definitely be useful. Experimenting briefly in my fork, wrapping Runnable._run in a simple context manager yields promising results, but I haven't had occasion to test my approach properly yet. Also interesting to consider is to which extend the profile stats could be considered as part of the test report.

To push the idea further, a more generic way to include profilers for the application under test and integrate their results in the reports could also be extremely valuable, for instance valgrind, callgrind, strace, dtrace etc.

that might end up being a lot of work though!

[ryan-collingham]

Sounds like it would be useful to add a profile option at the MultiTest scope then, to be able to profile user testsuites (or possibly even indivudal testcases). We could then include the profile output in the report.

I'll discuss with the team, and if they agree we can add to the list. It won't be a priority right now since we need to add some other key features (e.g. DB uploading) first, but maybe after that is done.

[francoisverbeek]

Sounds great 👍 Thank you!