search

itsjohncs / phial

A Python microframework for building static sites. Will not be completed.
BSD 2-Clause "Simplified" License
4 stars 0 forks source link

How can we automatically test performance? #56

Open itsjohncs opened 9 years ago

itsjohncs commented 9 years ago

I think I'm pretty far off from tackling any performance problems, but it's unclear to me how to go making tests for it. Creating a stable environment for performance tests seems the most difficult, especially where Travis is concerned...

It seems especially important to set up automatic tests for this because the vast majority of users including myself will not require Phial to be performant, so any performance features I add will rarely be manually tested.

itsjohncs commented 9 years ago

Something I thought of is trying to count the number of bytecode instructions that have been executed. This might be possible based on a stack overflow post I read. I think Travis might even allow C extensions, would would allow me to do this with Travis. I can also just use deterministic profiling and then try to do some statistical analysis on the results.

itsjohncs commented 9 years ago

IO would be tricky to keep track of above... I think I'd want to measure the number of bytes read and written...

itsjohncs commented 9 years ago

This might not be very fruitful. Just running performance tests and graphing the results will probably give me sufficiently useful results. I'll be able to answer the question "am I getting faster or slower". And if I want to make performance related improvements I'll have a way to measure success at a larger scale than "I improved this function's performance".

I think this will be best to approach when I'm actually making performance related features.