runtime stats for application versions

[drivere]

It would be nice to keep track of average run times and cpu times for each application version in the database.

This information could then be displayed directly in apps.php (and possibly other places). This would give desirable information for both project developers and users. For example, you could compare average run times between 32 bit app versions and 64 bit app versions; or between a CPU version and a GPU version.

To implement this, 4 new fields would need to be added to both the app_version table and the host_app_version table: runtime_n runtime_avg cputime_n cputime_avg These would then be tracked in the validator. Impact to server performance would be negligible.

SETIguy created a branch called runtime_stats, where the above solution was implemented as a proof of concept. I have been using it for several months in a private test server and it works very well. Here is a screenshot showing the runtimes in apps.php:

[RichardHaselgrove]

This concept could be useful where task runtimes are solely or predominantly determined by application version.

But for many projects, task runtime is determined by the data being analysed - sometimes deterministically, sometimes with a strong degree of randomness.

I'm surprised to see you quoting SETI in this context: the SETI project comes into the 'deterministic variablility' class (varies by angle range and search parameters). Has that been taken into account by your algorithm?

[drivere]

That is true, it would only be useful if the average runtimes were consistent for a given batch. It would be a moving average, so it would adjust over time as batches changed. Also note that randomness is not a problem, as long as the mean of the runtimes makes sense. If the underlying distribution of runtimes is multi-modal then the mean would not be a good measure of anything.

I actually was not quoting SETI in my original comment. I mentioned that "SETIguy" has already made the required changes, for no other reason than to give him credit, as he is the one who did the work. However, the fact that he was interested in this tells me that there might be an application for it at SETI, but I will let him speak to that.

[SETIguy]

The version in the runtime_stats branch, it uses the same average as the PFC code, (an exponential running average) For the version of this currently running on the SETI@home Beta, I'm using a running median with roughly the same time constant as the exponential average, but it is far more stable, and does not require outlier exclusion except in the case that you could get several thousand outliers in a row. We could include that in this PR or have a separate PR.

Of course, neither handles the case of a widely bifurcated distribution, but nothing would short of custom code to recognize that case (ala VLAR at S@H).

[davidpanderson]

Another approach (used, for example, in generating the web page comparing GPU models) is to enumerate a few thousand recently-completed and compute their stats. This also lets you show histograms etc.

On Tue, Oct 29, 2019 at 10:44 AM SETIguy notifications@github.com wrote:

The version in the runtime_stats branch, it uses the same average as the PFC code, (an exponential running average) For the version of this currently running on the SETI@home Beta, I'm using a running median with roughly the same time constant as the exponential average, but it is far more stable, and does not require outlier exclusion except in the case that you could get several thousand outliers in a row. We could include that in this PR or have a separate PR.

Of course, neither handles the case of a widely bifurcated distribution, but nothing would short of custom code to recognize that case (ala VLAR at S@H).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/BOINC/boinc/issues/3356?email_source=notifications&email_token=AAHQVAMDBICDM565AKTBZBTQRBY6ZA5CNFSM4JGDOJI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECROLSQ#issuecomment-547546570, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHQVAPRKNZIGDJQS72XBNTQRBY6ZANCNFSM4JGDOJIQ .

[SETIguy]

SETI@home has a separate reason for wanting this that would build this code. The number of processors (or fraction of a processor) used by GPU and multithreaded apps is currently an app constant or workunit create time computation. It's not tracked anywhere. The computation used with GPU apps is always wrong (because of variation in GPU models, driver versions, opencl versions). Having run time and CPU time in the app_version and host_app_version tables will allow n_cpus to be computed on a per host_app_version basis for use in the scheduler, allow proper app choice based upon the app efficiency, and will allow proper credit to be calculated in the validator for multithreaded/multiprocessing apps. I haven't submitted an issue for this (or looked to see if one already exists.

On Tue, Oct 29, 2019 at 3:38 AM RichardHaselgrove notifications@github.com wrote:

This concept could be useful where task runtimes are solely or predominantly determined by application version.

But for many projects, task runtime is determined by the data being analysed - sometimes deterministically, sometimes with a strong degree of randomness.

I'm surprised to see you quoting SETI in this context: the SETI project comes into the 'deterministic variablility' class (varies by angle range and search parameters). Has that been taken into account by your algorithm?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/BOINC/boinc/issues/3356?email_source=notifications&email_token=ACS5ZMVH6JZJAOOA55LI5G3QRAHA3A5CNFSM4JGDOJI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECQAMZA#issuecomment-547358308, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACS5ZMSD4UTFXT56WWDRTNTQRAHA3ANCNFSM4JGDOJIQ .

-- Eric Korpela korpela@ssl.berkeley.edu AST:7731^29u18e3

[drivere]

@davidpanderson Interesting approach. I was actually unaware of the gpu comparison web code (I just found gpu_list.php which I assume is what you are referring to). This is a neat feature, I wish I had known about it sooner.
The one concern I might have is a hit on server performance - it took about a minute for me to load the page, if many users were accessing this page it could bog down the server. I guess it could be run as a periodic task and dumped to an html file to reduce the load.

[RichardHaselgrove]

@SETIguy - try #2949