Limit number of WUs in WU history

[muziqaz]

Not sure if this is fahclient or web control.

So for now Work Units section is runing smooth, opens without delays, and is a pleasure to use. However with time, when one accumulates a lot of historical data, I believe that experience going to get worse, with loading times and general lag. Current testers using HFM, have accumulated 100s of thousands of WUs as a data points over the years. HFM user experience is not very pleasant to say the least with such amount of data points. I believe Work Units History is being saved in a file, so looking into the future would it be possible to keep that file to certain limit, and once the limit is reached new one is created, or maybe new file is created every year or something. Then fahclient/web control tags the WU being in certain file, and when accessing, it just opens up that specific file, which would minimise the loading times and lag. And obviously, in web control, page would need to have a date related cut off, and not display older than (let's say) 1 year WUs. And older data points could be hidden behind a link to older list (like click here to see WUs from 2023). Filters however would still be able to access old data.

This is just assumtion, that it can be done, or that database will struggle as it expands. Just thinking if it is worth planning into the future and implementing some sort of the solution.

Hope this makes some sort of sense.

[jcoffland]

This is already handled. The client simply limits the number of WU it reports per machine to a max of 500. The could still be an issue if you had say a thousand machines.

[muziqaz]

Is it possible to add a link to older WUs? Or at least filter be aware of older database entries. Though access times are still going to be up significantly when database file bloats from historic data.

[jcoffland]

It's too much data to transfer all the time. It would require a special mechanism to request older WU data from the clients. I don't think it's worth while.

[muziqaz]

Quite often we have users come to forums asking that their hardware cannot finish certain projects in time. One thing we check is how that project was running on our hardware. This way we can compare if those users' hardware is underperforming or we configured constraints the wrong way. This helps a lot of times with projects which have been in public FAH for a while, and server sometimes drops the constraints or researchers wipe them by mistake. Either way older historic data is very useful for these situations as well as with historic averages when calculating base credits. Maintaining older data, even if it segregated, is useful. We will blow through 500 unit limit within several months in internal, especially those who have farms running and are running them 24/7

[jcoffland]

There's always the log files.

[muziqaz]

Yeah, but going through logs takes a lot of time

[muziqaz]

So, I checked out my recent WU history and I'm at 218 entries. This is one Linux client out of potential 3-4. This client only folds on CPU with occasional GPU, during internal testing. So in this short of the time, with just a CPU folding I already am nearly half way towards the limit. Once I plug in other PCs, and start running GPUs on all of them, I will be hitting 500 WU store limit every week. Now, if we take other internal tester who have 20+ PCs folding 24/7, they will be hitting that limit every day. That will be extremely hard sell to them to switch to new client.

[jcoffland]

It's 500 WUs per machine. I believe that is sufficient.

[kbernhagen]

v8.4.4 I am only seeing the current WU for a remote machine. I see WU history only for localhost.

[jcoffland]

Is the remote v8.4.4? If the remote is too old it will not show WU history.

[kbernhagen]

Ah, yes. It is v8.3.18.