airbreather
commented
7 years ago This is still a concern even though 7-zip processes are spawned with lower priority than normal.
Open airbreather opened 7 years ago
airbreather
commented
7 years ago This is still a concern even though 7-zip processes are spawned with lower priority than normal.
Use Case
Skyrim is a very popular game with a comparably modest set of minimum hardware requirements. Through mods, however, players can crank up the hardware requirements for their own setups somewhat arbitrarily high. With many scripting mods installed, and (I expect) not much parallelism in the way Papyrus scripts are invoked, users will likely see significant benefits from overclocking their CPU, more so than likely any other hardware change.
Speaking from personal experience, this mix of "very popular" and "significant benefits from overclocking their CPU" probably means that some enthusiastic-but-inexperienced individuals will not thoroughly test their cooling systems and/or general CPU durability when all cores are active for a significant amount of time like, say, when a hundred 7z.exe processes spin up concurrently to use a handful of gigabytes to produce a much bigger handful of gigabytes.
I'm concerned that someone's going to try this out, and it's not going to work for them because of their overclocking setup. Assuming it doesn't fry their computer the first time they try (or the first few times), these individuals will probably still benefit from being able to run just ONE process at a time.
Proposed Solution
There are two high-level things about this solution that leverage concurrency: MD5 checking and actually running the <Tasks> themselves.
Most likely, the rate of MD5 checking is going to be limited by the speed of file I/O, though StepperUpper will gladly become CPU-bound here if the data stored in an extremely fast location, or if it's significantly memory-cached. Still, I don't think there's much of a point in doing too much here.
It's really the <Tasks> that I'm most concerned about. Well, it just so happens that all of the tasks from there are dispatched through a single method. It would be easy enough to just have that method optionally asynchronously wait for and hold a SemaphoreSlim through the duration of the task it actually runs, and release when the task is completed. In most cases, we don't have to worry about deadlocking because this is only called after all WaitFor tasks, if any, have also completed, and so there's almost never a multiple-lock situation.
The thing to really worry about is if a <Clean> task has individual plugins that want to wait for other tasks. This could easily lead to deadlocking if the dependent task is not allowed to start because too many cleaning tasks are waiting on them. I think this is a reason to just take "WaitFor" off of plugins themselves and just put it back on the <Clean>. It's a hypothetical performance hit in some places, but only for those plugins with UDRs. I think it'll be fine.