Makes large parts of the media export process multi-threaded.
We previously had a multi-threaded solution in the converter that was deactivated when the new converter was. This implementation doesn't reintroduce this behavior and instead opts for a simpler solution using Python's multiprocessing module with Pool, Manager and Queue. Low-level routines were not needed and the solution now works (with adjustments) for all media files.
The method used is based on the main thread loading the file data and feeding it as a producer to the worker threads. The worker threads convert the data and write the result to file.
The speedup is mostly negligable for the earlier game versions (until SWGB) where the files are small and fast to convert, so the main process cannot provide data fast enough to fully use all available resources. The time gains there are mostly eaten up by the overhead of the thread pool. However, there are signifiant speedups for DE1 and DE2 where export times are cut down to 25% of the previous runtime. Here are some before/after export times for comparison:
Makes large parts of the media export process multi-threaded.
We previously had a multi-threaded solution in the converter that was deactivated when the new converter was. This implementation doesn't reintroduce this behavior and instead opts for a simpler solution using Python's
multiprocessingmodule withPool,ManagerandQueue. Low-level routines were not needed and the solution now works (with adjustments) for all media files.The method used is based on the main thread loading the file data and feeding it as a producer to the worker threads. The worker threads convert the data and write the result to file.
The speedup is mostly negligable for the earlier game versions (until SWGB) where the files are small and fast to convert, so the main process cannot provide data fast enough to fully use all available resources. The time gains there are mostly eaten up by the overhead of the thread pool. However, there are signifiant speedups for DE1 and DE2 where export times are cut down to 25% of the previous runtime. Here are some before/after export times for comparison:
AoE1 (1997)
AoE2 (1999)
SWGB
HD Edition
DE1
DE2