Closed sliu008 closed 1 year ago
It's an interesting solution, but it seems kinda inefficient. Is there a way to determine whether or not merging should run in single-core vs multi-core before the run? Multi-core merging should be fairly deterministic in terms of how much memory is consumed. It's essentially the sum of each variable's data type size * max dim dimensions
+ some overhead since there should only be ~1 granule in memory at a time
Alternatively, the merge workers are limited by the number of variables that can fit in the queue. Could calculate the maximum number of variables that can fit in the queue for a given amount of free space and that would allow for similar multi-core performance while keeping the memory usage below a certain level
thanks for suggestions josh, i made modification where i calculate the size that will be needed by variables to determine if it will fit within the shared memory, and from there determine if we can do multicore or single core.
Description
Integration with SMAP_RSS_L2_SSS_V5 collection with concise
Overview of work done
Calculated how much memory a variable will need in shared memory, determine if the variable will fit in shared memory and run single core or multicore concise accordingly.
Overview of verification done
Tested locally with SMAP_RSS_L2_SSS_V5 collection.
Overview of integration done
Explain how this change was integration tested. Provide screenshots or logs if appropriate. An example of this would be a local Harmony deployment.
PR checklist:
See Pull Request Review Checklist for pointers on reviewing this pull request