Feature/podaac 5065

[sliu008]

Description

Integration with SMAP_RSS_L2_SSS_V5 collection with concise

Overview of work done

Calculated how much memory a variable will need in shared memory, determine if the variable will fit in shared memory and run single core or multicore concise accordingly.

Overview of verification done

Tested locally with SMAP_RSS_L2_SSS_V5 collection.

Overview of integration done

Explain how this change was integration tested. Provide screenshots or logs if appropriate. An example of this would be a local Harmony deployment.

PR checklist:

• [ ] Linted
• [ ] Updated unit tests
• [ ] Updated changelog
• [ ] Integration testing

See Pull Request Review Checklist for pointers on reviewing this pull request

[joshgarde]

It's an interesting solution, but it seems kinda inefficient. Is there a way to determine whether or not merging should run in single-core vs multi-core before the run? Multi-core merging should be fairly deterministic in terms of how much memory is consumed. It's essentially the sum of each variable's data type size * max dim dimensions + some overhead since there should only be ~1 granule in memory at a time

[joshgarde]

Alternatively, the merge workers are limited by the number of variables that can fit in the queue. Could calculate the maximum number of variables that can fit in the queue for a given amount of free space and that would allow for similar multi-core performance while keeping the memory usage below a certain level

[sliu008]

thanks for suggestions josh, i made modification where i calculate the size that will be needed by variables to determine if it will fit within the shared memory, and from there determine if we can do multicore or single core.