For data selection cases where 100's of chunks need to be quieted, it will likely be more efficient to invoke a lamda function for each chunk rather than dispatching to the DN nodes. AWS supports up to 1000-way parallelism for Lambda, which will provide better scaling than most server configurations.
jreadey
commented
4 years ago
For data selection cases where 100's of chunks need to be quieted, it will likely be more efficient to invoke a lamda function for each chunk rather than dispatching to the DN nodes. AWS supports up to 1000-way parallelism for Lambda, which will provide better scaling than most server configurations.