The changes here collapse multiple loops into a single traversal. This is an experimental change intended for review and discussion. These changes are incomplete and should be discussed before continuing.
Technical Overview
There are 4 requirements to the traversal of currentAssignment affected:
Group partitions (Set<String>) by taskId: newPartitionAssignmentMap.
Determine which tasks had a change in assignment count: tasksWithChangedPartition.
Find the total throughput for each task: taskThroughputMap.
Identify which tasks were below the max partition count: tasks.
The prototype proposed performs all 4 in a single traversal. The current implementation does 3. The first for 1 and 2. While 3 and 4 each have their own traversal. It isn't clear if the use of parallel() will affect performance.
ehoner
commented
1 year ago
The changes here collapse multiple loops into a single traversal. This is an experimental change intended for review and discussion. These changes are incomplete and should be discussed before continuing.
Technical Overview
There are 4 requirements to the traversal of
currentAssignmentaffected:Set<String>) bytaskId:newPartitionAssignmentMap.tasksWithChangedPartition.taskThroughputMap.tasks.The prototype proposed performs all 4 in a single traversal. The current implementation does 3. The first for 1 and 2. While 3 and 4 each have their own traversal. It isn't clear if the use of
parallel()will affect performance.