RSalman
commented
4 years ago Another thing to consider is the time to notify threads. Currently, this is dependent on the size of the pool as it's done with notify_all. This can be costly if we want to use a small subset of the pool. Waking up 2 thread vs. 64 threads is the same, the time is expected to be constant This can be seen by measuring time to notify various thread counts:
------ Thread Count: 64 -> Time to Notify: 212 ------
------ Thread Count: 63 -> Time to Notify: 189 ------
------ Thread Count: 62 -> Time to Notify: 189 ------
------ Thread Count: 61 -> Time to Notify: 211 ------
------ Thread Count: 60 -> Time to Notify: 242 ------
------ Thread Count: 59 -> Time to Notify: 261 ------
------ Thread Count: 58 -> Time to Notify: 269 ------
------ Thread Count: 57 -> Time to Notify: 249 ------
------ Thread Count: 56 -> Time to Notify: 253 ------
...
------ Thread Count: 10 -> Time to Notify: 252 ------
------ Thread Count: 9 -> Time to Notify: 253 ------
------ Thread Count: 8 -> Time to Notify: 255 ------
------ Thread Count: 7 -> Time to Notify: 254 ------
------ Thread Count: 6 -> Time to Notify: 254 ------
------ Thread Count: 5 -> Time to Notify: 258 ------
------ Thread Count: 4 -> Time to Notify: 253 ------
------ Thread Count: 3 -> Time to Notify: 249 ------
------ Thread Count: 2 -> Time to Notify: 254 ------
------ Thread Count: 1 -> Time to Notify: 260 ------If we do a notify for each thread:
for (uintptr_t threads = 0 ; threads < count; threads++ ) {
omrthread_monitor_notify(_slaveThreadMutex);
}This results in the following:
------ Thread Count: 60 -> Time to Notify: 643 ------
------ Thread Count: 59 -> Time to Notify: 653 ------
------ Thread Count: 58 -> Time to Notify: 643 ------
------ Thread Count: 57 -> Time to Notify: 611 ------
------ Thread Count: 56 -> Time to Notify: 576 ------
------ Thread Count: 55 -> Time to Notify: 548 ------
------ Thread Count: 54 -> Time to Notify: 672 ------
------ Thread Count: 53 -> Time to Notify: 804 ------
------ Thread Count: 52 -> Time to Notify: 578 ------
...
------ Thread Count: 14 -> Time to Notify: 134 ------
------ Thread Count: 13 -> Time to Notify: 128 ------
------ Thread Count: 12 -> Time to Notify: 133 ------
------ Thread Count: 11 -> Time to Notify: 105 ------
------ Thread Count: 10 -> Time to Notify: 100 ------
------ Thread Count: 9 -> Time to Notify: 77 ------
------ Thread Count: 8 -> Time to Notify: 69 ------
------ Thread Count: 7 -> Time to Notify: 57 ------
------ Thread Count: 6 -> Time to Notify: 46 ------
------ Thread Count: 5 -> Time to Notify: 36 ------
------ Thread Count: 4 -> Time to Notify: 24 ------
------ Thread Count: 3 -> Time to Notify: 15 ------
------ Thread Count: 2 -> Time to Notify: 8 ------
------ Thread Count: 1 -> Time to Notify: 0 ------This approach is better when waking up threads < ~20, but its’s worse off with more threads. We see notification times up-to 500-650us which decrease as thread count decreases. We obtain similar times of notify_all when threads are cut in half (~32 threads of 64). But the clear advantage is when threads are < ~20, we see times <100us.
amicic
The current implementation of thread dispatching for parallel GC tasks is inefficient.
Threads are pre-selected for the task. This is not optimal since an unselected thread may become available earlier than a selected thread (may take some more time, depends on system when thread is made available, or order in which thread reacquires monitor)
From the pool of slave threads, all threads are woken up regardless if they're selected or not to perform the task, as noted in the code:
https://github.com/eclipse/omr/blob/9e5dfc852daa61a2347183b387ca75e3a637d998/gc/base/ParallelDispatcher.cpp#L381-L386
The current implementation simply does a
notify_alland the count is irrelevant in the actual wake up notification: https://github.com/eclipse/omr/blob/9e5dfc852daa61a2347183b387ca75e3a637d998/gc/base/ParallelDispatcher.cpp#L387-L391Given that threads are pre-selected and that all threads are notified, we can have contention between selected and unselected threads. All threads wait on the same mutex, they need to reacquire the mutex before continuing. This means threads proceed in a serial fashion after being notified. The mutex is made available when the owning thread either
For instance, if a single thread needs to be used for the task, it can be the last thread to acquire the mutex (or it may not be available until some later time). In this case it must wait for all other threads in the pool to sleep again before continuing.
These effects can be seen here:
Here we can see that it takes around ~1.5ms for 1 selected thread (out of 64 threads) to startup. Ideally, the selected thread should start shortly after notification.
An ideal scenario looks like this:
Overall, the time it takes to start a task can be dependent on: