From the jobs I've observed over the last year, GC is the most common way back-pressure gets applied. When the system is clearing backed up queues nodes keep on reading data until some limit is hit. The limit that gets hit might be the maxSpoutPending limit in the good case but it's difficult to find the right value for this setting. More often, the system runs into GC before this or any other limit comes into play. GC reduces cpu time available to read tuples and thus applies back pressure. Even though this mechanism works it has a lot of issues:
High frequency of ParNew means that more data ends up in old generation leading to CMS which is more expensive computationally than ParNew. Both FinalFlatMap and Summer nodes need to keep around data to get good crushdown so both are prone to this issue when ParNew frequency increases. The time spent on CMS is totally extra work. So the system in this state is less efficient than before.
In a bad case the system might go into GC death spiral due to the above
More commonly, system performance will suffer and make the recovery slow.
a. If loss of efficiency due to GC isn't too bad than rate of processing tuples will be higher than rate of incoming tuples and the system will ultimately recover, although after a longer period of time
b. If loss of efficiency is large enough that rate of processing tuples is lower than rate of incoming tuples then the system will never recover. At this point restarting the node or adding more resources is the only option. The possibility of running into such issues will prompt the owners of these jobs to add extra resources permanently. Resource requirement is governed by the worst case performance.
Ideal situation is for the system to stop consuming tuples before running into bad GC. I propose the following mechanism:
User defines a setting, say maxTimeSpentDoingGCPercent, which defines the max % time spent on GC that user thinks is ok (we may be able to choose a good default value e.g. 10%, which hopefully works for most cases so users don't have to worry about this setting in the common case). A component in summingbird keeps track of % time spent doing GC in past x seconds and if that time is more than maxTimeSpentDoingGCPercent then a proportional amount of back pressure is applied directly on the Slave thread of storm/heron.
This is a broad idea at this point and I want to request discussion on this to ratify/refine/work out the details. I think this mechanism will help a large number of summingbird jobs and would make summingbird more efficient.
From the jobs I've observed over the last year, GC is the most common way back-pressure gets applied. When the system is clearing backed up queues nodes keep on reading data until some limit is hit. The limit that gets hit might be the maxSpoutPending limit in the good case but it's difficult to find the right value for this setting. More often, the system runs into GC before this or any other limit comes into play. GC reduces cpu time available to read tuples and thus applies back pressure. Even though this mechanism works it has a lot of issues:
Ideal situation is for the system to stop consuming tuples before running into bad GC. I propose the following mechanism: User defines a setting, say maxTimeSpentDoingGCPercent, which defines the max % time spent on GC that user thinks is ok (we may be able to choose a good default value e.g. 10%, which hopefully works for most cases so users don't have to worry about this setting in the common case). A component in summingbird keeps track of % time spent doing GC in past x seconds and if that time is more than maxTimeSpentDoingGCPercent then a proportional amount of back pressure is applied directly on the Slave thread of storm/heron.
This is a broad idea at this point and I want to request discussion on this to ratify/refine/work out the details. I think this mechanism will help a large number of summingbird jobs and would make summingbird more efficient.