COSTOR-894: fix notification failures during cluster start/stop

Seagate / halon

High availability solution

Apache License 2.0

1 stars 0 forks source link

COSTOR-894: fix notification failures during cluster start/stop #1578

Closed andriytk closed 5 years ago

andriytk commented 5 years ago

The default 2 seconds delay for the notifications aggregation introduced at commit 20a96f16 was too small to capture all the notifications, so the CEP engine suffered again from overloading with processing too many notification Acks. It was noticed on as large as 7 SSUs configurations already.

Now we improved the aggregation algorithm by introducing an additional idle timer for the new notifications to be added into the aggregation (we reused old parameter name for this --notification-aggr-delay, 5 seconds by default). And also by increasing the existing maximum time for the aggregation buildup from 2 to 20 seconds (we renamed old parameter name for this to --notification-aggr-max-delay). Thus we don't wait too long when there are few notifications to be sent, from one side. And, from another side, we wait long enough to accumulate all the notifications when there are many of them.

andriytk commented 5 years ago

On small configurations, it might be slower, yes. (But it will be faster and workable on bigger ones for sure.) That's one of the reasons why I want to keep the timeout in ms - to be able to specify less than a second for small configs.

vvv commented 5 years ago

merged

vvv commented 5 years ago

added 7 commits

6342dd3e...2496354b - 5 commits from branch master
b728a66f - COSTOR-894: fix notification failures during cluster start/stop
b4f25f91 - st/common: wait_for_fid_state() timeout increased to 1.5 mins

Compare with previous version

vvv commented 5 years ago

Does this mean that when this patch is merged Halon will need 18/12 = 1.5x more time to start m0d and mark it as online?

vvv commented 5 years ago

[optional] We could have used seconds instead. “5000 ms” is a noisier version of “5 sec”.

andriytk commented 5 years ago

added 1 commit

6342dd3e - st/common: wait_for_fid_state() timeout increased to 1.5 mins

Compare with previous version

andriytk commented 5 years ago

unmarked as a Work In Progress

andriytk commented 5 years ago

added 1 commit

5045d792 - COSTOR-894: fix notification failures during cluster start/stop

Compare with previous version

andriytk commented 5 years ago

changed the description

andriytk commented 5 years ago

changed the description

andriytk commented 5 years ago

added 5 commits

7e38ecf3...83ae77fe - 4 commits from branch master
e29814b5 - COSTOR-894: fix notification failures during cluster start/stop

Compare with previous version

andriytk commented 5 years ago

changed the description

andriytk commented 5 years ago

marked as a Work In Progress

andriytk commented 5 years ago

assigned to @vvv