search

MetPX / sarracenia

https://MetPX.github.io/sarracenia
GNU General Public License v2.0
45 stars 22 forks source link

sleeping beauties #245

Closed michelgrenier closed 2 years ago

michelgrenier commented 5 years ago

The migration of sundew pulls to sr-poll/sr-sarra processes has spread around lots or processes on each node of the sarra cluster. Most of theses, run their task on a single sarra node ... according to the given vip the poll was set to work on, posting locally ... sr_sarra listening locally fetching locally and reannouncing locally .... etc. Only a few poll would spread out their messages onto DDSR and their sr_sarra counter part would share the same queue on all nodes hence spreading the fetching through out the sarra cluster.

For all the sr-poll/sr-sarra that does their thing locally, a cluster of 8 nodes ends up with 7 pairs of sr-poll/sr-sarra that basically are sleeping beauties. It would be nice to specify the vip for both processes (now it is only for the poll that it is specified) and for all the sleeping one make the code wise enough to strictly uses the resources it needs in this vip sleeping state... Ex.: no amqp connections while we are asleep,

The iteration would check for vip, do the work needed if sleeping (for poll that the matching sr_sarra do not delete... update the ls* files) ...and go back to sleep... It the vip is found, the sr_amqp stuff is set not being asleep (consumer and publisher) and consuming reactivates the connection and returns a message Iif available) and the publishing reactivate its connection and publishes... There was some seeds of this idea earlier in the code where sr_amqp had and was testing against self.sleeping... but seeds was not enough not to look like dead code and it was removed.

Because there are a non-negligable amount of sleeping beauties and I have read lately some worries about node resources... I thought it was worth sharing and underlined as a possible feature to implement

petersilva commented 5 years ago

+1 for the name :-)

petersilva commented 5 years ago

might be worth providing some numbers... on the ddsr's (our main broker cluster), we are running something like 100 polls? so there are 800 polling processes connecting to the broker. could save 700 processes and just as many connections to the broker.

petersilva commented 5 years ago

There is something fundamentally odd about sr_poll. Everyone expects that when has_vip is false, the process does nothing, but that isn't how it works. sr_poll was originallly applied to cases where we needed to run on both nodes with and without vip to keep the remote listing (state) uptodate, so on_poll plugin runs whether you have the vip or not. This is very unexpected. it would be nice to figure out an elegant way to have it do the right thing for both use cases. Everyone expects that on_poll will only be run if has_vip is true. Right now, every on_poll needs to check has_vip again... And most of them don't because nobody thinks they need to, so most of the on_poll scripts are wrong. I think it's actually sr_poll that is wrong (violating the https://en.wikipedia.org/wiki/Principle_of_least_astonishment )

one answer ... There should be a setting ....

poll_without_vip True|False?

right now, this is always true. It should default to False, and be explicitly set to true for specific cases. I think that is potentially a large contributor to load on the ddsr's.

another possible way of expressing it:

on_poll_no_vip

so if you want a poll to run regardless of whether you have the vip, you specify the plugin twice ( on_poll and on_poll_no_vip ) by default on_poll would only run when one has the vip. This one has the advantage of being able to specify different processing when one does not have the vip.

This would obviously be a breaking change. haven't done it so far because unsure... perhaps should be separate issue.

petersilva commented 4 years ago

OK so:

time ./sr_poll.py foreground f62

with poll_without_vip=True (how it works now, with no setting available:)

real    0m16.894s
user    0m4.201s
sys 0m0.650s

with poll_without_vip=False

real    0m18.673s
user    0m0.213s
sys 0m0.029s

so the overhead (user cpu) is 20x less with the addition of this setting.

commit: 8d9aa79e873f507fe0b70100b87e8cc8feb6f17b

petersilva commented 4 years ago

To see the difference when the setting is used properly, one should be able to look in the log of a poll before and after the setting is changed and see the consumption of user cpu time should increase much less between heartbeat intervals:

2019-11-10 01:34:31,488 [INFO] hb_memory cpu_times user=27.89 system=3.41 elapse=17965298.78

a grep of such lines should make the difference obvious.

petersilva commented 4 years ago

I don't know how many of the polls can have this setting turned off... but if so the load average could drop down substantially.

petersilva commented 4 years ago

OK, I have what I think is a better idea. We will use the above for now, but this issue will stay open while we work on this idea. the idea is as follows:

This way, any number of participating nodes can keep their caches upto date, and they don´t need to do fancy work to reproduce polls to the destination. This is likely much cheaper than multi-polling. It is also easier to understand than the options listed above.

petersilva commented 4 years ago

This idea is likely much better, but requires major surgery. for example, looks like the python library we´re using is synchronous... so hard to implement prime cache algorithm... probably should do interim release.

petersilva commented 2 years ago

This problem is addressed by #394, which involves a heavy re-factor of poll that addresses this issue thoroughly.

michelgrenier commented 2 years ago

while migrating pxatx, sr_poll was used intensively for all the pooling processes

The idea was that the poll config would define a vip (the target server)

has_vip returning false meant that the target vip *(provided in the config) was on the server hence sr_poll would sleep on all servers but the one corresponding to the provided vip.

it was heavily used to spread polls under the sarra cluster.

So I cant figure out why it started not to work.

MG

Le 22 sept. 2022 à 22:29, Peter Silva @.***> a écrit :

Closed #245 https://github.com/MetPX/sarracenia/issues/245 as completed.

— Reply to this email directly, view it on GitHub https://github.com/MetPX/sarracenia/issues/245#event-7444057009, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZQQL6CVG7PT3372KISYOTV7UIZLANCNFSM4IMHVS7A. You are receiving this because you authored the thread.

petersilva commented 2 years ago

Great to hear from you !

Your first post in the issue describes the old poll well the polls: "update the ls* files) ...and go back to sleep..." The problem you had was that when a poll participant did not have the vip, how does it keep the list of files posted uptodate? The answer you had was that all the servers contact the destination and list the files there... and not having the vip only prevents the inactive polls from posting the results of getting an updated ls.

So that meant if we had a poll with a vip shared by 8 servers, there were 8 servers polling the destination doing "ls" commands to update their lists... and that is like 99% of the work the active poll is doing, so they will not sleep and it is very heavy... especially for the server being polled. not to bad with two servers, but with eight...

in sr3:

the 7 servers without the vip are doing much less work now, and the remote server is only being polled once for an ls, per polling interval. I'm not sure if that is the sleeping beauty problem or not... perhaps the problem you had in mind is different... not sure...

in any event, now you know how the new poll works ;-)

petersilva commented 2 years ago

I just re-read the original problem again... I think you just put the same vip on the sarra as on the poll, and that would fix the original problem. However, I think it is sub-optimal to configure things that way. Instead, I think the polls, should post to the main, central broker (not the node-local one.) and the sarras should subscribe to the global broker also... so they all work (but less hard.) and there are no sleeping beauties. One way or another, this problem can be solved without code change.

michelgrenier commented 2 years ago

Completely agree

Envoyé de mon iPhone

Le 24 sept. 2022 à 23:33, Peter Silva @.***> a écrit :

 I just re-read the original problem again... I think you just put the same vip on the sarra as on the poll, and that would fix the original problem. However, I think it is sub-optimal to configure things that way. Instead, I think the polls, should post to the main, central broker (not the node-local one.) and the sarras should subscribe to the global broker also... so they all work (but less hard.) and there are no sleeping beauties. One way or another, this problem can be solved without code change.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.