search

markrmiller / solr

An experimental branch of Apache Solr, serving as a reference for various performance, scale, and stability improvements.
https://solr.apache.org/
Apache License 2.0
1 stars 1 forks source link

Overview of ZooKeeper Issues and Related #2

Closed markrmiller closed 3 years ago

markrmiller commented 3 years ago

A kind of broad initial overview of ZooKeeper challenges and issues and the many things that tie into that. More detail and various solutions and improvements to come in sub issues.

markrmiller commented 3 years ago

A handful of Videos to overview and outline will be pipelining into this issue today. Detail and break out and text will show up in the various sub and related issues that are in this category.

https://user-images.githubusercontent.com/448788/114565797-403ff600-9c37-11eb-823e-13eea6be3fb4.mp4

markrmiller commented 3 years ago

Starting to dig into Leader Election...

https://user-images.githubusercontent.com/448788/114571286-294fd280-9c3c-11eb-8d9b-21bc543f2db9.mp4

markrmiller commented 3 years ago

A few more coming on issue overview and then we will do one focused more on how things are supposed to be done.

markrmiller commented 3 years ago

https://user-images.githubusercontent.com/448788/114584224-0cb99780-9c48-11eb-82dc-74c924bac2fb.mp4

markrmiller commented 3 years ago

https://user-images.githubusercontent.com/448788/114595842-717aef00-9c54-11eb-8556-6d9b30aad405.mp4

murblanc commented 3 years ago

I do sympathize with your examples of unneeded complexity and misuse of ZK. That's the same direction I'm trying to steer my work (that you might not have seen if your snapshot is almost a year old). In current main I've already merged (a while ago) doing cluster state updates from any thread (i.e. from the Collection API threads essentially) without going through Overseer, and now I have a PR out for doing the Collection API calls from any thread (i.e. from the CollectionsHandler receiving the user requests).

These two aspects put together, then there's no longer a need for Overseer, and the bottom part of your class diagram goes away and/or is collapsed into the upper part.

And then (next items I plan to work on), stop watching all the state in the world all the time. I believe for most of our cluster management needs (and API handling), we do not need watches (we do need them when we send a request to another node to find out when that node is done, but this is pretty rare occurrence of creating a replica etc).

markrmiller commented 3 years ago

This largely wraps the quick spin through ZK - skips lots of stuff and other things I've tried and that you can do, but IMO ends pretty positive in terms of potential. The scale and efficiency and problem areas - so much of that can just be excised.

https://user-images.githubusercontent.com/448788/114607774-8b233300-9c62-11eb-9c0b-d95e78848933.mp4

markrmiller commented 3 years ago

Yes, I have seen some of this work! I don't often know who is responsible for what, but I have seen a variety of things.

In fact, you can go back and find solid and relevant contributions and various positive directions all over. Some great stuff.

As I look through and see a lot of that, I often find the resulting benefits sadly underwhelming. The problem being the ofen narrow focus and huge drag against limited changes and the often overwhelming influence of other interactions and items.

That's why some demonstrations will be very useful. There is so much weight against positive change.

And it tends to be getting worse as Jetty and libraries and Java move towards a world that makes sense for them, and increases our issues.

Like Jetty - they are going after async and http2 in a way thaat pushes our system too really abuse their elegant and wonderful solutions around threads and async and blocking. We have to start riding their wave before all these guys sink us with their assumption that we are not going abuse their unlimited growing queues and desire for good actor behavior with their thread pool control. As Jetty gets better, we lose out on not working to advantage with them, and also eat more pain. And this happens more and more in our bocking, locking, waiting world, as everyone else moves to async and reactive and good thread utilization and behavior.

murblanc commented 3 years ago

I think I got the idea in your last video above: one persistent recursive watch getting all change notifications. I didn't get though how the notifications get to the other clients, running on other nodes, that want them (assuming we're trying to replace existing architecture, like all nodes having a replica for a given shard might be interested in changes to the shard). Do we maintain a parallel id hierarchy and the single watcher notifies through synthetic changes to that hierarchy (that the others are... watching?) or a different distribution mechanism that is not ZK based? Or something else?

markrmiller commented 3 years ago

I just hit wrong buttons, but corrected reply above.

markrmiller commented 3 years ago

I also should state, in any individual topic, improvement, feature, way of doing things - any smart devs, random devs, they will validly know better - be able to correct and improve on whatever I say or claim. I've got to spread information, there's a line that can look distorted from the other side.

Because the big issue to confront is the huge surface area and connection and interaction of things. So I've got to lay things out in a way that's possible - have to communicate, have to have a flow of information. But what we are aiming towards is not any of these early or individual things. That's why it's crucial to take a little time as I get it all out. Incorrect, insufficient, distorted, misremembered, misspoken or whatever. What we want to get to is not any of my lecture points - but the large survey results. The lay of the land in terms of how many things are working against good work. There is plenty of good work. Of things that I look at and say, yeah, nice move, likely would not have thought of it or done it. Some great people have come through. This is the tip of the iceberg that I want to start getting conveyed so that instead of me laying out all the troubles and results and barriers and crap I've run into and found blah blah, we can move to sensible collaboration and attack plans.