jgallagher
commented
9 months ago I think this is not a bug, and instead a side effect of effectively telling the planner that you want 4 Nexus instances on what is effectively a 3-node system. The ordering here is:
- RSS kicked off with 3 nodes.
- RSS generates a plan that allocates 3 Nexus instances, one to each node.
- RSS completes; all 3 sleds have 1 Nexus instance.
- A fourth sled is added to the cluster at the trust quorum level. It has no zones (yet).
- We generate a base blueprint from the latest inventory collection. (This could swap places with the previous step with no effect, I think.)
- We ask the planner to generate a new blueprint.
Inputs to the planner in step 6 include:
- There are four sleds.
- The fourth sled has no zones.
- There are three Nexus instances, one on each of the first three sleds.
- The policy says there should be 4 Nexus instances.
What should the planner do given this set of inputs? The current implementation (which I think is correct based on conversations we've had so far) is that when generating a plan, if a sled doesn't yet have an NTP zone, the planner should give it an NTP zone and nothing else. So the planner does that, but then sees that the policy says there should be 4 Nexus instances. Therefore, it allocates an additional Nexus instance to one of the eligible sleds, which is one of the three that already has one Nexus instance.
If this were a larger system with at least one "eligible for service zones" sled that did not already have a Nexus, the planner would've put the Nexus there instead of on a sled that already has a Nexus. In this system there are no such sleds, so it's forced to double up. I don't think it would be correct for the planner to attempt to give a new sled an NTP zone and a Nexus zone in the same plan generation, right?
andrewjstone
davepacheco
I created a 3 node testbed off the code on main, and then added node
g2. I ran the planner enough times to deploy ntp and crucible zones to sledg2. I then ran the planner to try to get it to place the nexus zone ong2. Prior to testbed launch I had previously modified the code as instructed by @jgallagher so that there would be a 4th nexus instance to place by the planner.I noticed the zone wasn't being placed on sled
g2when looking at blueprints and that the generation number for that sled was stuck at3with only crucible and ntp zones. I then ranomdb nexus blueprints regeneratea few times to see if the nexus zone would get placed on g2 after some time. I then realized that there was a second nexus running on sledg3. I then went and ran some commands backtracking through the blueprints to see if the second nexus was actually placed after the first blueprint was constructed and it looks like it was. So it appears we have a bug in the planner. The omdb commands I used to diagnose this after a few calls toregenerateare below.