BalanceReplicas called even when PopulatePodsOnScaleUp and VacatePodsOnScaleDown are false

[sigram]

When both of these options are set to false (in an environment where replica management is handled externally, not using Solr placement plugins) the natural expectation is that the Solr Operator will not initiate any replica movements on scale-up / scale-down or rolling updates.

However, this is not the case in 0.8.1. When Solr Operator discovers that this API is available it will call /api/cluster/replicas/balance in quite a few scenarios, causing replica movements even when both of the above options are disabled.

I can work around this by implementing a custom PlacementPluginFactory but I think that this behavior should be optional, with an option to turn it on when desired. And the situations when BalanceReplicas is called should be better documented (listing cases when and how often this API is called). I think that the comments in BalanceReplicasForCluster about re-tries on errors and async operations should be included in the main documentation, because they can result in potentially resource-intensive operations.

[HoustonPutman]

It definitely calls it when doing a rolling restart of an ephemeral collection, but that's because we are moving replicas around anyways. We can definitely document that better, but I'm not sure I'd want to remove that feature.

Can you describe the other scenarios in which its called?

[sigram]

If I'm reading it right, this section solr_cluster_ops_util.determineScaleClusterOpLockIfNecessary:200 always calls Balancing if a scaleDown fails for some reason, even if vacate/populate is false, regardless of the storage options.

[HoustonPutman]

Ahhhh ok then yeah certainly that's a bug. Should be an easy enough fix.