Operators do not reconcile on changes to shared objects (no managed-by label)

[maltesander]

The operators reconcile objects that are watched (ConfigMap, StatefulSet etc.) and labeled managed-by=hive.stackable.... Many operators do read (resolve) additional data from AuthenticationClasses or ConfigMaps.

If e.g. vector logging is enabled and a vector aggregator discovery ConfigMap is deployed (not buy any operator), containing the aggregator address, changes to that discovery ConfigMap are not picked up by the operators.

Without any other event, the operator will not reconcile and distribute the new aggregator address to the products. This leads the vector containers to fail (if the aggregator address changed) to push their data to the aggregator.

Fixes:

• Restart the operator to start the reconcile loop
• Reconcile every once in a while (polling, not just events)
• Add another label indicating operators to listen to changes (if the object e.g. ConfigMap is watched anyways)

Known affected operators (incomplete, probably all):

• hive
• trino
• superset

[nightkr]

IMO, the fix would be to add a watch like (sorry, untested pseudocode):

controller.watches(
  client.get_api::<PartialObjectMeta<ConfigMap>>(), 
  watcher::Config::default(),
  |cm| stacklet_store.state().into_iter()
    .filter(|stacklet| stacklet.refers_to_cm(&cm))
    .map(|stacklet| stacklet.object_ref(&())
  )

Where Stacklet::refers_to_cm() matches the CM against the vector discovery config and any other fields that refer to CMs (and are read by the operator, not just forwarded into the pod config).

This would end up similar to how the restart controller works.

Alternatively, we could try to move the read into the product container so that it's handled by the existing restart controller.