search

stackabletech / issues

This repository is only for issues that concern multiple repositories or don't fit into any specific repository
2 stars 0 forks source link

Verify and fix overrides in all operators #548

Closed razvan closed 2 months ago

razvan commented 7 months ago

Issue checklist

Verify (and fix if necessary) that property overrides (env, conf) work as intended in all operators.

There have been reports that this is not the case. Examples:

This issue tracks the state of this feature across all operators:

### Tasks
- [x] Rust tests added (if possible) - similar to https://github.com/stackabletech/trino-operator/pull/556
- [x] Decision regarding property overrides https://github.com/stackabletech/decisions/issues/22
- [x] https://github.com/stackabletech/operator-rs/pull/841
- [x] Bump `stackable-operator` version in operators to `0.74.0`

Release Notes

Trino, Spark, HBase, Airflow: These used to have https://github.com/stackabletech/issues/issues/548[issues] where config and environment variable overrides would not always work as expected, this has now been fixed

lfrancke commented 2 months ago

Can you please include a snippet that we can use for the release notes later?

e.g. (the table is hard to read)

razvan commented 2 months ago

Here is a snippet that demonstrates how environment and configuration overrides work for Apache Zookeeper:

spec:
  servers:
    envOverrides:
      COMMON_VAR: role-value # overridden by role group below
      ROLE_VAR: role-value   # only defined here at role level
    configOverrides:
      zoo.cfg:
        prop.common: role
        prop.role: role
    roleGroups:
      primary:
        replicas: 2
        config:
          myidOffset: 10
        envOverrides:
          COMMON_VAR: group-value # overrides role value
          GROUP_VAR: group-value # only defined here at group level
        configOverrides:
          zoo.cfg:
            prop.common: group
            prop.group: group

This example is fictional because the properties written to the zoo.cfg file are not used by Zookeeper in any way.

What ends up being written in the Pod's environment are the variables:

COMMON_VAR=group-value
ROLE_VAR=role-value
GROUP_VAR=group-value

Similarly, the zoo.cfgfile of the primary group will contain:

prop.common=group
prop.role=role
prop.group=group

Of course, these are not the only settings that the operator will configure.

lfrancke commented 2 months ago

Thank you, that is useful but not really all that usually goes into the release notes. We'd need a sentence on what has been changed/fixed and this could then go in there in addition to that.

razvan commented 1 month ago

This fine ?

  • Added tests for config and environment overrides in all operators.
  • Fixed config and overrides in operators where it wasn't working as expected (ex. Trino, Spark, etc).
lfrancke commented 1 month ago

The tests are not really release notes material. If I were to read release notes I'd be interested in knowing which operators were actually fixed. It's a bit hard to see from the list above, can you give me the list of operators that had bugs before but are fixed in 24.11?

razvan commented 1 month ago

It's a bit hard to see from the list above, can you give me the list of operators that had bugs before but are fixed in 24.11?

Yes, it's not easy for me either. Luckily we used conventional commit messages so I'm pretty sure it was: