Feature: config reference mapping

[ChrisRousey]

This PR implements configuration referencing for top-level resources to provide the users of Datacater with reusable and collaborative configurations

[ChrisRousey]

Thanks for the Feedback on streams. From the conversation this morning, we still have a few pain-points which need to be addressed in this PR.

• We need a different way for matching objects and configs.
• We need more transparancy for Resources (streams, deployments) when they use config options, and a config is updated.
• We need to create the possibility to refence multiple configs for each resource.

[ChrisRousey]

We need a different way for matching objects and configs:

We will go forward by using a k8 style selector for configs and resources. Configs should have a selector label in the metadata node app.datacater.io/name and resources should refence these labels for matching.

[ChrisRousey]

We need more transparancy for Resources (streams, deployments) when they use config options, and a config is updated:

• As of now, we will add a versioning to configs. This means, when updating a config, we will not overwrite it but add a new config object with a newer version of the old config.
• In turn, streams must also reference a version of a config.
  • Users will not be able to change the version, the most recent version will always be used.
  • When updating a stream, the config version will be automatically updated if there is a newer version.
• This allows us to have a reference to a config at the time of a stream creation, not losing any information but also allowing us to retain the original stream object definition without overwriting properties.

*streams are used as an example, this information also applies to other resources

[ChrisRousey]

We need to create the possibility to refence multiple configs for each resource:

• Streams should be able to reference multiple configs.
  • Configs could separately hold connection and config information for example.
• When multiple config versions are found, the most recent will be used and the stream config reference will be updated
• When multiple config references are found, we will try and match them together. If multiple configs contains the same property, we will fail and return the information to the user.
  • it was discussed to add a priority field to configs. I don't see this as an optimal implementation seeing as that we could still have configs with the same prio and containing the same config properties. Using priorities only adds an extra level of maintenance for the users while only moving the problem from one area to another.

*streams are used as an example, this information also applies to other resources

[ChrisRousey]

It is still up for discussion whether to Prioritize stream or config properties.

• We currently Prioritize properties coming from configs, so we overwrite stream fields.

  • We consider configs to be at a higher level, since configs are used to change streams

• It could also be argued, that streams should have a higher priority than configs.

  • Streams are "aware" of having configs, meaning we could expect streams to be the source of truth and configs only considered to be an addon.

*streams are used as an example, this information also applies to other resources

[ChrisRousey]

@flippingbits @HknLof I have added some information from our discusison this morning. Feel free to add information where you see needed and don't hesitate if you see any problems/errors in the proposed approaches. Thanks 🙂

[flippingbits]

* As of now, we will add a versioning to configs. This means, when updating a config, we will not overwrite it but add a new config object with a newer version of the old config.

I feel that versioning is a huge topic and would increase the complexity/scope of this PR a lot. We might need it for other resources than configs as well.

Would it make sense to move the implementation of object versioning to another issue/PR or do we need it to make progress with this PR?

[ChrisRousey]

I feel that versioning is a huge topic and would increase the complexity/scope of this PR a lot. We might need it for other resources than configs as well.

Would it make sense to move the implementation of object versioning to another issue/PR or do we need it to make progress with this PR?

We could leave the config creation/updating as is for now without versioning, always overwrite and for now the information would go lost. Then open a new PR afterwards to implement the versioning. I don't feel like it would be more work to do it that way around. Sounds good to me 🙂

[flippingbits]

We could leave the config creation/updating as is for now without versioning, always overwrite and for now the information would go lost. Then open a new PR afterwards to implement the versioning. I don't feel like it would be more work to do it that way around. Sounds good to me 🙂

To be on the safe side 🤞 , can you please clarify for now the information would go lost?

[ChrisRousey]

To be on the safe side 🤞 , can you please clarify for now the information would go lost?

When working with streams/deployments inside this PR (and after merging until the versioning pr is finished) the exact information regarding which config was used for the creation of the given resource will not be saved anywhere.

[flippingbits]

When working with streams/deployments inside this PR (and after merging until the versioning pr is finished) the exact information regarding which config was used for the creation of the given resource will not be saved anywhere.

Thank you for the clarification! 👍 That is not optimal but it is something that we could - from my point of view- walk with until we have the versioning in place. In the meantime, we could add log statements that help with debugging config/stream integrations. What do you think @ChrisRousey @HknLof?

[ChrisRousey]

@HknLof @flippingbits Regarding the label matching:

• We have a labels field in deployments and streams defined as Map<String, List> to reference more than one item with the same label.
• When creating a config, we automatically create a label in the Metadata mapping the config name to the label 'app.datacater.io/name'
• Config MetaData is saved as a JsonNode in Postgres. To simplify the matching when looking for a referenced config, instead of searching through the metadata, i have mapped it to only look for the referenced name.

Does this sound ok? I also have a question regarding the label naming.

• Config labels are mapped to a name label -> 'app.datacater.io/name'
• should the resources (stream/deployment) label also be name or rather 'app.datacater.io/config'?

[ChrisRousey]

@flippingbits @HknLof @olis1996

Thank you guys for your Feedback regarding the resource/config (selector-)label matching. A quick recap of my takeaways and todos:

• We will move the config referencing in the resources from the labels node and add a new optional node configSelector
  • This node will be on the same level as the resource spec, not inside the individual resource spec-node
• We will no longer expect the resource selectorLabels to contain an array of configs, but rather use the kubernetes way of selector labeling and match key/values with the labels from config.
  • We will no longer "pseudomatch" the referenced configs by there names

Feel free to comment if i missed something or my explanation isn't understandable 🙂

[sonarcloud[bot]]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

90.4% Coverage
0.0% Duplication