Old assets are not removed on cluster config changes

[johananl]

Scenario:

• User runs lokoctl cluster apply with disable_self_hosted_kubelet = false.
• Self-hosted kubelet chart is written to disk.
• User sets disable_self_hosted_kubelet to true and re-runs the command.
• Self-hosted kubelet chart is still present in the assets directory.

The result is that self-hosted kubelet is still applied to the cluster despite being disabled in the config.

Need to verify this but I have a strong suspicion this bug exists. Also, it applies to basically anything that is conditionally written to disk - another example is backend.tf.

[invidian]

I think it should be the other way around, first disable_self_hosted_kubelet = false, then disable_self_hosted_kubelet = true. And this not only leaves the configuration files in the assets directory, but also leaves the DaemonSet running. If this bug report is only about this specific case, I suggest we close it as a duplicate of #658. But if we look at it more broadly (that old files are not being removed from the assets directory, which is 100% the case right now), then this is something to definitely look at.

Perhaps when we fix #24 and #608, we could make asset directory disposable by default, and users which wants to carry some custom patches, should be informed, that they need to make sure that old files are being removed between the releases.