Automated update of containers

[stoyan-zoubev]

I'd like to propose an extension of the feature for initial automated deployment of containers (#95).

The initial automated deployment implementation (#103) expects a pre-configured directory holding container descriptor files and these descriptor files are processed only upon the first start of Kanto container management. After the initial start of Kanto CM, any changes to this directory are ignored (even if Kanto CM is restarted).

The suggested extension is that Kanto CM does not process the directory only on its initial start, but on any further restarts of Kanto CM. Upon each start of Kanto CM, it checks the descriptor files in the configured directory and:

• if it finds a new file that descibes a container that does not exist, it will automatically deploy and start a new container. No different behaviour than the initial deployment that we already have, just triggered not only on first run.
• if it finds a descriptor with modified configuration of an existing container, then it will automatically update or recreate the container with the new configuration.
• it is not necessary to preserve the previous contents of directory and deal with deleted container descriptor files. Keep it simple, do not consider container removals if a descriptor file is deleted.
• runtime watching the folder for changes of existing container deployment descriptors or new files is not necessary and may be considered as an optional feature. To keep it simple, runtime watching may be skipped.

Identifying if a descriptor files matches an existing container:

• One option is to identify the containers by their IDs which should be unique. Container IDs are usually auto-generated, but these could also be provided by the user in the descriptor file too. The user will have the responsibility to set unique container IDs.
• Another option is to use container's name for identification. As by definition, the container names are not unique, this puts an extra requirement for the descriptor files.

Optional notes: This extension can be implemented as optional and switched on/off by a Kanto-CM configuration setting.

Tasks:

• eclipse-kanto/container-management#148
• eclipse-kanto/kanto#219

[dimitar-dimitrow]

The proposal is good, I think it will cover more user scenarios than the current initial deployment functionality. There are some aspects that need to be noted:

• It should be decided whether to keep the current initial deployment functionality or not. If the extension described by @stoyan-zoubev , also covers #95. There is no need e.g. to have two descriptor directories - one for initial deploy and one for deploy on Kanto CM start. @mikehaller can you help here?
• As of now removing container when descriptor is deleted is not needed. However, if you want to add such functionality in the future, you should be able to make a difference between container deployed via a file descriptor and a container from an update. Otherwise, you would not know whether to delete a container that is not described in the pre-configured directory.
• Identifying if a descriptor files matches an existing container. I prefer that the container name is used, as it is specified by the user. The container identifier is internally generated. Indeed you can configure it in the descriptor, but this seems a hacky way, that may not work in a future version.
• About the scenario with the modified descriptor, when a container should be re-created. It is a good idea to keep the old container in a stopped state until the new one is started and just then remove it. If the new container fails to start you can default to starting the old one.
• Finding a difference between a persisted container and a container descriptor from a file is not trivial. The persisted container has some runtime configuration, that must not be compared for equality. This must be taken in account when the feature is implemented.

This extension can be implemented as optional and switched on/off by a Kanto-CM configuration setting.

It may even be a mode rather than a true/false flag, e.g. mode: off, initial deploy, on start deploy.

[mikehaller]

at first, i did not see any difference between "initial deployment" and "deployment on restart of service". i was actually a bit puzzled why "initdir" did not actually deploy any additional containers.

implementing this feature is good. i don't see a reason why there need to be two separate directories for deployment descriptors. it can just be one folder.

the idea to make it a "mode" is good, because i think the difference is in the operational mode. on development devices, you may want to have additional deployments done by simply restarting kanto cm.

on a more production device, where you only want the init deployment and all other subsequent container deployments are not done by files but rather via remote apis, you dont need this feature to redeploy on restart. (think of readonly filesystems....)

[stoyan-zoubev]

It should be decided whether to keep the current initial deployment functionality or not. If the extension described by @stoyan-zoubev , also covers https://github.com/eclipse-kanto/container-management/issues/95. There is no need e.g. to have two descriptor directories - one for initial deploy and one for deploy on Kanto CM start. @mikehaller can you help here?

I do not see a reason to have two directories either. Maybe it is better to rename the config key init_dir to something else, as we will change the logic a little bit and it will no longer be init only.

As of now removing container when descriptor is deleted is not needed. However, if you want to add such functionality in the future, you should be able to make a difference between container deployed via a file descriptor and a container from an update. Otherwise, you would not know whether to delete a container that is not described in the pre-configured directory.

Exactly my point. Adding such functionality makes the implementation more complex, so let's skip it for the moment. K3S does not support deleted manifest files either.

About the scenario with the modified descriptor, when a container should be re-created. It is a good idea to keep the old container in a stopped state until the new one is started and just then remove it. If the new container fails to start you can default to starting the old one.

Good point, as this will provide some recovery options. This also implies the requirement that identification is done by container name, as you cannot have two containers (STOPPED, newly CREATED) with the same IDs.

Finding a difference between a persisted container and a container descriptor from a file is not trivial. The persisted container has some runtime configuration, that must not be compared for equality. This must be taken in account when the feature is implemented.

I know. We have already written some source code that does such a comparison and we can contribute it.

It may even be a mode rather than a true/false flag, e.g. mode: off, initial deploy, on start deploy.

I like it.

[stoyan-zoubev]

@dimitar-dimitrow @k-gostev You can assign this ticket to me and I will provide the implementation.

[k-gostev]

Feature and documentation are now implemented, thank you all!