Preserve containers across plugin reconfigurations and server shutdowns

[JeanRev]

Currently, managed containers are disposed as soon as the corresponding cloud client is disposed (either upon reconfiguration of the plugin, removal/inactivation of the cloud profile or shutdown of the TC server).

This simplifies significantly the container management logic, but may disrupt currently running build jobs (if any), and diminish the reusability of agent containers. Depending on the build setup, preserving containers when a cloud client is disposed can therefore be an interesting use case.

To be further clarified:

• Containers with outdated configurations: If no idle timeout is set, the user will need at some point to destroy them manually (either using the cloud administration interface or Docker CLI).
• Removal of a cloud profile: how to detect it ? What to do with existing containers?
• How is the TeamCity server handling the "resuscitation" of cloud instances (also for currently running builds)?
• ...

[g0t4]

When this happens, sometimes I am having agents get "stuck" as if they still exist but they don't and so I lose licenses on my server until I reboot the server and delete the "stuck" agents that show up under the disconnected tab.

[JeanRev]

When this happens, no count of free license is displayed in the "Unauthorized" agent tab, despite no agents being actually authorized?

If yes, this definitively sounds like issue #6, although the disconnected agents showing up after reboot is something I have not experienced the only time I was able to reproduce the issue.

It might be indeed that this issue has something to do with containers being not gracefully shutdown (they are effectively killed upon restart and reconfiguration of the plugin)... But this is just a wild guess, it would not explain why the server is then reaching a deadlock situation where no build can be processed anymore.

Can you tell me which version of the plugin you are using, as well as your OS/TeamCity version?

[g0t4]

Yes, there was no count on the Unauthorized tab.

2017.1.2 official docker images