Open AmitBhatnagar24 opened 7 years ago
Hmm, these are complicated questions.
Not sure why the standard images get stuck, they seem to handle termination in a similar way that I do, although there are differences so who can tell...
Regarding the handling of SIGTERM
etc. In my startup script I do:
first, trap the signals: trap clean_up HUP INT QUIT TERM
. See https://github.com/ppanyukov/vso-agent-docker/blob/master/autobuild/vsts-agent-d/centos7-2.107.1/files/home/vsoagent/run-vsts-agent-d.sh#L244
then, I wait for all child processes to terminate using wait
. See https://github.com/ppanyukov/vso-agent-docker/blob/master/autobuild/vsts-agent-d/centos7-2.107.1/files/home/vsoagent/run-vsts-agent-d.sh#L272
when any of the signals arrive to my script (HUP
, INT
, QUIT
, TERM
), the clean_up
routine runs. You will see it kills all child processes by sending them SIGINT
signal and wait for 2 seconds for them to really die (the reason for SIGINT
is the agent does not respond to SIGTERM
due to bug in dotnet for which I raised issue long time ago). As far as I can see the "standard" containers don't do that. The last step is to try unregister the agent from VSTS which we both seem to do in the same way. See the cleanup routine here: https://github.com/ppanyukov/vso-agent-docker/blob/master/autobuild/vsts-agent-d/centos7-2.107.1/files/home/vsoagent/run-vsts-agent-d.sh#L59
I've spent quite a lot of time getting this to work reliably, it's very fiddly. Feel free to reuse any parts of this in any way that might help.
Regarding the "I would like the benefit of this to be able to handle the restart, but I need to base it off of their standard image so I can get Java tooling."
This is tricky and not so easy to do as far as I can tell.
I can suggest these approaches:
Use my vsts-agent-d
as the base for your own images and add whatever tools (JDK, dotnet etc) there. You can publish your image to docker hub and there you go. This should be pretty easy to do. This is the easiest option. The image on docker hub is here: https://hub.docker.com/r/ppanyukov/vsts-agent-d/
If you don't want to build your own images, the vsts-agent-d
supports running other docker images using host's docker daemon. This involves a whole lot of volume mapping etc, so not such an easy option. But that's what we do at work: we have a generic VSTS agent which just invokes other images with whatever tools and libraries we need. This would require quite a bit of work to get done properly though.
If you really must use their standard images, then I suppose you can replace their start.sh
script with your own based on my startup script in the hope to fix the "hanging" issue.
Hope this helps a bit. Let me know if I can help further.
Thanks @ppanyukov. Have you considered submitting a PR containing your handling of the SIGTERM to the MS vsts-agent-docker repo? (https://github.com/Microsoft/vsts-agent-docker).. I think what you have done is an extremely important contribution.
@AmitBhatnagar24 I can see there was some work done in Microsoft repo to address your issue. Suppose we can close this one?
The MS docker images (https://github.com/Microsoft/vsts-agent-docker) have "standard" variations that include some of the common non dotnet tooling (i.e.: java, ant, maven, etc) that I use to build our Java projects.
I stumbled upon this when trying to find a solution to their docker images getting stuck in a restarting state when the docker daemon is restarted. This appears to be caused by improper handling of SIGTERM which your docker agent seems to handle (how?!)
So I would like the benefit of this to be able to handle the restart, but I need to base it off of their standard image so I can get Java tooling.