search

JetBrains / teamcity-docker-agent

TeamCity agent docker image sources
https://hub.docker.com/r/jetbrains/teamcity-agent/
Apache License 2.0
77 stars 64 forks source link

Container dies during builds #43

Open tg73 opened 5 years ago

tg73 commented 5 years ago

I suspect that this is a Windows Containers problem, not a TeamCity problem, but I'm also logging it here as it may affect fellow users. I have had to abandon using containerized build agents and revert to using VMs.

When running a build via the agent running within the container, at some arbitrary point, the container becomes unresponsive. With process isolation, RDP to the host OS also becomes unresponsive and the host eventually reboots. With hyperv isolation, the container becomes unresponsive and then stops, but the host OS stays up and responsive. Builds do sometimes complete, but more often than not they fail. TeamCity server reports a loss of connection to the build agent, and eventually the build is marked as failed.

The original issue is logged here, with a bit more detail: https://github.com/docker/for-win/issues/426#issuecomment-423147259

dtretyakov commented 5 years ago

@tg73, thanks for reporting this case. Could you please describe what type of hardware (physical box, cloud, virtualized machine) was used to run host machine with Windows Server Core 1803?

tg73 commented 5 years ago

The Windows Server Core 1803 Docker host is running as a VM hosted by ESXi 6.7.0, configured with 8 vCPU, 16GB RAM. Expose hardware virtualization to guest was originally disabled when I was trying with process isolation, then turned on when trying hyperv isolation. IOMMU and virutalized CPU performance counters are exposed to the guest. The non-docker agent VMs that I am now using instead of docker containers run on the same physical ESXi box without any problems (Windows Server Core 1803 VMs), so hardware issues are very unlikely to be the cause.