search

JeanRev / TeamcityDockerCloudPlugin

Docker Cloud Plugin for the TeamCity build server
Apache License 2.0
33 stars 10 forks source link

Cannot test Docker agent container on Mac #31

Open spbkelt opened 7 years ago

spbkelt commented 7 years ago

Hello,

I did setup of cloud profile and custom image that i pull from private registry. I have local TC server setup on Mac OS X 10.12 Sierra. TC server url is binded to my local public IP. Not localhost

Here is screenshot with error message once i push Test Container button: https://www.dropbox.com/s/zpu7xt6ksmveo3a/Screenshot%202017-08-16%2018.15.42.png?dl=0

From teamcity-server.log

[2017-08-16 18:12:11,253] ERROR - jetbrains.buildServer.SERVER - Error java.lang.IllegalStateException: getWriter() has already been called for this response while processing req uest: POST '/plugins/docker-cloud/test-container.html?action=create&action=edit&prop:profileName=Docker%20Profile%20for%20Linux%20Agents%20cloud%20profile&prop:profileDescription=&p rop:terminateTimeOut=30&prop:terminateAfterFirstBuild=true&-ufd-teamcity-ui-cloudType=Docker&cloudType=VRDC&prop:run.var.teamcity.docker.cloud.use_default_unix_socket=false&prop:run .var.teamcity.docker.cloud.instance_uri=tcp%3A%2F%2F10.69.11.211&prop:run.var.teamcity.docker.cloud.tested_image=%7B%22Administration%22%3A%7B%22Version%22%3A4%2C%22RmOnExit%22%3Afa lse%2C%22PullOnCreate%22%3Atrue%2C%22MaxInstanceCount%22%3A2%2C%22UseOfficialTCAgentImage%22%3Afalse%2C%22Profile%22%3A%22m1_java7_python_base%22%7D%2C%22Container%22%3A%7B%22Image% 22%3A%22registry2.swarm.devfactory.com%2Faurea%2Fteamcity-linux%2Fm1%3Amoemscid-23604-java7-python%22%2C%22HostConfig%22%3A%7B%22OomKillDisable%22%3Afalse%2C%22Privileged%22%3Afalse %7D%7D%2C%22Editor%22%3A%7B%22MemoryUnit%22%3A%22bytes%22%2C%22MemorySwapUnit%22%3A%22bytes%22%7D%7D&prop:run.var.teamcity.docker.cloud.client_uuid=&prop:source_images_json=%5B%7B%2 2source-id%22%3A%22m1_java7_python_base%22%7D%5D&prop:run.var.teamcity.docker.cloud.img_param=%5B%7B%22Administration%22%3A%7B%22Version%22%3A4%2C%22RmOnExit%22%3Afalse%2C%22PullOnC reate%22%3Atrue%2C%22MaxInstanceCount%22%3A2%2C%22UseOfficialTCAgentImage%22%3Afalse%2C%22Profile%22%3A%22m1_java7_python_base%22%7D%2C%22Container%22%3A%7B%22Image%22%3A%22registry 2.swarm.devfactory.com%2Faurea%2Fteamcity-linux%2Fm1%3Amoemscid-23604-java7-python%22%2C%22HostConfig%22%3A%7B%22OomKillDisable%22%3Afalse%2C%22Privileged%22%3Afalse%7D%7D%2C%22Edit or%22%3A%7B%22MemoryUnit%22%3A%22bytes%22%2C%22MemorySwapUnit%22%3A%22bytes%22%7D%7D%5D&prop:run.var.teamcity.docker.cloud.daemon_info=&prop:run.var.teamcity.docker.cloud.server_url=http%3A%2F%2F192.168.31.247%3A8111%2F&prop:agentPushPreset=&publicKey=009d9a4358f47f437bacff072e09522bee8caaf60c5c5f254cf9369a4eb854cdadd672c0447ce02ea002c289bf91e38751feabe654c4a31a2bc4c3b63a36cac2e3235a58a3cf901e800e9daad00d16c0f06fbdcbdd32c3e5c99e61eb4344c3bcd0b4b0b11b04b280417b97a84db03505ff8d9274168928bfaa503270ae5212767d&profileId=Cntnr-1&save=Save', from client 192.168.31.247:60392, authenticated as 'admin' (admin) {id=1} java.lang.IllegalStateException: getWriter() has already been called for this response at org.apache.catalina.connector.Response.getOutputStream(Response.java:629) at org.apache.catalina.connector.ResponseFacade.getOutputStream(ResponseFacade.java:196) at javax.servlet.ServletResponseWrapper.getOutputStream(ServletResponseWrapper.java:95) at run.var.teamcity.cloud.docker.web.BaseFormJsonController.doHandle(BaseFormJsonController.java:26) at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:75) at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147) at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869) at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:33) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:3) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:107) at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:70) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112) at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:4) at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112) at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:73) at jetbrains.buildServer.web.DelegatingFilter.doFilter(DelegatingFilter.java:14) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:11) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

JeanRev commented 7 years ago

Hi @spbkelt ! Looks like something is wrong with your Docker connection settings. The error you're seeing in the logs is not necessarily related to your issue (such exception may happens from time to time depending on various factors). You should see some more information in the teamcity-clouds.log file, or alternatively (as the error message displayed in the dialog suggest it, although it could be more explicit) you can check the connection with the Docker daemon using the "Check connection" button on the main page, you should then get a more precise diagnostic.

spbkelt commented 7 years ago

I still don't have exact understanding. My debug steps are:

  1. Check Docker daemon connection - SUCCESS https://www.dropbox.com/s/lewo6e1uo3toxxc/Screenshot%202017-08-16%2021.51.25.png?dl=0
  2. Create Docker container using local Docker daemon in order to check connectivity between alpine test container and TC server endpoint - SUCCESS
/ # curl http://192.168.31.247:8111/ -v
*   Trying 192.168.31.247...
* TCP_NODELAY set
* Connected to 192.168.31.247 (192.168.31.247) port 8111 (#0)
> GET / HTTP/1.1
> Host: 192.168.31.247:8111
> User-Agent: curl/7.55.0
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Server: Apache-Coyote/1.1
< WWW-Authenticate: Basic realm="TeamCity"
< Cache-Control: no-store
< Content-Type: text/plain;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Wed, 16 Aug 2017 18:49:47 GMT
< 
Authentication required
* Connection #0 to host 192.168.31.247 left intact
To login manually go to "/login.html" page/ #
  1. Logs above seems to be reaction of Save button click in UI

Once again

[2017-08-16 21:48:21,877]  ERROR -   jetbrains.buildServer.SERVER - Error java.lang.IllegalStateException: getWriter() has already been called for this response while processing req
uest: POST '/plugins/docker-cloud/test-container.html?action=create&action=edit&prop:profileName=Docker%20Profile%20for%20Linux%20Agents%20cloud%20profile&prop:profileDescription=&p
rop:terminateTimeOut=30&prop:terminateAfterFirstBuild=true&-ufd-teamcity-ui-cloudType=Docker&cloudType=VRDC&prop:run.var.teamcity.docker.cloud.use_default_unix_socket=false&prop:run
.var.teamcity.docker.cloud.instance_uri=tcp%3A%2F%2F127.0.0.1%3A2376&prop:run.var.teamcity.docker.cloud.tested_image=%7B%22Administration%22%3A%7B%22Version%22%3A4%2C%22RmOnExit%22%
3Afalse%2C%22PullOnCreate%22%3Atrue%2C%22MaxInstanceCount%22%3A2%2C%22UseOfficialTCAgentImage%22%3Afalse%2C%22Profile%22%3A%22m1_java7_python_base%22%7D%2C%22Container%22%3A%7B%22Im
age%22%3A%22registry2.swarm.devfactory.com%2Faurea%2Fteamcity-linux%2Fm1%3Amoemscid-23604-java7-python%22%2C%22HostConfig%22%3A%7B%22OomKillDisable%22%3Afalse%2C%22Privileged%22%3Af
alse%7D%7D%2C%22Editor%22%3A%7B%22MemoryUnit%22%3A%22bytes%22%2C%22MemorySwapUnit%22%3A%22bytes%22%7D%7D&prop:run.var.teamcity.docker.cloud.client_uuid=&prop:source_images_json=%5B%
7B%22source-id%22%3A%22m1_java7_python_base%22%7D%5D&prop:run.var.teamcity.docker.cloud.img_param=%5B%7B%22Administration%22%3A%7B%22Version%22%3A4%2C%22RmOnExit%22%3Afalse%2C%22Pul
lOnCreate%22%3Atrue%2C%22MaxInstanceCount%22%3A2%2C%22UseOfficialTCAgentImage%22%3Afalse%2C%22Profile%22%3A%22m1_java7_python_base%22%7D%2C%22Container%22%3A%7B%22Image%22%3A%22regi
stry2.swarm.devfactory.com%2Faurea%2Fteamcity-linux%2Fm1%3Amoemscid-23604-java7-python%22%2C%22HostConfig%22%3A%7B%22OomKillDisable%22%3Afalse%2C%22Privileged%22%3Afalse%7D%7D%2C%22
Editor%22%3A%7B%22MemoryUnit%22%3A%22bytes%22%2C%22MemorySwapUnit%22%3A%22bytes%22%7D%7D%5D&prop:run.var.teamcity.docker.cloud.daemon_info=&prop:run.var.teamcity.docker.cloud.server_url=&prop:agentPushPreset=&publicKey=00c90c69a6776c2f32b91472e84bb621bb081788c3d945af58b674e821acd042a832ee6d0b87695d5d379fbed9abe87f7e75cbfb58dbce0a11e646c075fefdf146fc166246d553f753fa61c8ca321c812aa0c0fe8ef35f4dd9e9a29376087ae4aa8a628f165c54afde6f0cc5643123702b93ae4c695ddf379f8058c229369bba8f&profileId=Cntnr-1&save=Save', from client 192.168.31.247:55534, authenticated as 'admin' (admin) {id=1}
spbkelt commented 7 years ago

From teamcity-clouds.log

[2017-08-16 21:31:08,574]   INFO [nio-8111-exec-4] - ker.client.DockerClientFactory - Negotiation successful for target version 1.26 (daemon current version: 1.30, minimal API version: 1.12). 
[2017-08-16 21:37:27,791]   INFO [io-8111-exec-10] - ker.client.DockerClientFactory - Negotiation successful for target version 1.26 (daemon current version: 1.30, minimal API version: 1.12). 
[2017-08-16 21:37:28,203]   INFO [nio-8111-exec-6] - ker.client.DockerClientFactory - Negotiation successful for target version 1.26 (daemon current version: 1.30, minimal API version: 1.12). 
[2017-08-16 21:37:44,924]   INFO [nio-8111-exec-1] - cloud.docker.DockerImageConfig - Loading cloud profile configuration version 4. 
[2017-08-16 21:38:25,994]   INFO [nio-8111-exec-4] - ker.client.DockerClientFactory - Negotiation successful for target version 1.26 (daemon current version: 1.30, minimal API version: 1.12). 
[2017-08-16 21:46:18,015]   INFO [nio-8111-exec-9] - ker.client.DockerClientFactory - Negotiation successful for target version 1.26 (daemon current version: 1.30, minimal API version: 1.12). 
[2017-08-16 21:47:59,789]   INFO [nio-8111-exec-8] - ker.client.DockerClientFactory - Negotiation successful for target version 1.26 (daemon current version: 1.30, minimal API version: 1.12). 
[2017-08-16 21:48:07,672]   INFO [nio-8111-exec-2] - cloud.docker.DockerImageConfig - Loading cloud profile configuration version 4.
JeanRev commented 7 years ago

Ok, that's unusual. Can you please try with this version of the plugin ? It's based on version 0.4.1 with a few minor bugfixes not yet released: docker-cloud_0.4.2-SNAPSHOT_c15f381.zip

The changes in this version are not targetting your particular issue, but should provide more logs in the teamcity-clouds.log file to track the root of the problem you're experiencing.

spbkelt commented 7 years ago

Thanks @JeanRev ! I will give you feedback soon BTW, is it possible to use inside Environment Variables section of Edit Image->Run UI Configuration Parameters from build configurations in TC "native" way, like %ConfigurationParameter1%, %env.EnvVariable2%, or whatever?

Example: https://www.dropbox.com/s/s1f9h6wma72lu7l/Screenshot%202017-08-16%2023.13.42.png?dl=0

spbkelt commented 7 years ago

After 0.4.2 testing still same error in UI: Invalid configuration. Please check your connection settings.

In teamcity-clouds.log:

[2017-08-16 23:14:35,845]  ERROR [nio-8111-exec-2] - er.web.ContainerTestController - Invalid configuration. 
run.var.teamcity.cloud.docker.DockerCloudClientConfigException: [InvalidProperty{myInvalidReason='Cloud UUID ist not set', myPropertyName='run.var.teamcity.docker.cloud.client_uuid'}]
        at run.var.teamcity.cloud.docker.DockerCloudClientConfig.processParams(DockerCloudClientConfig.java:177)
        at run.var.teamcity.cloud.docker.web.ContainerTestController.doPost(ContainerTestController.java:147)
        at run.var.teamcity.cloud.docker.web.BaseFormJsonController.doHandle(BaseFormJsonController.java:24)
        at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:75)
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
        at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:33)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:3)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:107)
        at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:70)
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
        at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:4)
        at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
        at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:73)
        at jetbrains.buildServer.web.DelegatingFilter.doFilter(DelegatingFilter.java:14)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:11)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1757)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1716)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
spbkelt commented 7 years ago

@JeanRev fixed! So i've just removed my cloud profile and created new one. Now it works: https://www.dropbox.com/s/cbzp3gzb7b1uq0p/Screenshot%202017-08-16%2023.20.00.png?dl=0

JeanRev commented 7 years ago

Glad to hear it :-)

At the same time, I think I've located the cause of the regression that may affect old installations of the plugin (a required property field that may not have been generated at the time the profile was created). This will be adressed.

Regarding your question about the build parameters notation (using percent signs), it's currently not supported, but the idea sounds interesting... I'll see if I can come up with a concept. Probably not all fields would be candidate for such parameter expansion, and it will be necessary to guarantee backward compatibility with existing configs, but it might be doable. I'll prepare an issue.

Thank you for reporting!

spbkelt commented 7 years ago

Thanks for support! One more question, actually use case: how to update docker image in the cloud profile if i pushed image with same tag to the private docker registry? Seems it has not been updated automatically, but image has new ID. I think it's reasonable to poll registry using some configurable schedule in order to fetch fresh image. What do you think?

JeanRev commented 7 years ago

Actually, if you set the "Pull image before creating container" option on the first tab of the image settings, the plugin should already try to update the image with the given name in the local daemon storage before creating each container.

If for some reason the pull failed, but an image with the given name already exists locally, the plugin will simply log the failure in the teamcity-clouds log file and start a container with the available image.

As a side note: pulling will only occurs when creating new containers. If you choose in the plugin settings to preserver the containers once they are stopped, then they will still be based on the same image when they are later restarted...

spbkelt commented 7 years ago

@JeanRev Thanks for clarification.