search

JeanRev / TeamcityDockerCloudPlugin

Docker Cloud Plugin for the TeamCity build server
Apache License 2.0
33 stars 10 forks source link

dispach jobs across cloud setups #35

Open jeandet opened 6 years ago

jeandet commented 6 years ago

In my setup I have 2 identical cloud profiles, same machines and same agent images. It looks like this plugin starts all allowed images on the first machine before starting on the second machine.

For example, I allow to start 2 linux images and one BSD on each machine if in the queue there are 2 linux jobs and 1 BSD, then all jobs will run on the first machine while it would make sense to split jobs on two machines.

JeanRev commented 6 years ago

Hi @jeandet,

Dispatching of jobs between cloud profiles (when several them are compatible) is made by the TC server itself, and the plugin can't have an impact on this. So, in this situation, there is really nothing you can do.

What you most likely need is support for Docker Swarm, which is currently incubating and will be released at the beginning of next year. This will allow you to have a single cloud profile that will interact with your machine cluster, and the Swarm orchestrator will then be in charge to dispatch tasks accross the Swarm nodes in a smart way. The orchestration settings that will be exposed by the plugin will be a bit limited at first, but the default ones should already provide the behavior you describe.

If you want to try it out, you may use this dev artifact (it hasn't been toroughly tested yet): docker-cloud_0.6.0-SNAPSHOT_935f633.zip

It will basically add a new type of cloud profile "Docker Swarm", in which you will need to specify the IP of a Swarm manager instead of a standalone Docker daemon. To init and manage a Swarm you can refer to the Docker documentation: https://docs.docker.com/engine/swarm/swarm-tutorial/create-swarm/

If you do try this artifact, just let me know if it works as intended :-)

jeandet commented 6 years ago

Hi @JeanRev ,

Thank you for your answer, it's clear. I didn't know that TeamCity itself dispatch jobs. I'm definitely going to switch to Docker Swarm but first I have to setup a manager for that :).

jeandet commented 6 years ago

Hello @JeanRev ,

I tried this plugin version but it seems to crash with both swarm mode and previous mode :

Docker mode:

Failed to initialize cloud client 'VRDC'.jetbrains.buildServer.clouds.CloudImageParameters.collectionToJson(Ljava/util/Collection;)Ljava/lang/String;
Hide stacktrace
java.lang.NoSuchMethodError: jetbrains.buildServer.clouds.CloudImageParameters.collectionToJson(Ljava/util/Collection;)Ljava/lang/String;
at run.var.teamcity.cloud.docker.DockerCloudClientFactory.createNewClient(DockerCloudClientFactory.java:58)
at jetbrains.buildServer.clouds.server.impl.ProjectCloudManagerImpl.createNewClient(ProjectCloudManagerImpl.java:384)
at jetbrains.buildServer.clouds.server.impl.ProjectCloudManagerImpl.getClient(ProjectCloudManagerImpl.java:303)
at jetbrains.buildServer.clouds.server.impl.CloudManagerBaseImpl.getClient(CloudManagerBaseImpl.java:408)
at jetbrains.buildServer.clouds.server.impl.CloudInstancesProviderImpl.iterateProfileInstances(CloudInstancesProviderImpl.java:75)
at jetbrains.buildServer.clouds.server.impl.CloudInstancesProviderImpl.iterateInstances(CloudInstancesProviderImpl.java:65)
at jetbrains.buildServer.clouds.server.impl.instances.RunningInstanceTrackerImpl.update(RunningInstanceTrackerImpl.java:98)
at jetbrains.buildServer.clouds.server.impl.instances.RunningInstanceTrackerImpl$1.serverStartup(RunningInstanceTrackerImpl.java:70)
at sun.reflect.GeneratedMethodAccessor145.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at jetbrains.buildServer.util.EventDispatcher$3.run(EventDispatcher.java:126)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:71)
at jetbrains.buildServer.util.EventDispatcher.dispatch(EventDispatcher.java:120)
at jetbrains.buildServer.serverSide.ServerSideEventDispatcher.superDispatch(ServerSideEventDispatcher.java:2)
at jetbrains.buildServer.serverSide.ServerSideEventDispatcher$1.run(ServerSideEventDispatcher.java:2)
at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAs(SecurityContextImpl.java:32)
at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAsSystem(SecurityContextImpl.java:50)
at jetbrains.buildServer.serverSide.ServerSideEventDispatcher.dispatch(ServerSideEventDispatcher.java:14)
at jetbrains.buildServer.util.EventDispatcher$2.invoke(EventDispatcher.java:70)
at com.sun.proxy.$Proxy21.serverStartup(Unknown Source)
at jetbrains.buildServer.serverSide.impl.BuildServerLifecycleProcessor.doStartup(BuildServerLifecycleProcessor.java:22)
at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet$WebApplicationCreatorAndDestroyer.createApplication(TeamCityDispatcherServlet.java:22)
at jetbrains.buildServer.maintenance.StartupProcessor.doApplicationStarting(StartupProcessor.java:736)
at jetbrains.buildServer.maintenance.StartupProcessor.access$700(StartupProcessor.java:499)
at jetbrains.buildServer.maintenance.StartupProcessor$3.call(StartupProcessor.java:14)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:86)
at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStage(StartupProcessor.java:806)
at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStageSafe(StartupProcessor.java:311)
at jetbrains.buildServer.maintenance.StartupProcessor.processTeamCityLifecycle(StartupProcessor.java:11)
at jetbrains.buildServer.maintenance.StartupProcessor.access$000(StartupProcessor.java:93)
at jetbrains.buildServer.maintenance.StartupProcessor$1.run(StartupProcessor.java:2)
at java.lang.Thread.run(Thread.java:748)

Swarm mode:

Error accessing server. HTTP status: 500

Response: 

<html>

  <head>

    <title>Unexpected Error</title>

  </head>

  <body>

  <div class="errorPage">

    <h1 class="errorTitle">Unexpected Error</h1>

<p>

    This was not supposed to happen.

    Please provide the error details to your TeamCity server maintainer.<br/>

    If you maintain this TeamCity installation and the request seems legit please

    <a href="https://confluence.jetbrains.com/display/TW/Feedback">report</a> this error to JetBrains.

</p>

<div class="errorMessage">

  Error: java.lang.NoSuchMethodError: jetbrains.buildServer.clouds.CloudImageParameters.collectionFromJson(Ljava/lang/String;)Ljava/util/Collection;

</div>

<br/>

<div>

  Server time: 2017-12-28 16:38:46

</div>

<div>

  TeamCity: 2017.2.1 (build 50732)

</div>

<div>

  Servlet container: Apache Tomcat/8.5.23

</div>

<br/>

  <a id="stackTraceLink" href="#" onclick="$('stackTrace').show(); $('stackTraceLink').hide(); return false">Show stacktrace</a>

  <pre id="stackTrace" style="display: none" class="stackTrace">Trace: java.lang.NoSuchMethodError: jetbrains.buildServer.clouds.CloudImageParameters.collectionFromJson(Ljava/lang/String;)Ljava/util/Collection;
    at run.var.teamcity.cloud.docker.DockerImageConfig.processParams(DockerImageConfig.java:159)
    at run.var.teamcity.cloud.docker.DockerCloudPropertiesProcessor.process(DockerCloudPropertiesProcessor.java:39)
    at jetbrains.buildServer.clouds.server.web.admin.CloudProfileFormAction$2.process(CloudProfileFormAction.java:303)
    at jetbrains.buildServer.controllers.ActionErrors.fillErrors(ActionErrors.java:153)
    at jetbrains.buildServer.clouds.server.web.admin.CloudProfileFormAction.processProfileAction(CloudProfileFormAction.java:79)
    at jetbrains.buildServer.clouds.server.web.project.CloudEditProfileController.processAction(CloudEditProfileController.java:121)
    at jetbrains.buildServer.clouds.server.web.project.CloudEditProfileController.doPost(CloudEditProfileController.java:115)
    at jetbrains.buildServer.clouds.server.web.project.CloudEditProfileController.doPost(CloudEditProfileController.java:109)
    at jetbrains.buildServer.controllers.BaseFormXmlController$1.handleRequest(BaseFormXmlController.java:54)
    at jetbrains.buildServer.controllers.AjaxRequestProcessor.processRequest(AjaxRequestProcessor.java:47)
    at jetbrains.buildServer.controllers.BaseFormXmlController.doHandle(BaseFormXmlController.java:52)
    at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:101)
    at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:174)
    at jetbrains.buildServer.controllers.BaseController.handleRequest(BaseController.java:80)
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.processedByMainServlet(TeamCityDispatcherServlet.java:8)
    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:4)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jetbrains.buildServer.web.jsp.JspPrecompilerFilter.doFilter(JspPrecompilerFilter.java:161)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:8)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:107)
    at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:46)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:10)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.web.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:2)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.web.CSRFFilter.doFilter(CSRFFilter.java:100)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:73)
    at jetbrains.buildServer.web.DelegatingFilter.doFilter(DelegatingFilter.java:37)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:9)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2381)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
</pre>

  </div>

  <!--

  IE does not display error page content if size of the page is relatively small...

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  01010101001010101010100101010101010010101010101001010101010100101010101010010101010101001010101010100101

  -->

  </body>

</html>
jeandet commented 6 years ago

I don't understand anything in Java and TeamCity internals, but maybe this commit isn't merged in the 0.6.0?

JeanRev commented 6 years ago

Hi @jeandet,

Sorry for the delayed answer. You assume right, the dev artifact in the link lacks a required upstream patch for the latest TC version.

Here is a newer artifact that should be compatible: docker-cloud_0.6.0-SNAPSHOT_a210e8a.zip

jeandet commented 6 years ago

Hi @JeanRev ,

It seems to work, at first view it seems to miss the "Management:" entry and security tab. Is this due to swarm? Anyway thank you! This plugin is really nice!

JeanRev commented 6 years ago

Thank you for the feedback :-)

Yes, the settings panel is somewhat different between swarm and "regular" mode, simply because the Docker API does not offer the same capabilities for swarm managed containers and standalone containers. In both cases, the settings UI only offers only a subset of all the available options, but this will be improved over time.