Support "by agent build history" mode

[ogerovich]

Hi Grigory,

We have a need to force the build to the agent that ran it previously. We have a massive compilation build that takes ~1.5 hours from scratch, but less time if it runs on the same agent with incremental compilation. Your plugin has a nice feature of showing the proposed order of matching agents. So I'd like to propose new rules for the agent order that would suit our needs:

1. Agent that ran the build last should be first in the list.
2. Agent that ran the build before the last agent should be second in the list.
3. Repeat until all agents that appeared in build history are in the front of list.
4. The rest of the agents can use existing JetBrains logic.

Perhaps implementation should pull the list for existing JetBrains logic (I think that will take care of agent compatibility) and push the agents that fit my criteria to the front of the list.

I'm not sure if "by agent build history" is the best name, so feel free to propose a different name. Thanks!

[grundic]

Hi!

By default, Teamcity tries to use agents, where the given build was executed. I could implement this functionality as well, but this should work already for you, as this is default behaviour. Also, there is ByBuildStatus priority, which also works with the history of the build, but it moves failed build to the end, so if you have one agent with successful build, one with failed and another with no build, it will order them as successful -> no build -> failed. I suppose you would like it to be like successful -> failed -> no build.

As a solution that we used before for the same problem, you can create a "caching" build:

• configure the same VCS rules and checkout directory,
• remove all run configurations
• let it run on all agents
• trigger it once a night/week

Right now I'm on vacation, so don't expect quick answers/implementations from me.

Good luck and happy new year!

[ogerovich]

Hi Grigory,

Happy holidays!

The default TeamCity behavior does not guarantee that the last agent will be used. ByBuildStatus does not seem to fit my needs. I want the last agent regardless of build status. In case the last agent is currently busy, the logic I described should apply.

I don't have an urgent need - I have a Java plugin that currently handles #1 and #4 step. It just seems that your plugin is much better organized and has that display feature. My request is really a future improvement, so there is no rush. I'd implement the change myself, but I'm not sure when I'll have the time. Let's see which one of us gets to it first. :-)

[grundic]

Hi @ogerovich,

I've reviewed your requirements and still think that ByBuildStatus priority can work for you.

Here how the weight for each build/agent is calculated:

        int coefficient = index * historyCoefficient + baseScale;
        Status status = build.getBuildStatus();
        if (status.isSuccessful()) {
            weight += successfulScore * coefficient;
        } else {
            weight += failedScore * coefficient;
        }

The default values are: Build limit: 10 History coefficient: -1 Base scale: 0 Successful score: 1 Failed score: -1

So for your case you can use these values: Build limit: 1 History coefficient: 1 Base scale: 0 Successful score: 1 Failed score: 1

This will take the results of the last build and would make no difference between successful and failed one.

[ogerovich]

Hi Grigory,

Thank you for the instructions. I will try your suggestion.

I just noticed a couple of small issues:

1. Typo in the label for History coefficient in the priority popup: "The valued is multiplied..."
2. Edit project pages show "Agent priorit" as tab title instead of "Agent priority".
3. "Available ordering of build agents:By build status, By configuration parameter, By CPU benchmark index, By name, By random number." Add space after : for readability?

If you want me to enter separate issues for each one, I will. :-)

[grundic]

@ogerovich, Thank you for reporting these small issues. Number 1 and 3 I'll fix occasionally, and number 2 is just not rendered correctly. I believe it's not enough space for it.

As for solution I've provide you: I think I was wrong about Build limit. In case you will set it to 1, only last build would be taken into account. And if the agent is busy, the rest agents won't be sorted as expected. So it's better to increase the number slightly.

[ogerovich]

I think there should be enough space. The same pane is showing "Shared Resources", "Issue Trackers 1" and "Versioned History (checkmark)", which are longer.

Let's say I set it to 4 and all 4 agents that ran the build last are busy, but there are other free compatible agents. What will happen then?

[grundic]

Ah, taking more precise look at the code I've figured out that getTabTitle method was missing a space, while adding number of configured agent priorities to the title. Adding it fixed the problem, thank you for obstinacy!

What about the example with 4 builds/agents: in this case some random agent would be choosed.

[ogerovich]

Random agent is not bad, but my rule 4 would be best. Is there any way to do that?

[grundic]

Yep, just increase the Build limit to the desired value: plugin will explore as much builds as configured by this parameter.

[ogerovich]

Hi Grigory!

I finally got around to testing your suggestion:

So for your case you can use these values: Build limit: 1 History coefficient: 1 Base scale: 0 Successful score: 1 Failed score: 1

This works as far as sending the last agent to the top of the list. However, the rest of the list seems to be in alphabetical order, which is undesirable.

Consider this history of agents (most recent build first):

HV58 HV58 HV58 HV58 HV58 HV58 HV58 HV42 HV42 HV58 HV44 HV44 HV44

If I set build limit to 1, the order is:

HV58 HV40 HV41 HV42 ...

If I set build limit to 10, the order is:

HV58 HV42 HV40 HV41 ...

What I need is this:

HV58 HV42 HV44 ...

I think I am looking for something other than what your formula computes. I want to walk back in history and append agents to the resulting list if they have not yet appeared in the resulting list. It might be fine to limit the build history lookup as you do now or provide a way to specify the max number of agents to append to the resulting list. Once this is done, add the rest of the compatible agents to the list. You can get compatible agents like this: Collection<SBuildAgent> possibleAgents = context.getAgentsForStartingBuild(); Then exclude agents already on the resulting list and append to the end of the list, thus keeping the order provided by TC.

Let me know if you need me to clarify my needs further.

Thanks, Oleg.

[ogerovich]

Here is another scenario:

Build limit: 2 History coefficient: 1 Base scale: 0 Successful score: 1 Failed score: 1

History of agents (most recent build first):

CH32 CH28 CH28 CH28 CH28 CH28 CH28 CH28 CH32 CH32 CH32 ...

The order of the agent priority is: CH28 CH11 CH13 CH24 CH27 CH29 CH30 CH32 CH33

The order I expected is: CH32 CH28 CH11 CH13 CH24 CH27 CH29 CH30 CH33 ...

At the time of my testing, this build was running on CH13. It seems like agent priority skipped the latest build and used the second and third build to come up with CH28 in the first position. That's incorrect. Is there a bug in this section of the code?

        if (build.getAgent() != buildAgent) {
            return true;
        }

        if (index > buildLimit) {
            return false;
        }

[ogerovich]

There seems to be another problem. Build Queue page is slow to load, depending on the number of builds in the queue. The more builds in the queue - the slower it loads. If I disable the priority that I previously set in the Root project, the problem goes away, so I'm convinced it's the plugin. I took a few thread dumps and found several stacks that reference the plugin:

"TC: 10:36:50 SQL SELECT: select BS.build_type_id, BS.build_id, BS.id, A.build_number, A.agent_name, A.agent_type_id, A.build_start_time_server, A.build_start_time_agent, A.build_finish_time_server, A.status, A.status_text, A.user_status_text, A.pin, A.requestor, A.is_canceled, C.user_id, C.description, BS.remove_from_queue_time from build_state BS join history A on BS.build_type_id = A.build_type_id and BS.build_id = A.build_id left join canceled_info C on C.build_id = A.build_id  where (BS.build_type_id = ?) and (BS.remove_from_queue_time is not null and BS.is_deleted = 0) and (A.is_canceled = 0) and (A.is_personal = 0) order by BS.build_type_id desc, BS.remove_from_queue_time desc, BS.build_id desc | PARAMETERS: "bt152"; 10:36:49 Flush Queue 1" daemon group="main" prio=5 tid=227 nid=227 runnable 
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
    at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3922)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2551)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2731)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2812)
    at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:5339)
    at org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:378)
    at org.apache.commons.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:378)
    at org.apache.commons.dbcp.PoolableConnectionFactory.activateObject(PoolableConnectionFactory.java:704)
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1204)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:115)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1140)
    at jetbrains.buildServer.serverSide.db.ReliablePoolingDataSource.getConnection(ReliablePoolingDataSource.java:68)
    at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
    at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:225)
    at com.sun.proxy.$Proxy36.prepareStatement(Unknown Source)
    at jetbrains.buildServer.serverSide.db.DBFunctions.prepareStatementForMoreThanOneRow(DBFunctions.java:2276)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuplesInternal(DBFunctions.java:1519)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuples(DBFunctions.java:1456)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuples(DBFunctions.java:1431)
    at jetbrains.buildServer.serverSide.impl.history.HistoryQueryRunner$9.run(HistoryQueryRunner.java:1)
    at jetbrains.buildServer.serverSide.db.DBFacade$3.run(DBFacade.java:223)
    at jetbrains.buildServer.serverSide.db.DBFacade$3.run(DBFacade.java:221)
    at jetbrains.buildServer.serverSide.db.DBFacade.withDB(DBFacade.java:242)
    at jetbrains.buildServer.serverSide.db.DBFacade.withDB(DBFacade.java:221)
    at jetbrains.buildServer.serverSide.impl.history.HistoryQueryRunner.processItemsAtOnce(HistoryQueryRunner.java:159)
    at jetbrains.buildServer.serverSide.impl.history.HistoryQueryRunner.processItems(HistoryQueryRunner.java:151)
    at jetbrains.buildServer.serverSide.impl.BuildsManagerImpl.processBuilds(BuildsManagerImpl.java:8)
    at jetbrains.buildServer.serverSide.impl.history.DBBuildHistory.processEntries(DBBuildHistory.java:9)
    at jetbrains.buildServer.serverSide.impl.auth.SecuredBuildHistory.processEntries(SecuredBuildHistory.java:14)
    at com.github.grundic.agentPriority.prioritisation.impl.ByBuildStatus.apply(ByBuildStatus.java:100)
    at com.github.grundic.agentPriority.prioritisation.impl.ByBuildStatus.apply(ByBuildStatus.java:45)
    at com.google.common.collect.ByFunctionOrdering.compare(ByFunctionOrdering.java:46)
    at com.google.common.collect.CompoundOrdering.compare(CompoundOrdering.java:43)
    at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
    at java.util.TimSort.sort(TimSort.java:220)
    at java.util.Arrays.sort(Arrays.java:1438)
    at com.google.common.collect.Ordering.sortedCopy(Ordering.java:685)
    at com.github.grundic.agentPriority.manager.AgentPriorityManager.sort(AgentPriorityManager.java:73)
    at com.github.grundic.agentPriority.buildDistribution.PriorityAgentsFilter.filterAgents(PriorityAgentsFilter.java:71)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.DefaultAgentsFilter.filterAgents(DefaultAgentsFilter.java:27)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.BuildAgentDistributor.filterAvailableAgents(BuildAgentDistributor.java:32)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.AbstractDistributor.doDistribute(AbstractDistributor.java:9)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.DefaultBuildDistributor.assignBuildsToAgents(DefaultBuildDistributor.java:9)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.BuildDistributorProxy.assignBuildsToAgents(BuildDistributorProxy.java:12)
    at jetbrains.buildServer.serverSide.impl.FlushQueueForConnectedAgents.distribute(FlushQueueForConnectedAgents.java:66)
    at jetbrains.buildServer.serverSide.impl.FlushQueueAction$2.distribute(FlushQueueAction.java:1)
    at jetbrains.buildServer.serverSide.impl.FlushQueueTemplate.flushRealAgents(FlushQueueTemplate.java:35)
    at jetbrains.buildServer.serverSide.impl.FlushQueueTemplate.access$000(FlushQueueTemplate.java:14)
    at jetbrains.buildServer.serverSide.impl.FlushQueueTemplate$1.run(FlushQueueTemplate.java:3)
    at jetbrains.buildServer.serverSide.impl.FlushQueueAction$QueueAction.run(FlushQueueAction.java:1)
    at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupState.runUnderLock(ServerCleanupState.java:35)
    at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupState.runReadAction(ServerCleanupState.java:8)
    at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.executeWithInactiveCleanup(ServerCleanupManagerImpl.java:45)
    at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$$FastClassBySpringCGLIB$$ba2c8525.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at jetbrains.buildServer.serverSide.impl.auth.TeamCityMethodSecurityInterceptor.invoke(TeamCityMethodSecurityInterceptor.java:28)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$FixedChainStaticTargetInterceptor.intercept(CglibAopProxy.java:604)
    at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$$EnhancerBySpringCGLIB$$8091412.executeWithInactiveCleanup(<generated>)
    at jetbrains.buildServer.serverSide.impl.FlushQueueAction.flushQueue(FlushQueueAction.java:23)
    at jetbrains.buildServer.serverSide.impl.BuildServerImpl.flushQueue(BuildServerImpl.java:79)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$2.doSomething(BuildServerRunner.java:1)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.runAction(BuildServerRunner.java:11)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.run(BuildServerRunner.java:63)
    at java.lang.Thread.run(Thread.java:745)

"TC: 10:36:50 SQL SELECT: select BS.build_type_id, BS.build_id, BS.id, A.build_number, A.agent_name, A.agent_type_id, A.build_start_time_server, A.build_start_time_agent, A.build_finish_time_server, A.status, A.status_text, A.user_status_text, A.pin, A.requestor, A.is_canceled, C.user_id, C.description, BS.remove_from_queue_time from build_state BS join history A on BS.build_type_id = A.build_type_id and BS.build_id = A.build_id left join canceled_info C on C.build_id = A.build_id  where (BS.build_type_id = ?) and (BS.remove_from_queue_time is not null and BS.is_deleted = 0) and (A.is_canceled = 0) and (A.is_personal = 0) order by BS.build_type_id desc, BS.remove_from_queue_time desc, BS.build_id desc | PARAMETERS: "bt152"; 10:36:28 Calculating queued builds estimates; 10:36:28 Build estimates updater; Build estimates updater 1" daemon group="main" prio=5 tid=215 nid=215 runnable 
   java.lang.Thread.State: RUNNABLE
    at com.mysql.jdbc.ResultSetRow.getString(ResultSetRow.java:815)
    at com.mysql.jdbc.ByteArrayRow.getString(ByteArrayRow.java:70)
    at com.mysql.jdbc.ResultSetImpl.getStringInternal(ResultSetImpl.java:5816)
    at com.mysql.jdbc.ResultSetImpl.getString(ResultSetImpl.java:5693)
    at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:214)
    at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:214)
    at org.apache.commons.dbcp.DelegatingResultSet.getString(DelegatingResultSet.java:214)
    at jetbrains.buildServer.serverSide.db.DBFunctions.peekValue(DBFunctions.java:1721)
    at jetbrains.buildServer.serverSide.db.DBFunctions.peekTupleValues(DBFunctions.java:1710)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuplesInternal(DBFunctions.java:1524)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuples(DBFunctions.java:1456)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuples(DBFunctions.java:1431)
    at jetbrains.buildServer.serverSide.impl.history.HistoryQueryRunner$9.run(HistoryQueryRunner.java:1)
    at jetbrains.buildServer.serverSide.db.DBFacade$3.run(DBFacade.java:223)
    at jetbrains.buildServer.serverSide.db.DBFacade$3.run(DBFacade.java:221)
    at jetbrains.buildServer.serverSide.db.DBFacade.withDB(DBFacade.java:242)
    at jetbrains.buildServer.serverSide.db.DBFacade.withDB(DBFacade.java:221)
    at jetbrains.buildServer.serverSide.impl.history.HistoryQueryRunner.processItemsAtOnce(HistoryQueryRunner.java:159)
    at jetbrains.buildServer.serverSide.impl.history.HistoryQueryRunner.processItems(HistoryQueryRunner.java:151)
    at jetbrains.buildServer.serverSide.impl.BuildsManagerImpl.processBuilds(BuildsManagerImpl.java:8)
    at jetbrains.buildServer.serverSide.impl.history.DBBuildHistory.processEntries(DBBuildHistory.java:9)
    at jetbrains.buildServer.serverSide.impl.auth.SecuredBuildHistory.processEntries(SecuredBuildHistory.java:14)
    at com.github.grundic.agentPriority.prioritisation.impl.ByBuildStatus.apply(ByBuildStatus.java:100)
    at com.github.grundic.agentPriority.prioritisation.impl.ByBuildStatus.apply(ByBuildStatus.java:45)
    at com.google.common.collect.ByFunctionOrdering.compare(ByFunctionOrdering.java:46)
    at com.google.common.collect.CompoundOrdering.compare(CompoundOrdering.java:43)
    at java.util.TimSort.countRunAndMakeAscending(TimSort.java:360)
    at java.util.TimSort.sort(TimSort.java:220)
    at java.util.Arrays.sort(Arrays.java:1438)
    at com.google.common.collect.Ordering.sortedCopy(Ordering.java:685)
    at com.github.grundic.agentPriority.manager.AgentPriorityManager.sort(AgentPriorityManager.java:73)
    at com.github.grundic.agentPriority.buildDistribution.PriorityAgentsFilter.filterAgents(PriorityAgentsFilter.java:71)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.DefaultAgentsFilter.filterAgents(DefaultAgentsFilter.java:27)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.BuildAgentDistributor.filterAvailableAgents(BuildAgentDistributor.java:32)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.AbstractDistributor.doDistribute(AbstractDistributor.java:9)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.DefaultBuildDistributor.assignBuildsToAgents(DefaultBuildDistributor.java:9)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.BuildDistributorProxy.assignBuildsToAgents(BuildDistributorProxy.java:12)
    at jetbrains.buildServer.serverSide.impl.timeEstimation.DefaultBuildEstimatesCalculator.estimateBuilds(DefaultBuildEstimatesCalculator.java:45)
    at jetbrains.buildServer.serverSide.impl.timeEstimation.DefaultBuildEstimatesCalculator.estimateBuilds(DefaultBuildEstimatesCalculator.java:53)
    at jetbrains.buildServer.serverSide.impl.BuildEstimatorFeeder.getEstimates(BuildEstimatorFeeder.java:13)
    at jetbrains.buildServer.serverSide.impl.timeEstimation.CachingBuildEstimator.computeEstimates(CachingBuildEstimator.java:85)
    at jetbrains.buildServer.serverSide.impl.timeEstimation.CachingBuildEstimator$2$1.run(CachingBuildEstimator.java:6)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:71)
    at jetbrains.buildServer.util.ExceptionUtil$1.run(ExceptionUtil.java:41)
    at jetbrains.buildServer.serverSide.impl.timeEstimation.CachingBuildEstimator$2.run(CachingBuildEstimator.java:1)
    at java.lang.Thread.run(Thread.java:745)

"TC: 10:36:35 GET '/queue.html', from client 10.175.72.29:35828, authenticated as 'o_gerovich' (Oleg Gerovich) {id=42}; http-nio-8111-exec-572" daemon group="main" prio=5 tid=9312894 nid=9312894 runnable 
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:170)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
    at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
    at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
    at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3116)
    at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:2224)
    at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1999)
    at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:3507)
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:490)
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:3198)
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:2366)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2789)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2818)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2157)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2324)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:100)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:100)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:100)
    at jetbrains.buildServer.serverSide.db.DBFunctions$8.run(DBFunctions.java:743)
    at jetbrains.buildServer.serverSide.db.DBFunctions$8.run(DBFunctions.java:740)
    at jetbrains.buildServer.serverSide.db.DBInternalUtils.rawOperation(DBInternalUtils.java:77)
    at jetbrains.buildServer.serverSide.db.DBInternalUtils.rawOperation(DBInternalUtils.java:60)
    at jetbrains.buildServer.serverSide.db.DBFunctions.executeQuery(DBFunctions.java:740)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuplesInternal(DBFunctions.java:1521)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuples(DBFunctions.java:1456)
    at jetbrains.buildServer.serverSide.db.DBFunctions.queryForTuples(DBFunctions.java:1431)
    at jetbrains.buildServer.serverSide.impl.history.HistoryQueryRunner$9.run(HistoryQueryRunner.java:1)
    at jetbrains.buildServer.serverSide.db.DBFacade$3.run(DBFacade.java:223)
    at jetbrains.buildServer.serverSide.db.DBFacade$3.run(DBFacade.java:221)
    at jetbrains.buildServer.serverSide.db.DBFacade.withDB(DBFacade.java:242)
    at jetbrains.buildServer.serverSide.db.DBFacade.withDB(DBFacade.java:221)
    at jetbrains.buildServer.serverSide.impl.history.HistoryQueryRunner.processItemsAtOnce(HistoryQueryRunner.java:159)
    at jetbrains.buildServer.serverSide.impl.history.HistoryQueryRunner.processItems(HistoryQueryRunner.java:151)
    at jetbrains.buildServer.serverSide.impl.BuildsManagerImpl.processBuilds(BuildsManagerImpl.java:8)
    at jetbrains.buildServer.serverSide.impl.history.DBBuildHistory.processEntries(DBBuildHistory.java:9)
    at jetbrains.buildServer.serverSide.impl.auth.SecuredBuildHistory.processEntries(SecuredBuildHistory.java:14)
    at com.github.grundic.agentPriority.prioritisation.impl.ByBuildStatus.apply(ByBuildStatus.java:100)
    at com.github.grundic.agentPriority.prioritisation.impl.ByBuildStatus.apply(ByBuildStatus.java:45)
    at com.google.common.collect.ByFunctionOrdering.compare(ByFunctionOrdering.java:46)
    at com.google.common.collect.CompoundOrdering.compare(CompoundOrdering.java:43)
    at java.util.TimSort.binarySort(TimSort.java:296)
    at java.util.TimSort.sort(TimSort.java:221)
    at java.util.Arrays.sort(Arrays.java:1438)
    at com.google.common.collect.Ordering.sortedCopy(Ordering.java:685)
    at com.github.grundic.agentPriority.manager.AgentPriorityManager.sort(AgentPriorityManager.java:73)
    at com.github.grundic.agentPriority.buildDistribution.PriorityAgentsFilter.filterAgents(PriorityAgentsFilter.java:71)
    at jetbrains.buildServer.serverSide.impl.buildDistribution.DefaultAgentsFilter.filterAgents(DefaultAgentsFilter.java:27)
    at jetbrains.buildServer.serverSide.impl.QueuedBuildImpl.applyFilters(QueuedBuildImpl.java:15)
    at jetbrains.buildServer.serverSide.impl.QueuedBuildImpl.access$200(QueuedBuildImpl.java:101)
    at jetbrains.buildServer.serverSide.impl.QueuedBuildImpl$1.run(QueuedBuildImpl.java:1)
    at jetbrains.buildServer.serverSide.impl.QueuedBuildImpl$1.run(QueuedBuildImpl.java:2)
    at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAs(SecurityContextImpl.java:7)
    at jetbrains.buildServer.serverSide.impl.auth.SecurityContextImpl.runAsSystem(SecurityContextImpl.java:18)
    at jetbrains.buildServer.serverSide.impl.QueuedBuildImpl.getCanRunOnAgents(QueuedBuildImpl.java:62)
    at sun.reflect.GeneratedMethodAccessor530.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at jetbrains.buildServer.web.jsp.TCBeanELResolver.getValue(TCBeanELResolver.java:62)
    at org.apache.jasper.el.JasperELResolver.getValue(JasperELResolver.java:104)
    at org.apache.el.parser.AstValue.getValue(AstValue.java:184)
    at org.apache.el.parser.AstFunction.getValue(AstFunction.java:122)
    at org.apache.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:184)
    at org.apache.jasper.runtime.PageContextImpl.proprietaryEvaluate(PageContextImpl.java:950)
    at org.apache.jsp.tag.web.queue.queuedBuildAgent_tag._jspx_meth_c_005fset_005f1(queuedBuildAgent_tag.java:187)
    at org.apache.jsp.tag.web.queue.queuedBuildAgent_tag.doTag(queuedBuildAgent_tag.java:121)
    at org.apache.jsp.tag.web.queue.buildQueueList_tag._jspx_meth_queue_005fqueuedBuildAgent_005f0(buildQueueList_tag.java:2107)
    at org.apache.jsp.tag.web.queue.buildQueueList_tag._jspx_meth_c_005fforEach_005f1(buildQueueList_tag.java:1287)
    at org.apache.jsp.tag.web.queue.buildQueueList_tag._jspx_meth_c_005fif_005f3(buildQueueList_tag.java:623)
    at org.apache.jsp.tag.web.queue.buildQueueList_tag.doTag(buildQueueList_tag.java:276)
    at org.apache.jsp.queue.queue_jsp._jspx_meth_queue_005fbuildQueueList_005f0(queue_jsp.java:632)
    at org.apache.jsp.queue.queue_jsp.access$7(queue_jsp.java:622)
    at org.apache.jsp.queue.queue_jsp$Helper.invoke4(queue_jsp.java:785)
    at org.apache.jsp.queue.queue_jsp$Helper.invoke(queue_jsp.java:885)
    at org.apache.jsp.tag.web.refreshable_tag.doTag(refreshable_tag.java:148)
    at org.apache.jsp.queue.queue_jsp._jspx_meth_bs_005frefreshable_005f0(queue_jsp.java:353)
    at org.apache.jsp.queue.queue_jsp.access$3(queue_jsp.java:338)
    at org.apache.jsp.queue.queue_jsp$Helper.invoke3(queue_jsp.java:757)
    at org.apache.jsp.queue.queue_jsp$Helper.invoke(queue_jsp.java:882)
    at org.apache.jsp.tag.web.page_tag.doTag(page_tag.java:776)
    at org.apache.jsp.queue.queue_jsp._jspService(queue_jsp.java:249)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:439)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
    at jetbrains.spring.web.TeamCityJSPServletImpl.access$001(TeamCityJSPServletImpl.java:27)
    at jetbrains.spring.web.TeamCityJSPServletImpl$2.apply(TeamCityJSPServletImpl.java:3)
    at jetbrains.buildServer.util.Util.doUnderContextClassLoader(Util.java:78)
    at jetbrains.spring.web.TeamCityJSPServletImpl.service(TeamCityJSPServletImpl.java:18)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at jetbrains.spring.web.JspController.doHandle(JspController.java:14)
    at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:75)
    at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147)
    at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.processedByMainServlet(TeamCityDispatcherServlet.java:42)
    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:28)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:743)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:485)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:410)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:337)
    at org.springframework.web.servlet.view.InternalResourceView.renderMergedOutputModel(InternalResourceView.java:168)
    at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303)
    at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1246)
    at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1029)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:973)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.processedByMainServlet(TeamCityDispatcherServlet.java:42)
    at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:28)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at jetbrains.buildServer.web.jsp.JspPrecompilerFilter.doFilter(JspPrecompilerFilter.java:6)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at jetbrains.buildServer.web.DisableSessionIdFromUrlFilter.doFilter(DisableSessionIdFromUrlFilter.java:9)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:107)
    at jetbrains.buildServer.diagnostic.web.DiagnosticFilter.doFilter(DiagnosticFilter.java:13)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.web.DependencyParametersCalculationContextFilter.doFilter(DependencyParametersCalculationContextFilter.java:8)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.web.ContentSecurityPolicyFilter.doFilter(ContentSecurityPolicyFilter.java:10)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at jetbrains.buildServer.web.CSRFFilter.doFilter(CSRFFilter.java:100)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:73)
    at jetbrains.buildServer.web.DelegatingFilter.doFilter(DelegatingFilter.java:37)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:1)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1770)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1729)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)