Closed 90K2 closed 7 years ago
At least single upgrade is normal since agent has to download plugins from server. Though constant upgrade may be caused by many reasons.
Please provide full agent logs, they could be found in /opt/buildagent/logs
inside instance.
You could either share them as gist or send to our feedback email.
@VladRassokhin I had same error.
this is my logfile: teamcity-agent.log.zip
Looks like while agent exits for restart (performing upgrade) some other process starts it again, and history repeats
@VladRassokhin Can you help me? Why it happened.
Please attach all log files from this agent.
@pavelsher hey, thanks for your helping, this is the log the team agent generated.
Relevant part of the log: [2017-09-05 15:00:26,925] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/spring-scripting [2017-09-05 15:00:26,925] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/spring-scripting/spring-scripting-bsh.jar [2017-09-05 15:00:26,925] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/spring-scripting/spring-scripting-groovy.jar [2017-09-05 15:00:26,926] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/spring-scripting/spring-scripting-jruby.jar [2017-09-05 15:00:27,927] ERROR - buildServer.agent.LauncherUtil - Error deleting file: /opt/buildagent/lib/spring-scripting [2017-09-05 15:00:27,927] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/spring.jar [2017-09-05 15:00:27,928] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/trove-3.0.3.jar [2017-09-05 15:00:27,929] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/trove4j.jar [2017-09-05 15:00:27,929] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/util.jar [2017-09-05 15:00:27,929] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/xercesImpl.jar [2017-09-05 15:00:27,931] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/xml-rpc-wrapper.jar [2017-09-05 15:00:27,931] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/xmlrpc-2.0.1.jar [2017-09-05 15:00:27,932] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/xpp3-1.1.4c.jar [2017-09-05 15:00:27,932] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/xstream-1.4.8-custom.jar [2017-09-05 15:00:27,932] DEBUG - buildServer.agent.LauncherUtil - Deleting /opt/buildagent/lib/xz-1.5.jar [2017-09-05 15:00:27,933] ERROR - ver.agent.upgrade.AgentUpgrade - Upgrade failed :Problems deleting files under /opt/buildagent/lib. Check logs above for details java.io.IOException: Problems deleting files under /opt/buildagent/lib. Check logs above for details at jetbrains.buildServer.agent.upgrade.UpgradeFolder.doUpgrade(UpgradeFolder.java:58) at jetbrains.buildServer.agent.upgrade.UpgradeFolder.upgrade(UpgradeFolder.java:44) at jetbrains.buildServer.agent.upgrade.AgentUpgrade.doUpgrade(AgentUpgrade.java:110) at jetbrains.buildServer.agent.upgrade.AgentUpgrade.applyUpdates(AgentUpgrade.java:82) at jetbrains.buildServer.agent.upgrade.UpgradeRunBase.run(UpgradeRunBase.java:41) at jetbrains.buildServer.agent.upgrade.UpgradeMode$5.run(UpgradeMode.java:133) at jetbrains.buildServer.agent.upgrade.Upgrade2$1.lockHeld(Upgrade2.java:57) at jetbrains.buildServer.agent.upgrade.Upgrade2$1.lockHeld(Upgrade2.java:55) at jetbrains.buildServer.agent.lock.impl.misc.MockLock.assumeLocked(MockLock.java:22) at jetbrains.buildServer.agent.lock.impl.misc.MockLock.tryLock(MockLock.java:14) at jetbrains.buildServer.agent.upgrade.Upgrade2.main3(Upgrade2.java:54) at jetbrains.buildServer.agent.upgrade.Upgrade2.main2(Upgrade2.java:27) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at jetbrains.buildServer.agent.ClazzLoader.call(ClazzLoader.java:59) at jetbrains.buildServer.agent.ClazzLoader.callMain2(ClazzLoader.java:19) at jetbrains.buildServer.agent.upgrade.Upgrade.main(Upgrade.java:16)
So for some reason agent was not able to remove /opt/buildagent/lib/spring-scripting directory during the upgrade and this caused infinite loop. We're not sure how to reproduce it yet. If there is anything special about environment where you start the agent, please let us know.
@pavelsher this is my docker info:
[root@docker 2017-09-11]# docker info
Containers: 7
Running: 6
Paused: 0
Stopped: 1
Images: 75
Server Version: 17.06.2-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.26.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 17.06GiB
Name: docker.whatakitty.com
ID: 7MWP:LUVR:HVLD:ER24:5MBF:BVRI:ACCK:LBO7:6VMX:FLPC:YGQR:HJVV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://t4xli0ji.mirror.aliyuncs.com/
Live Restore Enabled: false
WARNING: overlay: the backing xfs filesystem is formatted without d_type support, which leads to incorrect behavior.
Reformat the filesystem with ftype=1 to enable d_type support.
Running without d_type support will not be supported in future releases.
the agent start command is :
teamcity-build-agent:
image: jetbrains/teamcity-agent
restart: always
depends_on:
- teamcity-server
environment:
- SERVER_URL=http://teamcity-server:8111
networks:
- teamcity_net
volumes:
- /var/teamcity/agent/conf:/data/teamcity_agent/conf
We reproduced problem on Cent OS: https://youtrack.jetbrains.com/issue/TW-51501
So the problem is not TeamCity specific, seems there is some bug in CentOS itself. Some related discussions: https://github.com/moby/moby/issues/27214 It looks like newer version of CentOS should not have this problem. Also reportedly problem does not reproduce in case of ext4 file system. Anyway, there is nothing to fix on our side.
For anyone who runs into this... Docker tries to use the default storage driver ("overlay2", or "overlay") If you are stuck in an upgrade loop with issue's on deleting files you can try changing the storage driver for docker to either vfs or devicemapper, but vfs is recommended over devicemapper, since it is known to be slow. as Per: https://docs.docker.com/engine/userguide/storagedriver/selectadriver/#docker-ee-and-cs-engine
Another option, as noted in this post: https://github.com/moby/moby/issues/31445 referencing this: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html
"this problem occurs if d_type is not available, due to the backing xfs filesystem not being formatted with ftype=1"
Yes it is reproducible in Cent OS
Hello. I have some problem with starting buildAgent. Inside the container begins infinity loop after start.
That is how i start container:
docker run -itd -e SERVER_URL="http://teamcity.domain.com" -v /opt/tc_agent/:/data/teamcity_agent/conf -p 0.0.0.0:9090:9090 jetbrains/teamcity-minimal-agent
Seems that each loop cycle finishing with
And next lines again from begin
What i'm doing wrong? Thanks