[JENKINS-13202] Artifact archiving from an ssh slave fails if symlinks are present

[timja]

When archiving artifacts from a job executed on a "Unix via SSH" slave, if a symlink is present and archived before its target, the archiving will fail. It looks like the master is trying to chmod the local symlink (because it has executable permissions stored in the tar used to do the remote copy), but that fails because the target doesn't exist yet, and the exception aborts the archiving. I think the solution is to just not chmod symlinks, since they don't have modes of their own. This was a regression from 1.455 to 1.456. I suspect the fixes for JENKINS-9118 are the cause.

To reproduce this problem, install the Jenkins master on a Linux machine and add a "Unix via SSH" slave that is also a Linux machine. Create a job with the following build script:

rm -rf stuff
mkdir stuff
cd stuff
touch zzfile
ln -s zzfile aafile
ln -s zzfile bbfile

Restrict this project to build on the slave. Set up archiving for "stuff/*". Run the job. The job will complete successfully, but archiving will abort partway though with a stack trace in the console.

Archiving artifacts
ERROR: Failed to archive artifacts: stuff/*
hudson.util.IOException2: Failed to extract /tmp/jenkins-slave-home/workspace/create_symlinks/stuff/*
    at hudson.FilePath.readFromTar(FilePath.java:1817)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:1729)
    at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
    at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:678)
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:656)
    at hudson.model.Build$RunnerImpl.post2(Build.java:162)
    at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:625)
    at hudson.model.Run.run(Run.java:1435)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:238)
Caused by: java.io.IOException: Failed to chmod /tmp/jenkins-home/jobs/create_symlinks/builds/2012-03-21_16-53-15/archive/stuff/aafile : No such file or directory
    at hudson.FilePath._chmod(FilePath.java:1248)
    at hudson.FilePath.readFromTar(FilePath.java:1813)
    ... 12 more

---

Originally reported by dreiss, imported from: Artifact archiving from an ssh slave fails if symlinks are present

• status: Reopened
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[timja]

leander:

We've seen this issue here, also coincident with upgrading from 1.455 to 1.456:

ERROR: Failed to archive artifacts: Engineering/iphone/build/**
hudson.util.IOException2: hudson.util.IOException2: Failed to extract /hudson/workspace/project-ios/Engineering/iphone/build/**
    at hudson.FilePath.readFromTar(FilePath.java:1817)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:1729)
    at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
    at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:678)
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:656)
    at hudson.model.Build$RunnerImpl.post2(Build.java:162)
    at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:625)
    at hudson.model.Run.run(Run.java:1435)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:238)
Caused by: java.io.IOException: Failed to chmod /var/empty/.hudson/jobs/project-ios/builds/2012-03-23_11-12-38/archive/Engineering/iphone/build/Debug-iphoneos/Payload/Project.app/CodeResources : No such file or directory
    at hudson.FilePath._chmod(FilePath.java:1248)
    at hudson.FilePath.readFromTar(FilePath.java:1813)
    ... 12 more

    at hudson.FilePath.copyRecursiveTo(FilePath.java:1736)
    at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
    at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:678)
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:656)
    at hudson.model.Build$RunnerImpl.post2(Build.java:162)
    at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:625)
    at hudson.model.Run.run(Run.java:1435)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:238)

[timja]

njamesp:

We have seen this too when we upgraded to 1.456.

[timja]

njamesp:

This might be related to JENKINS-13280.

[timja]

bsm_ghs:

I would agree. Please see my suggested patch in JENKINS-13280.

[timja]

brenuart:

The same problem appears during installation of tools like JDK or Maven packaged as tar.gz on a remote Unix slave. The deployment process fails with the same exception.
I tend to agree with the above description:

• the symlink is extracted from the tarball before its target;
• exception is thrown when deployment process tries to chmod the symlink;

I also agree with the proposed solution: there is no point in trying to chmod a symlink.

The root of the issue seems to be indeed located in the FilePath class and, as the number of linked issue suggest, tend to affect many plugins or different areas of Jenkins.

Are there any plans to fix this issue in the near future?
Is someone working on it?
Can we help somehow ?

[timja]

dreiss:

I created a pull request for this, including a trivial change to the original JENKINS-9118 test to expose the bug.

https://github.com/jenkinsci/jenkins/pull/469

[timja]

scm_issue_link:

Code changed in jenkins
User: David Reiss
Path:
changelog.html
core/src/main/java/hudson/FilePath.java
core/src/test/java/hudson/FilePathTest.java
http://jenkins-ci.org/commit/jenkins/e15b2e19e394f5d63183f01a2e72a14115a0c370
Log:
[FIXED JENKINS-13202] Don't set mtime or mode on symlinks

Previously, the untar code tries to set the last modified time and mode
on every untarred file. However, if the tar contains a broken symlink,
or a symlink that points to a file that has not been untarred yet, the
time/mode setting would fail on the broken symlink.

Symlinks don't have meaningful modified times or modes of their own, so
only set these values on non-symlinks.

Rename the file "a" in the test to expose the bug.

[timja]

dogfood:

Integrated in jenkins_main_trunk #1710
[FIXED JENKINS-13202] Don't set mtime or mode on symlinks (Revision e15b2e19e394f5d63183f01a2e72a14115a0c370)

Result = SUCCESS
Kohsuke Kawaguchi : e15b2e19e394f5d63183f01a2e72a14115a0c370
Files :

• changelog.html
• core/src/main/java/hudson/FilePath.java
• core/src/test/java/hudson/FilePathTest.java

[timja]

erwan_q:

We have a regression on 1.464 version and don't know if it's related to this commit. It's recent.
We try to archive a tar file in a matrix project. We have today an issue on AIX platform with the following message

[tar] Building tar: /database/products/ci/workspace/ida_trunk/label/aix/repository/ida.tar.gz

BUILD SUCCESSFUL
Total time: 11 minutes 56 seconds
[WARNINGS] Parsing warnings in console log with parser GNU compiler 4 (gcc)
[WARNINGS] Computing warning deltas based on reference build #227
Archiving artifacts
ERROR: Failed to archive artifacts: repository/ida.tar.gz
hudson.util.IOException2: java.lang.UnsupportedOperationException
at hudson.FilePath.copyRecursiveTo(FilePath.java:1752)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:710)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:685)
at hudson.model.Build$RunnerImpl.post2(Build.java:162)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:632)
at hudson.model.Run.run(Run.java:1459)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:239)
Caused by: java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException
at hudson.remoting.Channel$3.adapt(Channel.java:679)
at hudson.remoting.Channel$3.adapt(Channel.java:674)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1750)
... 10 more
Caused by: java.lang.UnsupportedOperationException
at hudson.os.PosixAPI$1.getCurrentWorkingDirectory(PosixAPI.java:59)
at org.jruby.ext.posix.util.ExecIt.run(ExecIt.java:59)
at org.jruby.ext.posix.util.ExecIt.runAndWait(ExecIt.java:51)
at org.jruby.ext.posix.JavaLibCHelper.readlink(JavaLibCHelper.java:196)
at org.jruby.ext.posix.JavaPOSIX.readlink(JavaPOSIX.java:160)
at hudson.Util.resolveSymlink(Util.java:1067)
at hudson.Util.resolveSymlink(Util.java:1030)
at hudson.util.DirScanner$Glob.scan(DirScanner.java:115)
at hudson.FilePath.writeToTar(FilePath.java:1788)
at hudson.FilePath.access$1000(FilePath.java:166)
at hudson.FilePath$36.invoke(FilePath.java:1729)
at hudson.FilePath$36.invoke(FilePath.java:1726)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2161)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:736)

[timja]

erwan_q:

Ok this bug is in the 1.465 version who will be released soon.

[timja]

erwan_q:

Sorry I still reproduce this issue with 1.465
Something is broken in the artifact archiving since few releases now.
My case to reproduce: matrix project
Simple artifact to archive (1 tgz file)
multi platform matrix: solaris, linux, aix
And it's broken ONLY for aix.

[timja]

erwan_q:

My issue is related to the duplicate JENKINS-13241
And the complete stack is:
[tar] Building tar: /database/products/ci/workspace/ida_trunk/label/aix/repository/ida.tar.gz

BUILD SUCCESSFUL
Total time: 11 minutes 9 seconds
[WARNINGS] Parsing warnings in console log with parser GNU compiler 4 (gcc)
[WARNINGS] Computing warning deltas based on reference build #154
Archiving artifacts
ERROR: Failed to archive artifacts: repository/ida.tar.gz
hudson.util.IOException2: java.lang.UnsupportedOperationException
at hudson.FilePath.copyRecursiveTo(FilePath.java:1768)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:710)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:685)
at hudson.model.Build$RunnerImpl.post2(Build.java:162)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:632)
at hudson.model.Run.run(Run.java:1459)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:239)
Caused by: java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException
at hudson.remoting.Channel$3.adapt(Channel.java:679)
at hudson.remoting.Channel$3.adapt(Channel.java:674)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1766)
... 10 more
Caused by: java.lang.UnsupportedOperationException
at hudson.os.PosixAPI$1.getCurrentWorkingDirectory(PosixAPI.java:59)
at org.jruby.ext.posix.util.ExecIt.run(ExecIt.java:59)
at org.jruby.ext.posix.util.ExecIt.runAndWait(ExecIt.java:51)
at org.jruby.ext.posix.JavaLibCHelper.readlink(JavaLibCHelper.java:196)
at org.jruby.ext.posix.JavaPOSIX.readlink(JavaPOSIX.java:160)
at hudson.Util.resolveSymlink(Util.java:1067)
at hudson.Util.resolveSymlink(Util.java:1030)
at hudson.util.DirScanner$Glob.scan(DirScanner.java:115)
at hudson.FilePath.writeToTar(FilePath.java:1804)
at hudson.FilePath.access$1000(FilePath.java:166)
at hudson.FilePath$36.invoke(FilePath.java:1745)
at hudson.FilePath$36.invoke(FilePath.java:1742)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2177)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:736)

[timja]

smorriso:

I see the same thing on AIX, and also on HP. I also run a matrix job, and the Linux and Sun part of that matrix is fine, but the HP and AIX axises fail with that exact same stack trace.

[timja]

dreiss:

It looks like 00cf9f43 added a stub implementation of POSIX that doesn't implement getCurrentWorkingDirectory. This is only used on platforms where JNA isn't working (see Util.resolveSymlink).

[timja]

scm_issue_link:

Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
core/src/main/java/hudson/os/PosixAPI.java
core/src/main/java/hudson/util/DirScanner.java
http://jenkins-ci.org/commit/jenkins/95c1728c05b11a55e72c8ed8240d2852528ea1db
Log:
[FIXED JENKINS-13202] fixed a regression in untar on exotic platforms.

Fall back in non-JNA case wasn't working.

[timja]

smorriso:

I took the 1.468 snapshot jar (Jenkins ver. 1.468-SNAPSHOT (private-05/31/2012 01:16 GMT-jenkins)), and this appears to be broken still, just a bit further on. Still just on AIX and HP:

hudson.util.IOException2: java.io.IOException: Cannot run program "readlink" (in directory "/u01/svxx/cbxxxn/jenkins/."): readlink: not found
at hudson.FilePath.copyRecursiveTo(FilePath.java:1771)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:709)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:684)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:631)
at hudson.model.Run.execute(Run.java:1481)
at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:239)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Cannot run program "readlink" (in directory "/u01/svxx/cbxxxn/jenkins/."): readlink: not found
at hudson.remoting.Channel$3.adapt(Channel.java:679)
at hudson.remoting.Channel$3.adapt(Channel.java:674)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1769)
... 10 more
Caused by: java.io.IOException: Cannot run program "readlink" (in directory "/u01/svxx/cbxxxn/jenkins/."): readlink: not found
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at java.lang.Runtime.exec(Runtime.java:593)
at org.jruby.ext.posix.util.ExecIt.run(ExecIt.java:61)
at org.jruby.ext.posix.util.ExecIt.runAndWait(ExecIt.java:51)
at org.jruby.ext.posix.JavaLibCHelper.readlink(JavaLibCHelper.java:196)
at org.jruby.ext.posix.JavaPOSIX.readlink(JavaPOSIX.java:160)
at hudson.Util.resolveSymlink(Util.java:1067)
at hudson.util.DirScanner$Glob.scan(DirScanner.java:115)
at hudson.FilePath.writeToTar(FilePath.java:1807)
at hudson.FilePath.access$1000(FilePath.java:166)
at hudson.FilePath$36.invoke(FilePath.java:1748)
at hudson.FilePath$36.invoke(FilePath.java:1745)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2180)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: readlink: not found
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:102)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 21 more

[timja]

dogfood:

Integrated in jenkins_ui-changes_branch #30
[FIXED JENKINS-13202] Don't set mtime or mode on symlinks (Revision e15b2e19e394f5d63183f01a2e72a14115a0c370)
[FIXED JENKINS-13202] fixed a regression in untar on exotic platforms. (Revision 95c1728c05b11a55e72c8ed8240d2852528ea1db)

Result = SUCCESS
Kohsuke Kawaguchi : e15b2e19e394f5d63183f01a2e72a14115a0c370
Files :

• core/src/main/java/hudson/FilePath.java
• changelog.html
• core/src/test/java/hudson/FilePathTest.java

Kohsuke Kawaguchi : 95c1728c05b11a55e72c8ed8240d2852528ea1db
Files :

• core/src/main/java/hudson/util/DirScanner.java
• core/src/main/java/hudson/os/PosixAPI.java
• changelog.html

[timja]

georgetk:

Still failing on AIX, access permisions are OK for the user with which the slave is running, readlink is missing by default on the AIX platform.

FATAL: HTML Publisher failure
hudson.util.IOException2: java.io.IOException: Cannot run program "readlink" (in directory "/opt/wcstst/hudson/."): error=13, The file access permissions do not allow the specified action.
at hudson.FilePath.copyRecursiveTo(FilePath.java:1771)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1663)
at htmlpublisher.HtmlPublisher.perform(HtmlPublisher.java:212)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:692)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:639)
at hudson.model.Run.execute(Run.java:1485)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:239)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Cannot run program "readlink" (in directory "/opt/wcstst/hudson/."): error=13, The file access permissions do not allow the specified action.
at hudson.remoting.Channel$3.adapt(Channel.java:679)
at hudson.remoting.Channel$3.adapt(Channel.java:674)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1769)
... 11 more
Caused by: java.io.IOException: Cannot run program "readlink" (in directory "/opt/wcstst/hudson/."): error=13, The file access permissions do not allow the specified action.
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:605)
at org.jruby.ext.posix.util.ExecIt.run(ExecIt.java:61)
at org.jruby.ext.posix.util.ExecIt.runAndWait(ExecIt.java:51)
at org.jruby.ext.posix.JavaLibCHelper.readlink(JavaLibCHelper.java:196)
at org.jruby.ext.posix.JavaPOSIX.readlink(JavaPOSIX.java:160)
at hudson.Util.resolveSymlink(Util.java:1067)
at hudson.util.DirScanner$Glob.scan(DirScanner.java:115)
at hudson.FilePath.writeToTar(FilePath.java:1807)
at hudson.FilePath.access$1000(FilePath.java:166)
at hudson.FilePath$36.invoke(FilePath.java:1748)
at hudson.FilePath$36.invoke(FilePath.java:1745)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2180)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:315)
at java.util.concurrent.FutureTask.run(FutureTask.java:150)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:898)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:920)
at java.lang.Thread.run(Thread.java:736)
Caused by: java.io.IOException: error=13, The file access permissions do not allow the specified action.
at java.lang.UNIXProcess.(UNIXProcess.java:53)
at java.lang.ProcessImpl.start(ProcessImpl.java:101)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 21 more
TestNG Reports Processing: START

[timja]

smorriso:

Still broken

[timja]

reechard:

@George @Stephen, exacly. "readlink" is missing on AIX and HP-UX platforms, as of Jenkins 1.471.

[timja]

scarytom:

with regard to "readlink missing on AIX etc." I've added a stopgap fix in this pull request. This doesn't represent a final solution, but might take the heat off things a bit.

https://github.com/jenkinsci/jenkins/pull/547

[timja]

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/util/DirScanner.java
core/src/main/java/hudson/util/FileVisitor.java
http://jenkins-ci.org/commit/jenkins/f50316b66a8e4761193c46f824fbd620ffccd6c0
Log:
JENKINS-13202 Amelioration of problem: if the platform cannot read symlinks, fall back to visiting them as plain files or directories.

[timja]

dogfood:

Integrated in jenkins_main_trunk #1896
JENKINS-13202 Amelioration of problem: if the platform cannot read symlinks, fall back to visiting them as plain files or directories. (Revision f50316b66a8e4761193c46f824fbd620ffccd6c0)

Result = SUCCESS
Jesse Glick : f50316b66a8e4761193c46f824fbd620ffccd6c0
Files :

• core/src/main/java/hudson/util/FileVisitor.java
• core/src/main/java/hudson/util/DirScanner.java

[timja]

jglick:

Anyone running AIX or HP-UX with some time on their hands, please test the above workaround, and test https://github.com/jenkinsci/jenkins/pull/559 if you are running Java 7.

[timja]

reechard:

@jglick Jesse, nice work. The job I have set up to track this issue runs successfully with your patch, on both AIX and HP-UX. Thanks!

CONSOLE OUTPUT:

Started by upstream project "jenkins-sanity-check" build number 2
Building remotely on aix.aix53 in workspace /home/hudson/edge/jenkins/workspace/jenkins-sanity-check/label/pkg_aix32

Deleting project workspace... done

[pkg_aix32] $ /bin/sh -xe /tmp/hudson234475666077177712.sh
+ 1> env.txt
+ tar cf test-pkg_aix32.tar env.txt
+ gzip test-pkg_aix32.tar
Archiving artifacts
Finished: SUCCESS
Page generated: Sep 8, 2012 10:57:13 PMREST APIJenkins ver. 1.482-SNAPSHOT (private-09/06/2012 15:32 GMT-jenkins)

[timja]

jglick:

Merged in 646154f.

[timja]

dogfood:

Integrated in jenkins_main_trunk #1910
JENKINS-13202 Noting. (Revision c5c55fe635031acf45d7fbc5f1d9418bc77f59ba)

Result = UNSTABLE
Jesse Glick : c5c55fe635031acf45d7fbc5f1d9418bc77f59ba
Files :

• changelog.html

[timja]

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
http://jenkins-ci.org/commit/jenkins/c5c55fe635031acf45d7fbc5f1d9418bc77f59ba
Log:
JENKINS-13202 Noting.

[timja]

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/src/main/java/hudson/Util.java
http://jenkins-ci.org/commit/jenkins/646154f18d8a44e9e99095f8af81e4a0923dcd9b
Log:
Merge pull request #559 from jglick/symlinks-JENKINS-13202

[FIXED JENKINS-13202] Try to use NIO.2 (Java 7) methods to work with symlinks.

Compare: https://github.com/jenkinsci/jenkins/compare/d73404d0d592...646154f18d8a

[timja]

jglick:

@reechard, @scarytom, others: fix should be in 1.483, please check when you can.

[timja]

erwan_q:

Can you confirm this fix is working with java 1.6 too?
If not do we have to migrate the master or the slave where the process is running on java 1.7?
I have an old slave AIX 5.3 and may be do not support java 1.7...
If the fix impact the master, no problem.

[timja]

reechard:

@erwan_q yes this fix works with java 6. I do not have java 7
On my HPUX OR AIX boxes.

[timja]

jglick:

@erwan_q: the full fix should have no effect on Java 5/6; it only takes effect on Java 7+. However the earlier f50316b should at least avoid the worst symptoms.

[timja]

nnau:

Still can't archive artifacts on ia64 slave.
Now the jobs just hangs on 'Archiving artifacts' step.
Using v1.487

[timja]

schultemarkus:

I can confirm to have still problems with 'Archiving artifacts' (hanging), see #15301.

[timja]

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/util/DirScanner.java
core/src/main/java/hudson/util/FileVisitor.java
http://jenkins-ci.org/commit/jenkins/af9977c1b28c2e212f707ab88b4f2ac561e37e50
Log:
JENKINS-13202 Amelioration of problem: if the platform cannot read symlinks, fall back to visiting them as plain files or directories.(cherry picked from commit f50316b66a8e4761193c46f824fbd620ffccd6c0)

[timja]

nnau:

didn't fix the problem.
I installed 1.488 that looks like contains the fix - but the artifacts can't be archived all the same.

[timja]

jglick:

@nnau: you may be seeing an unrelated problem. Capture thread dumps and file separately with as much information to reproduce as you can muster.

[timja]

bdellegrazie:

@Jesse Glick and others.
I've applied the patch on AIX-6.1 slave (Windows master) Running 1.466.3-SNAPSHOT (i.e. 1.466.2 patched)
http://jenkins-ci.org/commit/jenkins/af9977c1b28c2e212f707ab88b4f2ac561e37e50

However the problem is still occurring. I've got a test job which simply attempts to archive dummy stuff. the result is:

[test-archive-artefacts-aix] $ /bin/bash -ex /home/apps/DP/jenkins/tmp/hudson2194465448671283236.sh
+ mkdir -p a/b a/c dist
+ touch a/1.txt a/b/2.txt
+ ln -sf a/1.txt a/c/3.txt
+ /opt/freeware/bin/tar -czf dist/test.tar.gz a/
Archiving artifacts
ERROR: Failed to archive artifacts: dist/*.tar.gz
hudson.util.IOException2: java.lang.UnsupportedOperationException
at hudson.FilePath.copyRecursiveTo(FilePath.java:1784)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:710)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:685)
at hudson.model.Build$RunnerImpl.post2(Build.java:162)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:632)
at hudson.model.Run.run(Run.java:1463)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:239)
Caused by: java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException
at hudson.remoting.Channel$4.adapt(Channel.java:696)
at hudson.remoting.Channel$4.adapt(Channel.java:691)
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:55)
at hudson.FilePath.copyRecursiveTo(FilePath.java:1782)
... 10 more
Caused by: java.lang.UnsupportedOperationException
at hudson.os.PosixAPI$1.getCurrentWorkingDirectory(PosixAPI.java:59)
at org.jruby.ext.posix.util.ExecIt.run(ExecIt.java:59)
at org.jruby.ext.posix.util.ExecIt.runAndWait(ExecIt.java:51)
at org.jruby.ext.posix.JavaLibCHelper.readlink(JavaLibCHelper.java:196)
at org.jruby.ext.posix.JavaPOSIX.readlink(JavaPOSIX.java:160)
at hudson.Util.resolveSymlink(Util.java:1067)
at hudson.util.DirScanner$Glob.scan(DirScanner.java:121)
at hudson.FilePath.writeToTar(FilePath.java:1820)
at hudson.FilePath.access$1000(FilePath.java:166)
at hudson.FilePath$36.invoke(FilePath.java:1761)
at hudson.FilePath$36.invoke(FilePath.java:1758)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2193)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at hudson.remoting.Engine$1$1.run(Engine.java:60)
at java.lang.Thread.run(Thread.java:736)
Finished: SUCCESS

Looks like its failing to find 'readlink'

[timja]

jglick:

@bdellegrazie: for 1.466.x you also need to backport 95c1728.

[timja]

bdellegrazie:

@Jesse Glick and others.

I can confirm that with both patches applied, 1.466.x works successfully on AIX 6.1 with IBM Java 6.
Thanks very much for your help!
Any chance of a LTS build with these patches applied? Or if not, can we ensure that they are included in the next LTS release please?

[timja]

jglick:

@bdellegrazie: I know of no particular process for nominating these patches for a hypothetical 1.466.3. But 1.480.1, the next planned LTS, will have both af9977c and 95c1728 applied (the former in its branch, the latter because that was committed prior to 1.480).

As someone running on a relatively unusual platform (which cannot to my knowledge be easily tested by other Jenkins developers in e.g. VirtualBox), you are urged to help test the LTS release candidates: https://wiki.jenkins-ci.org/display/JENKINS/LTS+Release+Line

I think 1.480.1 RC was built before af9977c was cherry-picked, so ask on the dev list before using the RC, or build your own from the stable branch.

[timja]

nnau:

@Jesse Glick file a separate issue: https://issues.jenkins-ci.org/browse/JENKINS-15682
hope it will help, thank you!

[timja]

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/src/main/java/hudson/Util.java
core/src/test/java/hudson/UtilTest.java
http://jenkins-ci.org/commit/jenkins/0552f5ad75da1468050f8aa3ffb533bd3c5d4ca8
Log:
JENKINS-13202 646154f refinement: resolveSymlink is documented to return null, not throw an exception, when called on a regular file.

[timja]

dogfood:

Integrated in jenkins_main_trunk #2079
JENKINS-13202 646154f refinement: resolveSymlink is documented to return null, not throw an exception, when called on a regular file. (Revision 0552f5ad75da1468050f8aa3ffb533bd3c5d4ca8)

Result = SUCCESS
Jesse Glick : 0552f5ad75da1468050f8aa3ffb533bd3c5d4ca8
Files :

• core/src/test/java/hudson/UtilTest.java
• core/src/main/java/hudson/Util.java
• changelog.html

[timja]

chrisgwarp:

Hi All.
Re the readlink issue 9again).
Jenkins 1.510 on AIX 7.1, under WAS 8.0 (IBM JDK 1.6) I am seeing lots of:

[5/17/13 15:38:10:948 EST] 0000005e PeepholePerma W Failed to update hudson.maven.MavenModuleSet@9565b347[websphere-scripts] lastSuccessfulBuild permalink for websphere-scripts #92
java.io.IOException: Cannot run program "readlink" (in directory "/var/WebSphere/dumps/."): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:471)
at java.lang.Runtime.exec(Runtime.java:604)
at org.jruby.ext.posix.util.ExecIt.run(ExecIt.java:61)
at org.jruby.ext.posix.util.ExecIt.runAndWait(ExecIt.java:51)
at org.jruby.ext.posix.JavaLibCHelper.readlink(JavaLibCHelper.java:196)
at org.jruby.ext.posix.JavaPOSIX.readlink(JavaPOSIX.java:160)
at hudson.Util.resolveSymlink(Util.java:1236)
at jenkins.model.PeepholePermalink.updateCache(PeepholePermalink.java:146)
at jenkins.model.PeepholePermalink.resolve(PeepholePermalink.java:118)
at hudson.model.Job.getLastSuccessfulBuild(Job.java:802)
at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
at org.kohsuke.stapler.jelly.groovy.JellyBuilder.doInvokeMethod(JellyBuilder.java:276)
at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invoke(Namespace.java:92)
at $Proxy55.projectView(Unknown Source)
at lib.JenkinsTagLib$projectView.call(Unknown Source)
at hudson.model.View.main.run(main.groovy:14)
at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:69)
at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:62)
at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:112)
at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:666)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:736)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583)
at org.kohsuke.stapler.Stapler.service(Stapler.java:214)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1224)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:774)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:456)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:895)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1690)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:53)
at java.lang.ProcessImpl.start(ProcessImpl.java:112)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:464)
... 136 more

Is this expected?

If it is, then we have a large performance problem. We currently only have 43 jobs defined, but it is taking well over 2 minutes (!!!) to build the main page. Given the number of log entries that it's logging, I'm only left wondering if this is the root cause of the performance issue.
Other ajax type of requests are being responded too in <250mSec.

[timja]

jglick:

@chrisgwarp your issue is something else. Please read this thread and if necessary open a separate issue.

[timja]

matthiasfraass:

Is there a regression?
We're running Jenkins 1.514 under JDK6 and HP/UX and this error pops up - I think after a recent update to 1.514:

May 17, 2013 9:57:58 AM hudson.model.RunMap retrieve
WARNING: could not load /proj/halden/tools/hudson/jobs/SOMEJOB/builds/412
java.io.IOException: Cannot run program "readlink" (in directory "/opt/projekte/halden/tools/hudson/."): readlink
: not found
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at org.jruby.ext.posix.util.ExecIt.run(ExecIt.java:61)
at org.jruby.ext.posix.util.ExecIt.runAndWait(ExecIt.java:51)
at org.jruby.ext.posix.JavaLibCHelper.readlink(JavaLibCHelper.java:196)
at org.jruby.ext.posix.JavaPOSIX.readlink(JavaPOSIX.java:160)
at hudson.Util.resolveSymlink(Util.java:1236)
at hudson.Util.resolveSymlinkToFile(Util.java:1166)
at hudson.model.Run.parseTimestampFromBuildDir(Run.java:346)
at hudson.model.Run.(Run.java:294)
at hudson.model.AbstractBuild.(AbstractBuild.java:184)
at hudson.maven.AbstractMavenBuild.(AbstractMavenBuild.java:54)
at hudson.maven.MavenModuleSetBuild.(MavenModuleSetBuild.java:135)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at hudson.model.AbstractProject.loadBuild(AbstractProject.java:1122)
at hudson.model.AbstractProject$1.create(AbstractProject.java:322)
at hudson.model.AbstractProject$1.create(AbstractProject.java:320)
at hudson.model.RunMap.retrieve(RunMap.java:225)
at hudson.model.RunMap.retrieve(RunMap.java:59)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:667)
at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:629)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:368)
at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:526)
at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:379)
at jenkins.model.lazy.AbstractLazyLoadRunMap.newestBuild(AbstractLazyLoadRunMap.java:321)
at hudson.model.AbstractProject.getLastBuild(AbstractProject.java:1059)
at hudson.maven.AbstractMavenProject.createTransientActions(AbstractMavenProject.java:184)
at hudson.maven.MavenModuleSet.createTransientActions(MavenModuleSet.java:454)
...skipping...
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: readlink: not found
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.(UNIXProcess.java:147)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 133 more

[timja]

jglick:

@matthiasfraass if there is a regression you or someone with access to an HP/UX machine is going to have to diagnose it. Apparently not only can glibc not be loaded but /usr/bin/readlink is absent as well. Try running UtilTest on your machine, run a debugger, etc.

I suspect a fix of JENKINS-14351 (using JNR) would address this issue. But the recommendation remains to use Java 7 which has a supported way of dealing with symlinks and which Jenkins will use preferentially.

[timja]

chrisgwarp:

(I really wish this issue would be renamed: issues with missing readlink, as it appears to be the root cause of a number of issues).

I'm running this under WAS 8.0 with the IBM 1.6 JDK - which is not something that I can swap out; it is simply not supported.

I am getting massive performance issues (2 minutes to generate the front page with only 43 jobs) and the SystemOut.log is filled with a trace similar to the above.

I still do not understand the underlying need to differentiate between a real file and a sym link, and hence the need for readlink.

Can not some startup logic be used to test for the presence of readlink and record it's presence and then not attempt to use it if not present? Would certainly stop my logs filling 400Mb ever 2 days.