[JENKINS-13202] Artifact archiving from an ssh slave fails if symlinks are present

[timja]

When archiving artifacts from a job executed on a "Unix via SSH" slave, if a symlink is present and archived before its target, the archiving will fail. It looks like the master is trying to chmod the local symlink (because it has executable permissions stored in the tar used to do the remote copy), but that fails because the target doesn't exist yet, and the exception aborts the archiving. I think the solution is to just not chmod symlinks, since they don't have modes of their own. This was a regression from 1.455 to 1.456. I suspect the fixes for JENKINS-9118 are the cause.

To reproduce this problem, install the Jenkins master on a Linux machine and add a "Unix via SSH" slave that is also a Linux machine. Create a job with the following build script:

rm -rf stuff
mkdir stuff
cd stuff
touch zzfile
ln -s zzfile aafile
ln -s zzfile bbfile

Restrict this project to build on the slave. Set up archiving for "stuff/*". Run the job. The job will complete successfully, but archiving will abort partway though with a stack trace in the console.

Archiving artifacts
ERROR: Failed to archive artifacts: stuff/*
hudson.util.IOException2: Failed to extract /tmp/jenkins-slave-home/workspace/create_symlinks/stuff/*
    at hudson.FilePath.readFromTar(FilePath.java:1817)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:1729)
    at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:116)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
    at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:678)
    at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:656)
    at hudson.model.Build$RunnerImpl.post2(Build.java:162)
    at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:625)
    at hudson.model.Run.run(Run.java:1435)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
    at hudson.model.ResourceController.execute(ResourceController.java:88)
    at hudson.model.Executor.run(Executor.java:238)
Caused by: java.io.IOException: Failed to chmod /tmp/jenkins-home/jobs/create_symlinks/builds/2012-03-21_16-53-15/archive/stuff/aafile : No such file or directory
    at hudson.FilePath._chmod(FilePath.java:1248)
    at hudson.FilePath.readFromTar(FilePath.java:1813)
    ... 12 more

---

Originally reported by dreiss, imported from: Artifact archiving from an ssh slave fails if symlinks are present

• status: Reopened
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[timja]

matthiasfraass:

@Chris this is exactly our problem:

The jobs themselves are fine - it's the huge amount of log messages (40MB within 2 hours) which are annoying and seem to make loading of the dashboard very slow.

[timja]

jglick:

@chrisgwarp:

Can not some startup logic be used to test for the presence of readlink and record it's presence and then not attempt to use it if not present?

Probably; file a pull request. Only someone with AIX can test.

[timja]

chrisgwarp:

I don't get the pull request bit (not a git person).

However, I have more than enough AIX hosts to test on; and more than willing too!

[timja]

chrisgwarp:

AIX does have a readlink program, it's a part of the coreutils RPM package. It's located in /usr/freeware/bin, but by default, it does not get added to the default paths, so you'll need to do that yourself.

Here are some hard timing facts:

Jenkins 1.510:
<>
[10:36:46.966] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 98563ms]
[10:39:50.486] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 96515ms]

<>
[10:58:11.697] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 43375ms]
[10:59:11.871] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 37594ms]
[11:00:20.580] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 40219ms]

jenkins 1.500
<>
[11:30:52.488] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 7953ms]
[11:32:05.379] GET http://10.225.0.69/jenkins/ [HTTP/1.1 200 OK 4609ms]

For 45ish jobs, 40 secs to generate the front page is not good. And as can be seen above, the presence (or lack) of presence of readlink is a major contributor to the performance problems.

Has anyone opened a perf issue on the recent builds?

[timja]

jglick:

@chrisgwarp et al.: commenting on timing is fine but it does not help Jenkins developers make any progress since, again, we cannot test for you on AIX. You need to evaluate whether pull #770 makes a difference, e.g. by running https://jenkins.ci.cloudbees.com/job/core/job/jenkins_main_trunk/857/artifact/war/target/jenkins.war built from that branch.

[timja]

ntshako:

same problem here:
1.531

[timja]

pjos:

What is the status of this problem. Getting a lot of readlink complains in Jenkins 1.565.3 running as master on HP-UX 11.31.

[timja]

jglick:

pjos best to recheck in 1.586 with the fix of JENKINS-24521. Or if you can run Java 7, do so.

[timja]

danielbeck:

Is this still an issue in Jenkins 1.600+ / 1.596.x?

[timja]

patelakhil:

Hi ,

I am also observing the "copyRecursiveTo" is not completely copying artifacts/files to master from slave.

What i noticed that it had stopped copying one of the json file half. Because of same the next step which parse the json on master is failing.

Same is working fine on master. Let me know any other information is required.

I am writing this very first time on jenkins so let me know if i missed/not following any standard process here.

There is relevant issue but this issue marked as duplicate issue hence writing the comment here.

Thanks n Regards,
Akhilkumar Patel

[timja]

[Originally depends on: JENKINS-14351]