[JENKINS-12379] Archive the artifacts should allow specifying the target artifacts path

[timja]

This is a problem that was mentioned on multiple occasions and strangely enough, it was never really answered or sometimes not
understood.

Let me provide some explanation:

Job A
Produces the following artifact:
Build/release/x86/
build/release/x86/fr/
build/release/x86/ru/

Job B
Needs job A artifacts in the following folder: "lib/myJobA/"
However it should not get "build/release/x86" Job A artifact folder. What we really want here is:
lib/myJobA/
lib/myJobA/fr/
lib/myJobA/ru/

However, unless I'm missing something obvious, neither "Archive the artifacts" nor "Copy artifacts from another project" will let you do that.
"Archive the artifacts" will mirror the whole specified path "Build/release/x86/..." and "Copy artifacts from another project" will also copy the whole path to the specified target. (ie: lib/myJobA/build/release/x86/)

"flatten" option will not help as it will not keep "fr" and "ru" folder.
For that specific example, a possible workaround is to create 3 separate "Copy artifacts" build steps and to copy each folder
individually in the specific target folder (using the flatten option). However, not only it is quite cumbersome, but also makes it quite difficult to refactor the job at a later time with so much data specific to jobA workspace.
I suppose that a script could help (I'm new to Jenkis) but I believe that this should be a core feature of Jenkins "Archive the
artifacts" (even though it would be possible to implement it at the "copy artifact" level, it seems much better to have jobB agnostic of jobA workspace specifics).

Teamcity has a smart approach to handle this:
If you specify: build/release/x86/*/.dll, it will omit "build/release/x86/" when archiving the artifacts.

It also allows for more explicit target definition by using "=>" (ie: windows/.zip => winfiles,unix/.tgz => linuxFiles)
Having at least the first option seems a must-have. In order to keep compatibility, it should be possible to check an option: [x] rebase (or re-root or whatever sounds good in English). By checking this option (unchecked by default), one would agree to have the artifacts path interpreted à la Teamcity.

Hope all this makes sense.

---

Originally reported by frederic_latour, imported from: Archive the artifacts should allow specifying the target artifacts path

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[timja]

frederic_latour:

Two things following the change from bug to Improvement and the addition of copyartifact as a component:

1. A bug does not always mean an error is thrown. As it stands now, you have either to generate your artifacts by taking into account that you won't be able to rebase/change the path or add an additional script build step to move your artifacts within the current workspace before archiving your artifacts or create multiple "copy artifacts" build steps in your downstream project. I would consider this problem a functional bug.

2. It would be great that copyartifact also solve this problem. However, having this solved in the core is a must have. Basically, you want to expose your artifacts with a certain folder structure that is not necessary identical to the folder structure used within your workspace by the build.

[timja]

frederic_latour:

While trying to find out some workaround to this problem, I can say that I was a bit too optimistic.
Basically executing a script as the last build step before archiving the artifacts won't help that much because you would need to put your artifacts in your workspace root which would basically make it impossible to specify artifacts files and folders.

[timja]

tomobilplus:

I also run in trouble with same issue. I would like to have a functionality that I just specify a directory path in workspace where the artefacts are copies in, without creating recursive directory structure. The UI in my mind is like:

Artifacts to copy: /Build/projA/*/.*
Target directory: /lib/projA

Remove leading directories: YES (YES/NO)

[timja]

frederic_latour:

The teamcity approach is much more flexible than having a "remove leading directories" option.
This would be however better than nothing.

[timja]

stasl:

I came across the same issue, and since I have quite a few nested directories it makes it impractical to create a copy artifacts step for each one.

It would be really great to have this functionality built in, as right now it would seem like one has to resort to using batch scripting black magia, which is never a good sign, and rarely pleasant.

[timja]

iristyle:

I would much prefer being able to structure the dir layout somehow when the artifacts are created. I get unnecessary nesting of archive/foo/*.

[timja]

qwert:

It's a common requirement for many serious projects.

[timja]

kmilesdev:

To note, the "Publish Over SSH Plugin" handles this in a really simple way, by providing a "Remove prefix" field:

https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH+Plugin

[timja]

danielbeck:

Would the proposed solution in the attached screenshots work for everyone? It essentially splits pattern and user-defined common prefix in two. It's not as powerful as the TeamCity approach, but should cover almost all use cases without having to resort to shell scripts.

[timja]

danielbeck:

I guess not.

[timja]

danielbeck:

As I need this feature now, implementing as proposed above.

[timja]

danielbeck:

Proposed solution at https://github.com/jenkinsci/jenkins/pull/1493

[timja]

wgroper:

I think this would be a perfect solution for my needs.

[timja]

drewsky7821:

Hi danielbeck,

Looks like this thread lost some traction. The proposed solution looks good, is there anything I can do to help get the pull request pushed through?

Thanks!

[timja]

danielbeck:

Thanks for the reminder. I'll try to revisit Jesse's objection pointing to continued user interest.

[timja]

miq:

I have exactly the same problem and would love to see a fix upstream.

[timja]

tosborn:

I just spent several hours trying to resolve this exact same problem where we have nested subfolders we need to deploy without duplicating the parent directory structure. This would be a definite improvement in the functionality. Hope to see the enhancement implemented soon.

[timja]

ssbarnea:

I think that the correct PR is https://github.com/jenkinsci/jenkins/pull/1878 but is more than 2 years old and needs rebase and some work done to it.

Anyone willing to takeover if danielbeck cannot?

I am big afraid the the Windows testing is outside my power, I no longer have even a VM with it.

[timja]

[Originally duplicated by: JENKINS-16405]

[timja]

[Originally duplicated by: JENKINS-14367]

[timja]

[Originally related to: JENKINS-48489]