timja / jenkins-gh-issues-poc-06-18

0 stars 0 forks source link

[JENKINS-17236] Pluggable artifact transfer & storage #10076

Closed timja closed 11 years ago

timja commented 11 years ago

Currently artifact archiving follows a fixed model: artifacts are defined in ArtifactArchiver (or MavenArtifactArchiver) configuration; they are copied from slave to master using FilePath.copyRecursiveTo; they are stored in ${Run.rootDir}/archive; and they are served by browsing this directory in master.

This system has obvious scalability limitations: channel transfer is not particularly efficient; and storing enormous files in the builds directory can cause disk overruns.

The CloudBees Fast Archiver plugin tries to address the transfer speed issue by using a better algorithm. This helps, but the implementation (replacing the ArtifactArchiver build step) is very awkward, and cannot be made to work properly with Maven projects, fingerprints, etc.

Some installations may anyway not need to copy files at all. The master and slave might share a common network filesystem, or might have a way of efficiently transferring file references (e.g. ZFS send).

Finally, there should be the option to store artifacts in a better way: on a big slow drive, in a compressed archive, etc.

To handle all these requirements we need an extension point that encompasses both transfer and storage of artifacts for each build.


Originally reported by jglick, imported from: Pluggable artifact transfer & storage
  • assignee: jglick
  • status: Resolved
  • priority: Major
  • resolution: Fixed
  • resolved: 2013-09-13T19:39:23+00:00
  • imported: 2022/01/10
timja commented 11 years ago

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/tasks/ArtifactArchiver.java
core/src/main/java/jenkins/model/ArtifactManager.java
core/src/main/java/jenkins/model/StandardArtifactManager.java
maven-plugin/src/main/java/hudson/maven/MavenBuild.java
maven-plugin/src/main/java/hudson/maven/MavenBuildProxy.java
maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java
maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactRecord.java
http://jenkins-ci.org/commit/jenkins/e1eea67bb2fa1a356492e91313d84a523f441b34
Log:
JENKINS-17236 Initial version of ArtifactManager (no Maven support yet).


You received this message because you are subscribed to the Google Groups "Jenkins Commits" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-commits+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

timja commented 11 years ago

jglick:

API should be fleshed out; still need to add Maven support. Working on a prototype plugin archiving to ${build.rootDir}/archive.zip.

timja commented 11 years ago

scm_issue_link:

Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
core/src/main/java/hudson/FilePath.java
core/src/main/java/hudson/model/DirectoryBrowserSupport.java
core/src/main/java/hudson/model/Run.java
core/src/main/java/hudson/tasks/ArtifactArchiver.java
core/src/main/java/hudson/util/DirScanner.java
core/src/main/java/jenkins/model/ArtifactManager.java
core/src/main/java/jenkins/model/ArtifactManagerConfiguration.java
core/src/main/java/jenkins/model/ArtifactManagerFactory.java
core/src/main/java/jenkins/model/ArtifactManagerFactoryDescriptor.java
core/src/main/java/jenkins/model/StandardArtifactManager.java
core/src/main/java/jenkins/util/VirtualFile.java
core/src/main/resources/jenkins/model/ArtifactManagerConfiguration/config.groovy
core/src/test/java/hudson/model/RunTest.java
maven-plugin/src/main/java/hudson/maven/MavenBuild.java
maven-plugin/src/main/java/hudson/maven/MavenBuildProxy.java
maven-plugin/src/main/java/hudson/maven/MavenModuleSetBuild.java
maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifact.java
maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
maven-plugin/src/main/java/hudson/maven/reporters/MavenArtifactRecord.java
maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java
test/src/test/java/hudson/maven/MavenMultiModuleTest.java
test/src/test/java/hudson/model/RunTest.java
http://jenkins-ci.org/commit/jenkins/b698b672ec5d4033115d84b2bc2b179812080bdd
Log:
[FIXED JENKINS-17236] Introduce ArtifactManager

Merge branch 'ArtifactManager-JENKINS-17236'

Compare: https://github.com/jenkinsci/jenkins/compare/eea972ec11a1...b698b672ec5d

timja commented 11 years ago

dogfood:

Integrated in jenkins_main_trunk #2874
JENKINS-17236 Initial version of ArtifactManager (no Maven support yet). (Revision e1eea67bb2fa1a356492e91313d84a523f441b34)

Result = SUCCESS
Jesse Glick : e1eea67bb2fa1a356492e91313d84a523f441b34
Files :

timja commented 11 years ago

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
src/main/java/hudson/maven/MavenBuild.java
src/main/java/hudson/maven/MavenBuildProxy.java
src/main/java/hudson/maven/reporters/MavenArtifact.java
src/main/java/hudson/maven/reporters/MavenArtifactArchiver.java
src/main/java/hudson/maven/reporters/MavenArtifactRecord.java
http://jenkins-ci.org/commit/maven-plugin/fa816189a1581da05c3a9a604c6d7460232b7db7
Log:
JENKINS-17236 Initial version of ArtifactManager (no Maven support yet).
Originally-Committed-As: e1eea67bb2fa1a356492e91313d84a523f441b34

timja commented 10 years ago

scm_issue_link:

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/hudson/model/Run.java
http://jenkins-ci.org/commit/jenkins/c1fcb08e08c3ccae1451602af8ea20788dc9950e
Log:
JENKINS-17236 Run.deleteArtifacts needs to go through ArtifactManager.
(Also relevant to JENKINS-22395 diagnostics.)

timja commented 10 years ago

dogfood:

Integrated in jenkins_main_trunk #3347
JENKINS-17236 Run.deleteArtifacts needs to go through ArtifactManager. (Revision c1fcb08e08c3ccae1451602af8ea20788dc9950e)

Result = SUCCESS
Jesse Glick : c1fcb08e08c3ccae1451602af8ea20788dc9950e
Files :

timja commented 2 years ago

[Originally related to: JENKINS-7813]

timja commented 2 years ago

[Originally related to: JENKINS-10502]

timja commented 2 years ago

[Originally related to: JENKINS-21958]

timja commented 2 years ago

[Originally depends on: JENKINS-19752]

timja commented 2 years ago

[Originally depends on: JENKINS-20663]

timja commented 2 years ago

[Originally depends on: JENKINS-19947]