jenkinsci / artifactory-artifact-manager-plugin

artifactory-artifact-manager-plugin
https://plugins.jenkins.io/artifactory-artifact-manager/
MIT License
0 stars 5 forks source link

Long/failed page loads for jobs with large number of artifacts #41

Closed tim-goto closed 3 months ago

tim-goto commented 3 months ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.457 OS: Linux - 5.15.0-205.149.5.1.el8uek.x86_64 Java: 17.0.11 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- anka-build:2.11.0 ansicolor:1.0.4 ant:497.v94e7d9fffa_b_9 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 artifactory:4.0.6 artifactory-artifact-manager:73.v0e73747d2f6f asm-api:9.7-33.v4d23ef79fcc8 authentication-tokens:1.53.v1c90fd9191a_b_ aws-credentials:231.v08a_59f17d742 aws-java-sdk:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-api-gateway:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-autoscaling:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-cloudformation:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-cloudfront:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-codebuild:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-codedeploy:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-ec2:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-ecr:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-ecs:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-efs:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-elasticbeanstalk:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-elasticloadbalancingv2:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-iam:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-kinesis:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-lambda:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-logs:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-minimal:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-organizations:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-secretsmanager:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-sns:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-sqs:1.12.696-451.v0651a_da_9ca_ec aws-java-sdk-ssm:1.12.696-451.v0651a_da_9ca_ec azure-ad:476.vd1c42682ea_6a_ azure-sdk:157.v855da_0b_eb_dc2 basic-branch-build-strategies:81.v05e333931c7d bootstrap5-api:5.3.3-1 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1169.va_f810c56e895 caffeine-api:3.1.8-133.v17b_1ff2e0599 checks-api:2.2.0 cloudbees-bitbucket-branch-source:886.v44cf5e4ecec5 cloudbees-disk-usage-simple:203.v3f46a_7462b_1a_ cloudbees-folder:6.928.v7c780211d66e command-launcher:107.v773860566e2e commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-109.vfe16c66636eb_ config-file-provider:973.vb_a_80ecb_9a_4d0 configuration-as-code:1775.v810dc950b_514 copyartifact:722.v0662a_9b_e22a_c coverage:1.14.0 credentials:1337.v60b_d7b_c7b_c9f credentials-binding:677.vdc9d38cb_254d dark-theme:439.vdef09f81f85e data-tables-api:2.0.7-1 delivery-pipeline-plugin:1.4.2 display-url-api:2.204.vf6fddd8a_8b_e9 docker-build-publish:1.4.0 docker-commons:439.va_3cb_0a_6a_fb_29 docker-workflow:572.v950f58993843 durable-task:555.v6802fe0f0b_82 ec2-fleet:3.2.0 echarts-api:5.5.0-1 email-ext:1806.v856a_01a_fa_39a_ extended-read-permission:53.v6499940139e5 folder-properties:1.2.1 font-awesome-api:6.5.2-1 forensics-api:2.4.0 git:5.2.2 git-client:4.7.0 gradle:2.11 gson-api:2.10.1-15.v0d99f670e0a_7 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 hashicorp-vault-plugin:368.v48134f694db_f http_request:1.18 instance-identity:185.v303dc7c645f9 ionicons-api:73.vb_f522f227457 jackson2-api:2.17.0-379.v02de8ec9f64c jakarta-activation-api:2.1.3-1 jakarta-mail-api:2.1.3-1 javadoc:243.vb_b_503b_b_45537 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.9-1 jdk-tool:73.vddf737284550 jersey2-api:2.42-147.va_28a_44603b_d5 jfrog:1.5.0 job-dsl:1.87 joda-time-api:2.12.7-29.v5a_b_e3a_82269a_ jquery:1.12.4-1 jquery3-api:3.7.1-2 jsch:0.2.16-86.v42e010d9484b_ json-api:20240303-41.v94e11e6de726 json-path-api:2.9.0-58.v62e3e85b_a_655 junit:1265.v65b_14fa_f12f0 kubernetes:4214.vf10083a_42e70 kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2 kubernetes-credentials:0.11 lockable-resources:1255.vf48745da_35d0 mailer:472.vf7c289a_4b_420 matrix-auth:3.2.2 matrix-project:822.824.v14451b_c0fd42 maven-plugin:3.23 metrics:4.2.21-449.v6960d7c54c69 mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd node-iterator-api:55.v3b_77d4032326 okhttp-api:4.11.0-172.vda_da_1feeb_c6e parameter-separator:166.vd0120849b_386 parameterized-trigger:806.vf6fff3e28c3e pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-graph-analysis:216.vfd8b_ece330ca_ pipeline-graph-view:271.v330cf007b_fe6 pipeline-groovy-lib:704.vc58b_8890a_384 pipeline-input-step:495.ve9c153f6067b_ pipeline-milestone-step:119.vdfdc43fc3b_9a_ pipeline-model-api:2.2198.v41dd8ef6dd56 pipeline-model-definition:2.2198.v41dd8ef6dd56 pipeline-model-extensions:2.2198.v41dd8ef6dd56 pipeline-rest-api:2.34 pipeline-stage-step:312.v8cd10304c27a_ pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56 pipeline-stage-view:2.34 pipeline-utility-steps:2.16.2 plain-credentials:179.vc5cb_98f6db_38 plugin-util-api:4.1.0 prism-api:1.29.0-15 prometheus:763.vf8c26b_a_34b_d6 rebuild:332.va_1ee476d8f6d remote-file:1.24 resource-disposer:0.23 scm-api:690.vfc8b_54395023 script-security:1336.vf33a_a_9863911 slack:715.v1cfed1b_9c63c snakeyaml-api:2.2-111.vc6598e30cc65 sonar:2.17.2 ssh-agent:367.vf9076cd4ee21 ssh-credentials:337.v395d2403ccd4 ssh-slaves:2.948.vb_8050d697fec sshd:3.322.v159e91f6a_550 structs:337.v1b_04ea_4df7c8 theme-manager:215.vc1ff18d67920 timestamper:1.26 token-macro:400.v35420b_922dcb_ trilead-api:2.142.v748523a_76693 variant:60.v7290fc0eb_b_cd workflow-aggregator:596.v8c21c963d92d workflow-api:1291.v51fd2a_625da_7 workflow-basic-steps:1058.vcb_fc1e3a_21a_9 workflow-cps:3894.3896.vca_2c931e7935 workflow-durable-task-step:1336.v768003e07199 workflow-job:1400.v7fd111b_ec82f workflow-multibranch:783.va_6eb_ef636fb_d workflow-scm-step:427.v4ca_6512e7df1 workflow-step-api:657.v03b_e8115821b_ workflow-support:907.v6713a_ed8a_573 ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux

Reproduction steps

For jobs that archive a large number of artifacts (more than 50), the page for https://your-jenkins.com/job/lastSuccessfulBuild/artifact/ loads slow or even runs into timeouts.

Expected Results

the page loads in an acceptable timeframe (less than 10 seconds)

Actual Results

the page loads very slow (more than 60 seconds) or not at all

Anything else?

No response

Are you interested in contributing a fix?

Of course this depends on the latency to your artifactory instance but is also due to the fact that for every column in the resulting table an API call to the artifactory instance is made (file name, size, last accessed). In total I observe about 10 requests per file. The code for e.g. last modified looks like this:

return artifactory
                .repository(this.config.repository)
                .file(targetPath)
                .info()
                .getLastModified()
                .getTime();

The .info() would also already contain the information about the filename and size but for those there is a separate API call. This information could be buffered somewhere/somehow (probably in the virtual file). That way there would be no need for separate API calls. Please let me know if you would like me to create a PR.

tim-goto commented 3 months ago

e.g. the following page https://myjenkins.com/job/myjob/2/artifact/ causes the following number of api calls:

Jun 04, 2024 1:29:49 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to GET /artifactory/api/storage/myjob/2/artifacts HTTP/1.1

Jun 04, 2024 1:29:49 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to GET /artifactory/api/storage/myjob/2/artifacts/ HTTP/1.1

Jun 04, 2024 1:29:50 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to POST /artifactory/api/search/aql HTTP/1.1

Jun 04, 2024 1:29:50 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to GET /artifactory/api/storage/myjob/2/artifacts/out.txt HTTP/1.1

Jun 04, 2024 1:29:50 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to GET /artifactory/api/storage/myjob/2/artifacts/out.txt HTTP/1.1

Jun 04, 2024 1:29:51 PM FINEST io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient size

Getting size for jenkins-timhaering/short-term-storage/team-dois-jenkins-admins/moreenv/2/artifacts/out.txt

Jun 04, 2024 1:29:51 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to GET /artifactory/api/storage/myjob/2/artifacts/out.txt HTTP/1.1

Jun 04, 2024 1:29:51 PM FINEST io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lastUpdated

Getting last updated time for jenkins-timhaering/short-term-storage/team-dois-jenkins-admins/moreenv/2/artifacts/out.txt

Jun 04, 2024 1:29:51 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to GET /artifactory/api/storage/myjob/2/artifacts/out.txt HTTP/1.1

Jun 04, 2024 1:29:51 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to GET /artifactory/api/storage/myjob/2/artifacts/out.txt HTTP/1.1

Jun 04, 2024 1:29:52 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to GET /artifactory/api/storage/myjob/2/artifacts/ HTTP/1.1

Jun 04, 2024 1:29:52 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to POST /artifactory/api/search/aql HTTP/1.1

Jun 04, 2024 1:29:52 PM FINE io.jenkins.plugins.artifactory_artifacts.ArtifactoryClient lambda$buildArtifactory$2

Sending Artifactory request to GET /artifactory/api/storage/myjob/2/artifacts/out.txt HTTP/1.1

the job has only one artifact.

jonesbusy commented 3 months ago

Sure any help is appreciated to optimize call on Artifactory

jonesbusy commented 3 months ago

Should be fixed by #45 on https://github.com/jenkinsci/artifactory-artifact-manager-plugin/releases/tag/103.v1828c151fa_7c