Upload of large number of files fails with: Pending acquire queue has reached its maximum size of 1000 error.

ArkadiuszSkalski-TomTom commented 3 years ago

When I archive a large number of files some files are not archived.

Version report

Jenkins and plugins versions report:

Jenkins: 2.289.2
OS: Linux - 5.4.0-1055-azure
---
git-server:1.10
ldap:2.7
blueocean-jwt:1.24.8
trilead-api:1.0.13
handlebars:3.0.8
testdroid-run-in-cloud:2.114.0
disk-usage:0.28
docker-workflow:1.26
blueocean-pipeline-scm-api:1.24.8
jquery3-api:3.6.0-2
xunit:3.0.2
azure-artifact-manager:94.vcb8b76839313
blueocean-github-pipeline:1.24.8
dtkit-api:3.0.0
pipeline-stage-step:2.5
embeddable-build-status:2.0.3
workflow-scm-step:2.13
pubsub-light:1.16
antisamy-markup-formatter:2.1
ansicolor:1.0.0
blueocean-pipeline-editor:1.24.8
bootstrap5-api:5.0.2-1
bootstrap4-api:4.6.0-3
jsch:0.1.55.2
pipeline-model-api:1.9.1
structs:1.23
credentials-binding:1.27
pipeline-graph-analysis:1.11
cobertura:1.16
cloud-stats:0.27
variant:1.4
pipeline-milestone-step:1.3.2
github-api:1.123
resource-disposer:0.16
pipeline-model-extensions:1.9.1
javadoc:1.6
swarm:3.27
azure-credentials:182.v3ccd4a755864
metrics:4.0.2.8
branch-api:2.6.5
scm-api:2.6.5
pipeline-utility-steps:2.8.0
git-client:3.9.0
azure-vm-agents:783.v58077630847d
favorite:2.3.3
blueocean-rest-impl:1.24.8
windows-azure-storage:359.v09573e5ff73f
display-url-api:2.3.5
audit-trail:3.10
azure-commons:1.1.3
workflow-multibranch:2.26
ws-cleanup:0.39
openstack-cloud:2.58
ssh-slaves:1.32.0
pipeline-github-lib:1.0
blueocean-dashboard:1.24.8
run-condition:1.5
blueocean-autofavorite:1.2.4
apache-httpcomponents-client-4-api:4.5.13-1.0
workflow-durable-task-step:2.39
data-tables-api:1.10.25-3
pipeline-model-definition:1.9.1
email-ext:2.83
workflow-basic-steps:2.23
copyartifact:1.46.1
blueocean-core-js:1.24.8
ssh-credentials:1.19
plugin-util-api:2.4.0
blueocean-config:1.24.8
jjwt-api:0.11.2-9.c8b45b8bb173
build-monitor-plugin:1.12+build.201809061734
cloudbees-bitbucket-branch-source:2.9.10
git:4.8.1
http_request:1.10
stashNotifier:1.20
blueocean-bitbucket-pipeline:1.24.8
credentials:2.5
matrix-auth:2.6.8
forensics-api:1.2.1
mercurial:2.15
windows-slaves:1.8
plain-credentials:1.7
strict-crumb-issuer:2.1.0
pipeline-build-step:2.14
pipeline-input-step:2.12
external-monitor-job:1.7
authentication-tokens:1.4
workflow-aggregator:2.6
rebuild:1.32
timestamper:1.13
warnings-ng:9.4.0
mailer:1.34
echarts-api:5.1.2-3
token-macro:266.v44a80cf277fd
junit:1.51
ant:1.11
pipeline-stage-tags-metadata:1.9.1
generic-webhook-trigger:1.74
blueocean:1.24.8
script-security:1.78
blueocean-pipeline-api-impl:1.24.8
checks-api:1.7.2
blueocean-jira:1.24.8
ivy:2.1
basic-branch-build-strategies:1.3.2
analysis-model-api:10.2.5
parameterized-scheduler:1.0
jenkins-design-language:1.24.8
ssh-agent:1.23
blueocean-git-pipeline:1.24.8
command-launcher:1.6
blueocean-display-url:2.4.1
blueocean-i18n:1.24.8
workflow-cps:2.93
jquery-detached:1.2.1
ace-editor:1.1
role-strategy:3.2.0
workflow-api:2.46
cloudbees-folder:6.16
pam-auth:1.6
github:1.33.1
momentjs:1.1.1
durable-task:1.38
handy-uri-templates-2-api:2.1.8-1.0
popper2-api:2.5.4-3
bitbucket-scm-trait-commit-skip:0.4.0
jdk-tool:1.5
jira:3.5
snakeyaml-api:1.29.1
workflow-job:2.41
sshd:3.1.0
artifactory:3.12.5
maven-plugin:3.12
configuration-as-code:1.51
workflow-cps-global-lib:2.21
greenballs:1.15.1
workflow-step-api:2.24
gradle:1.37.1
extended-read-permission:3.2
azure-sdk:30.vf3165534d6e8
sse-gateway:1.24
caffeine-api:2.9.1-23.v51c4e2c879c8
bouncycastle-api:2.21
code-coverage-api:1.4.0
config-file-provider:3.8.1
matrix-project:1.19
slack:2.48
hashicorp-vault-plugin:3.8.0
monitoring:1.88.0
docker-commons:1.17
font-awesome-api:5.15.3-4
saferestart:0.3
workflow-support:3.8
blueocean-web:1.24.8
pipeline-rest-api:2.19
skip-notifications-trait:1.0.5
configuration-as-code-support:1.18
blueocean-commons:1.24.8
jacoco:3.3.0
htmlpublisher:1.25
blueocean-personalization:1.24.8
blueocean-rest:1.24.8
jackson2-api:2.12.4
popper-api:1.16.1-2
azure-keyvault:123.v1aba71c2d365
blueocean-events:1.24.8
lockable-resources:2.11
pipeline-stage-view:2.19
okhttp-api:3.14.9
github-branch-source:2.11.2

What Operating System are you using (both controller, and any agents involved in the problem)?

Linux 18.04

Reproduction steps

Generate a large amount of files (in my case 8336 files 1.3G in total)

execute archiveArtifacts artifacts: 'data_to_archive/*/.*', followSymlinks: false

pipeline {
agent {
    label "linux"
}

stages {
    stage('Hello') {
        steps {
            sh "cp -rf /home/ubuntu/data_to_archive ." // I uploaded the data to jenkins agent for easier testing
            archiveArtifacts artifacts: 'data_to_archive/**/*.*', followSymlinks: false
        }
    }
}
}

Results

Expected result:

All files are archived.

Actual result:

Some files are missing (reported 3536 times - almost half of all files)

2021-08-31 10:00:13.320+02:00 |  ERROR: [AzureStorage] Failed to upload file /jenkins/workspace/test/archive_test/data_to_archive/20210825/results/test_cases/case3/test_output/output.log, error: Pending acquire queue has reached its maximum size of 1000

timja commented 3 years ago

Could you provide a complete pipeline for generating the artifacts please? It will make it easier to fix

ArkadiuszSkalski-TomTom commented 3 years ago

In my case the data was not generated, I can't share my data set. I will try to provide you a snippet by tomorrow. Cheers.

ArkadiuszSkalski-TomTom commented 3 years ago

ok I actually managed to reproduce it with simple groovy:

pipeline {
    agent {
        label "linux"
    }

    stages {
        stage('Generate') {
            steps {
                sh "mkdir -p generated_files"
                generateFiles()
                archiveArtifacts artifacts: 'generated_files/*.txt', followSymlinks: false
            }
        }
    }
}

void generateFiles() {
    int i = 0
    def largeString = org.apache.commons.lang.RandomStringUtils.randomAlphanumeric(204800)
    10000.times {
        writeFile file: "generated_files/out${i++}.txt", text: largeString
    }
}

timja commented 3 years ago

Perfect thanks

timja commented 3 years ago

This worked on my machine but it depends how quickly files are uploaded I think.

In the meantime I would suggest not trying to upload more than 1500 files at once, 500 concurrent requests and 1000 pending appears to be the limit.

Would doing multiple archive artifacts calls work maybe for different directories? or tar/zipping the folder?

ArkadiuszSkalski-TomTom commented 3 years ago

Would doing multiple archive artifacts calls work maybe for different directories? or tar/zipping the folder?

It is a workaround for a Jenkinsfile owner. The Jenkins Maintainer can't enable the plugin as he has no knowledge about how many files are archived within pipelines.

I think the plugin should simply process files in batches to work around this limitation or fail the build at least as the artifacts are gone.

timja commented 3 years ago

@ArkadiuszSkalski-TomTom I think https://github.com/jenkinsci/azure-artifact-manager-plugin/releases/tag/97.v074e1332e88d should work for you. I wasn't able to reproduce this though

ArkadiuszSkalski-TomTom commented 3 years ago

thanks a ton. I'll test the plugin this week and let you know.

jenkinsci / azure-artifact-manager-plugin