search

jenkinsci / azure-artifact-manager-plugin

Jenkins Azure artifact manager plugin
https://plugins.jenkins.io/azure-artifact-manager/
MIT License
6 stars 10 forks source link

When uploading a large directory, timeouts occur. #95

Closed WenjunMao closed 7 months ago

WenjunMao commented 7 months ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.401.3 OS: Linux - 5.15.0-1056-azure Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- active-directory:2.33 antisamy-markup-formatter:162.v0e6ec0fcfcf6 apache-httpcomponents-client-4-api:4.5.14-208.v438351942757 authentication-tokens:1.53.v1c90fd9191a_b_ authorize-project:1.7.1 azure-artifact-manager:133.vf94ad3455cdc azure-credentials:312.v0f3973cd1e59 azure-keyvault:200.v115e9b_1644d5 azure-sdk:157.v855da_0b_eb_dc2 azure-vm-agents:859.v7213476e4fea_ batch-task:1.398.v550a_8e8ca_e5f blueocean:1.27.5 blueocean-bitbucket-pipeline:1.27.9 blueocean-commons:1.27.9 blueocean-config:1.27.9 blueocean-core-js:1.27.9 blueocean-dashboard:1.27.9 blueocean-display-url:2.4.2 blueocean-events:1.27.9 blueocean-git-pipeline:1.27.9 blueocean-github-pipeline:1.27.9 blueocean-i18n:1.27.9 blueocean-jira:1.27.9 blueocean-jwt:1.27.9 blueocean-personalization:1.27.9 blueocean-pipeline-api-impl:1.27.9 blueocean-pipeline-editor:1.27.9 blueocean-pipeline-scm-api:1.27.9 blueocean-rest:1.27.9 blueocean-rest-impl:1.27.9 blueocean-web:1.27.9 bootstrap5-api:5.3.2-3 bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9 branch-api:2.1128.v717130d4f816 build-failure-analyzer:2.4.2 build-keeper-plugin:19.va_df8a_2c65123 build-pipeline-plugin:1.5.8 build-symlink:1.1 build-timeout:1.31 build-user-vars-plugin:1.9 caffeine-api:3.1.8-133.v17b_1ff2e0599 categorized-view:1.12 checks-api:2.0.2 cloud-stats:336.v788e4055508b_ cloudbees-bitbucket-branch-source:856.v04c46c86f911 cloudbees-disk-usage-simple:203.v3f46a_7462b_1a_ cloudbees-folder:6.858.v898218f3609d cobertura:1.17 code-coverage-api:4.99.0 command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.13.0-62.v7d18e55f51e2 commons-text-api:1.11.0-95.v22a_d30ee5d36 conditional-buildstep:1.4.3 configuration-as-code:1670.v564dc8b_982d0 copyartifact:722.v0662a_9b_e22a_c coverage:1.10.0 credentials:1311.vcf0a_900b_37c2 credentials-binding:642.v737c34dea_6c2 data-tables-api:1.13.6-5 description-setter:1.10 display-url-api:2.200.vb_9327d658781 durable-task:543.v262f6a_803410 dynamic-axis:1.0.3 echarts-api:5.4.0-7 elastic-axis:464.va_7ed499b_9d75 email-ext:2.101 envinject:2.908.v66a_774b_31d93 envinject-api:1.199.v3ce31253ed13 extended-choice-parameter:376.v2e02857547b_a_ extra-columns:1.26 favorite:2.4.3 font-awesome-api:6.5.1-1 forensics-api:2.3.0 gerrit-trigger:2.39.1 git:5.2.1 git-client:4.6.0 github:1.37.3.1 github-api:1.318-461.v7a_c09c9fa_d63 github-branch-source:1771.v59b_6a_fa_1b_89e groovy:457.v99900cb_85593 gson-api:2.10.1-15.v0d99f670e0a_7 handy-uri-templates-2-api:2.1.8-30.v7e777411b_148 hidden-parameter:138.v2a_c30d04e566 htmlpublisher:1.32 instance-identity:185.v303dc7c645f9 ionicons-api:56.v1b_1c8c49374e jackson2-api:2.16.1-373.ve709c6871598 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-8 jaxb:2.3.9-1 jdk-tool:63.v62d2fd4b_4793 jenkins-design-language:1.27.9 jersey2-api:2.41-133.va_03323b_a_1396 jira:3.11 jjwt-api:0.11.5-77.v646c772fddb_0 job-import-plugin:3.6 jobConfigHistory:1229.v3039470161a_d jobrequeue:1.1 joda-time-api:2.12.6-21.vca_fd74418fb_7 join:1.21 jquery:1.12.4-1 jquery3-api:3.7.1-1 jsch:0.2.16-86.v42e010d9484b_ json-path-api:2.8.0-21.v8b_7dc8b_1037b_ junit:1252.vfc2e5efa_294f lockable-resources:1232.v512d6c434eb_d log-parser:2.3.1 mailer:463.vedf8358e006b_ mask-passwords:173.v6a_077a_291eb_5 matrix-auth:3.2.1 matrix-combinations-parameter:1.3.3 matrix-project:822.v01b_8c85d16d2 metrics:4.2.18-442.v02e107157925 mina-sshd-api-common:2.11.0-86.v836f585d47fa_ mina-sshd-api-core:2.11.0-86.v836f585d47fa_ naginator:1.421.v85c11e2db_507 next-executions:179.vb_e011d7e3b_64 okhttp-api:4.11.0-157.v6852a_a_fa_ec11 parameter-separator:87.va_1816d0b_39d1 parameterized-scheduler:262.v00f3d90585cc parameterized-trigger:2.46 pipeline-build-step:540.vb_e8849e1a_b_d8 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:689.veec561a_dee13 pipeline-input-step:477.v339683a_8d55e pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2150.v4cfd8916915c pipeline-model-definition:2.2150.v4cfd8916915c pipeline-model-extensions:2.2150.v4cfd8916915c pipeline-rest-api:2.34 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2150.v4cfd8916915c pipeline-stage-view:2.33 pipeline-utility-steps:2.16.0 plain-credentials:143.v1b_df8b_d3b_e48 plot:2.1.12 plugin-util-api:3.8.0 postbuild-task:1.9 preSCMbuildstep:71.v1f2990a_37e27 prism-api:1.29.0-8 prometheus:2.2.3 publish-over:0.22 publish-over-ssh:1.25 pubsub-light:1.18 rebuild:320.v5a_0933a_e7d61 repo:1.16.0 resource-disposer:0.23 role-strategy:689.v731678c3e0eb_ run-condition:1.7 saferestart:0.7 scm-api:676.v886669a_199a_a_ script-security:1313.v7a_6067dc7087 skip-certificate-check:1.1 snakeyaml-api:2.2-111.vc6598e30cc65 sse-gateway:1.26 ssh-agent:333.v878b_53c89511 ssh-credentials:308.ve4497b_ccd8f4 sshd:3.249.v2dc2ea_416e33 structs:325.vcb_307d2a_2782 text-finder:1.26 throttle-concurrents:2.14 timestamper:1.26 token-macro:400.v35420b_922dcb_ trilead-api:2.133.vfb_8a_7b_9c5dd1 uno-choice:2.7.2 validating-string-parameter:183.v3748e79b_9737 variant:60.v7290fc0eb_b_cd view-job-filters:369.ve0513a_a_f5524 windows-azure-storage:386.v673495b0a5de workflow-aggregator:596.v8c21c963d92d workflow-api:1283.v99c10937efcb_ workflow-basic-steps:1042.ve7b_140c4a_e0c workflow-cps:3837.v305192405b_c0 workflow-durable-task-step:1313.vcb_970b_d2a_fb_3 workflow-job:1326.ve643e00e9220 workflow-multibranch:770.v1a_d0708dd1f6 workflow-scm-step:415.v434365564324 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:865.v43e78cc44e0d ws-cleanup:0.45 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller: Docker image jenkins/jenkins:2.401.3-jdk11 Agents: azure-vm-agents. os ubuntu 18.04

Reproduction steps

uploading a large directory(over 10, 000 files)

Expected Results

upload files to storageaccount as expected

Actual Results

jobs are failing with the following error stack (and no message on the controller logs):

00:39:10 Archiving artifacts
00:40:12 ERROR: Step ‘Archive the artifacts’ aborted due to exception: 
00:40:12 java.lang.IllegalStateException: Timeout on blocking read for 30000000000 NANOSECONDS
00:40:12    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:124)
00:40:12    at reactor.core.publisher.Mono.block(Mono.java:1766)
00:40:12    at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:135)
00:40:12    at com.azure.storage.blob.BlobClient.uploadFromFileWithResponse(BlobClient.java:504)
00:40:12    at com.microsoft.jenkins.artifactmanager.AzureArtifactManager$UploadToBlobStorage.lambda$invoke$1(AzureArtifactManager.java:309)
00:40:12    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
00:40:12    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
00:40:12    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
00:40:12    at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
00:40:12    at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746)
00:40:12    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
00:40:12    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
00:40:12    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
00:40:12    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
00:40:12    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
00:40:12 Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to cov-build-slave97c630
00:40:12        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
00:40:12        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
00:40:12        at hudson.remoting.Channel.call(Channel.java:1000)
00:40:12        at hudson.FilePath.act(FilePath.java:1192)
00:40:12        at hudson.FilePath.act(FilePath.java:1181)
00:40:12        at com.microsoft.jenkins.artifactmanager.AzureArtifactManager.archive(AzureArtifactManager.java:159)
00:40:12        at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:258)
00:40:12        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
00:40:12        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
00:40:12        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
00:40:12        at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
00:40:12        at hudson.model.Build$BuildExecution.post2(Build.java:179)
00:40:12        at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)
00:40:12        at hudson.model.Run.execute(Run.java:1925)
00:40:12        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
00:40:12        at hudson.model.ResourceController.execute(ResourceController.java:101)
00:40:12        at hudson.model.Executor.run(Executor.java:442)
00:40:12 Caused: java.lang.IllegalStateException
00:40:12    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
00:40:12    at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
00:40:12    at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
00:40:12    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
00:40:12    at java.base/java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:600)
00:40:12    at java.base/java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:678)
00:40:12    at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:737)
00:40:12    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
00:40:12    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
00:40:12    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
00:40:12    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
00:40:12    at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:661)
00:40:12    at com.microsoft.jenkins.artifactmanager.AzureArtifactManager$UploadToBlobStorage.invoke(AzureArtifactManager.java:303)
00:40:12    at com.microsoft.jenkins.artifactmanager.AzureArtifactManager$UploadToBlobStorage.invoke(AzureArtifactManager.java:245)
00:40:12    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3578)
00:40:12    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
00:40:12    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
00:40:12    at hudson.remoting.Request$2.run(Request.java:377)
00:40:12    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
00:40:12    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
00:40:12    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
00:40:12    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
00:40:12    at java.base/java.lang.Thread.run(Thread.java:829)
00:40:12 Caused: java.io.IOException
00:40:12    at com.microsoft.jenkins.artifactmanager.AzureArtifactManager.archive(AzureArtifactManager.java:166)
00:40:12    at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:258)
00:40:12    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
00:40:12    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
00:40:12    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
00:40:12    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
00:40:12    at hudson.model.Build$BuildExecution.post2(Build.java:179)
00:40:12    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)
00:40:12    at hudson.model.Run.execute(Run.java:1925)
00:40:12    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
00:40:12    at hudson.model.ResourceController.execute(ResourceController.java:101)
00:40:12    at hudson.model.Executor.run(Executor.java:442)

Anything else?

No response

Are you interested in contributing a fix?

yes

timja commented 7 months ago

PR welcome if you are able to tune it, otherwise a pipeline that sets up test data (using linux tooling ideally not windows) and can reliably reproduce it would be good to.

WenjunMao commented 7 months ago

I'm not entirely sure why it happened. My guess is that the slave VM had 32 vCPU, so simultaneously handling too much upload tasks caused some of them to make no progress and eventually timeout. Therefore, I'm considering changing the timeout from 30 seconds to 10 minutes.

I will raise the PR when I confirm that it works.