Upload failing randomly with `com.microsoftopentechnologies.windowsazurestorage.exceptions.WAStorageException: Fail to upload individual files to blob` error

dduportal commented 1 year ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.387.2 OS: Linux - 5.4.0-1105-azure Java: 11.0.18 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- analysis-model-api:11.1.0 ansicolor:1.0.2 ant:481.v7b_09e538fcca antisamy-markup-formatter:159.v25b_c67cd35fb_ apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 authentication-tokens:1.53.v1c90fd9191a_b_ aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-cloudformation:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-codebuild:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-ec2:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-ecr:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-ecs:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-efs:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-elasticbeanstalk:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-iam:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-kinesis:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-logs:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-minimal:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-sns:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-sqs:1.12.447-380.v65b_d0c044a_b_a_ aws-java-sdk-ssm:1.12.447-380.v65b_d0c044a_b_a_ azure-artifact-manager:109.vaf4232a14e2e azure-container-agents:253.vd2f5cd5c5040 azure-credentials:254.v64da_8176c83a azure-sdk:118.v43f74dd9ca_dc azure-vm-agents:825.v470cb_9e7361a_ basic-branch-build-strategies:71.vc1421f89888e beer:42.v776b_04d96de3 blueocean:1.27.3 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.3 blueocean-commons:1.27.3 blueocean-config:1.27.3 blueocean-core-js:1.27.3 blueocean-dashboard:1.27.3 blueocean-display-url:2.4.1 blueocean-events:1.27.3 blueocean-git-pipeline:1.27.3 blueocean-github-pipeline:1.27.3 blueocean-i18n:1.27.3 blueocean-jira:1.27.3 blueocean-jwt:1.27.3 blueocean-personalization:1.27.3 blueocean-pipeline-api-impl:1.27.3 blueocean-pipeline-editor:1.27.3 blueocean-pipeline-scm-api:1.27.3 blueocean-rest:1.27.3 blueocean-rest-impl:1.27.3 blueocean-web:1.27.3 bootstrap4-api:4.6.0-5 bootstrap5-api:5.2.2-2 bouncycastle-api:2.27 branch-api:2.1071.v1a_188a_562481 build-discarder:139.v05696a_7fe240 buildtriggerbadge:251.vdf6ef853f3f5 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:2.0.0 cloud-stats:267.v577e3742c282 cloudbees-bitbucket-branch-source:800.va_b_b_9a_a_5035c1 cloudbees-folder:6.815.v0dd5a_cb_40e0e code-coverage-api:4.2.2 command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-36.vc008c8fcda_7b_ conditional-buildstep:1.4.2 config-file-provider:3.11.1 configuration-as-code:1625.v27444588cc3d copyartifact:686.v6fd37018d7c2 credentials:1224.vc23ca_a_9a_2cb_0 credentials-binding:604.vb_64480b_c56ca_ cvs:2.19.1 dark-theme:315.va_22e7d692ea_a data-tables-api:1.13.3-3 disable-github-multibranch-status:1.2 display-url-api:2.3.7 docker-commons:419.v8e3cd84ef49c docker-workflow:563.vd5d2e5c4007f durable-task:504.vb10d1ae5ba2f ec2:2.0.7 echarts-api:5.4.0-3 embeddable-build-status:369.vb_a_68a_575a_b_11 extended-read-permission:3.2 external-monitor-job:203.v683c09d993b_9 favorite:2.4.1 font-awesome-api:6.3.0-2 forensics-api:2.1.0 generic-tool:1.0 git:5.0.0 git-client:4.2.0 git-forensics:2.0.0 github:1.37.0 github-api:1.303-417.ve35d9dd78549 github-branch-source:1703.vd5a_2b_29c6cdc github-checks:1.0.19 groovy:453.vcdb_a_c5c99890 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.31 http_request:1.16 instance-identity:142.v04572ca_5b_265 ionicons-api:45.vf54fca_5d2154 jackson2-api:2.14.2-319.v37853346a_229 jacoco:3.3.3 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:226.v71211feb_e7e9 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:63.v62d2fd4b_4793 jenkins-design-language:1.27.3 jersey2-api:2.39.1-1 jira:3.9 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.16-2 jobConfigHistory:1207.vd28a_54732f92 jquery:1.12.4-1 jquery3-api:3.6.4-1 jsch:0.1.55.61.va_e9ee26616e7 junit:1189.v1b_e593637fa_e junit-attachments:131.vef2117770555 junit-realtime-test-reporter:102.v3f020960a_646 keyboard-shortcuts-plugin:1.4 kubernetes:3910.ve59cec5e33ea_ kubernetes-client-api:6.4.1-215.v2ed17097a_8e9 kubernetes-credentials:0.10.0 ldap:671.v2a_9192a_7419d lockable-resources:1141.v7c5f8f31d2ee mailer:448.v5b_97805e3767 mapdb-api:1.0.9-28.vf251ce40855d matrix-auth:3.1.6 matrix-project:785.v06b_7f47b_c631 maven-plugin:3.21 metrics:4.2.13-420.vea_2f17932dd6 mina-sshd-api-common:2.9.2-62.v199162f0a_2f8 mina-sshd-api-core:2.9.2-62.v199162f0a_2f8 node-iterator-api:49.v58a_8b_35f8363 okhttp-api:4.10.0-132.v7a_7b_91cef39c pam-auth:1.10 parallel-test-executor:383.vde60119c849a parameterized-trigger:2.45 pipeline-build-step:488.v8993df156e8d pipeline-github:2.8-147.3206e8179b1c pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-graph-view:183.v9e27732d970f pipeline-groovy-lib:656.va_a_ceeb_6ffb_f7 pipeline-input-step:466.v6d0a_5df34f81 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2125.vddb_a_44a_d605e pipeline-model-definition:2.2125.vddb_a_44a_d605e pipeline-model-extensions:2.2125.vddb_a_44a_d605e pipeline-rest-api:2.32 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2125.vddb_a_44a_d605e pipeline-stage-view:2.32 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:3.2.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 prism-api:1.29.0-4 pubsub-light:1.17 run-condition:1.5 scm-api:631.v9143df5b_e4a_a script-security:1229.v4880b_b_e905a_6 snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 sse-gateway:1.26 ssh-agent:327.v230ecd01f86f ssh-credentials:305.v8f4381501156 ssh-slaves:2.877.v365f5eb_a_b_eec sshd:3.275.v9e17c10f2571 structs:324.va_f5d6774f3a_d subversion:2.17.1 support-core:1266.v6d096c154c90 theme-manager:1.6 timestamper:1.24 token-macro:321.vd7cc1f2a_52c8 translation:1.16 trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb warnings-ng:10.1.0 windows-azure-storage:380.va3a027b784f4 workflow-aggregator:596.v8c21c963d92d workflow-api:1208.v0cc7c6e0da_9e workflow-basic-steps:1010.vf7a_b_98e847c1 workflow-cps:3653.v07ea_433c90b_4 workflow-durable-task-step:1244.vee71f675dee6 workflow-job:1289.vd1c337fd5354 workflow-multibranch:733.v109046189126 workflow-scm-step:408.v7d5b_135a_b_d49 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller: Docker LTS image 2.387.2 JDK11 (Debian) running on Ubuntu 18.04 Host
Agents:
- Linux Ubuntu 22.04 base OS (https://github.com/jenkins-infra/packer-images/releases/tag/1.2.1) Docker image
- Kubernetes agents: either on AWS EKS or Digital Ocean managed cluster

Reproduction steps

Configured the following Azure Storage Account in the controller (+ Verified successfully through the "Verify Account" button):

Click to view the JSON storage account definition

```json { "sku": { "name": "Standard_LRS", "tier": "Standard" }, "kind": "StorageV2", "id": "/subscriptions//resourceGroups//providers/Microsoft.Storage/storageAccounts/cijenkinsioartifacts", "name": "cijenkinsioartifacts", "type": "Microsoft.Storage/storageAccounts", "location": "", "properties": { "dnsEndpointType": "Standard", "defaultToOAuthAuthentication": false, "publicNetworkAccess": "Enabled", "keyCreationTime": { "key1": "", "key2": "" }, "allowCrossTenantReplication": false, "privateEndpointConnections": [], "minimumTlsVersion": "TLS1_2", "allowBlobPublicAccess": true, "allowSharedKeyAccess": true, "networkAcls": { "bypass": "AzureServices", "virtualNetworkRules": [], "ipRules": [], "defaultAction": "Allow" }, "supportsHttpsTrafficOnly": true, "encryption": { "requireInfrastructureEncryption": false, "services": { "file": { "keyType": "Account", "enabled": true, "lastEnabledTime": "" }, "blob": { "keyType": "Account", "enabled": true, "lastEnabledTime": "" } }, "keySource": "Microsoft.Storage" }, "accessTier": "Hot", "provisioningState": "Succeeded", "creationTime": "", "primaryEndpoints": { "dfs": "https://cijenkinsioartifacts.dfs.core.windows.net/", "web": "https://cijenkinsioartifacts.z20.web.core.windows.net/", "blob": "https://cijenkinsioartifacts.blob.core.windows.net/", "queue": "https://cijenkinsioartifacts.queue.core.windows.net/", "table": "https://cijenkinsioartifacts.table.core.windows.net/", "file": "https://cijenkinsioartifacts.file.core.windows.net/" }, "primaryLocation": "", "statusOfPrimary": "available" } } ```

Configured the Artifact Management as below:

Expected Results

All of the archiveArtifacts and stash methods should work as expected

Actual Results

Some jobs are failing with the following error stack (and no message on the controller logs):

io.netty.handler.ssl.SslHandshakeTimeoutException: handshake timed out after 10000ms
    at io.netty.handler.ssl.SslHandler$7.run(SslHandler.java:2113)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Also:   java.lang.Exception: #block terminated with an error
            at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
Caused: reactor.core.Exceptions$ReactiveException
    at reactor.core.Exceptions.propagate(Exceptions.java:392)
    at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
    at reactor.core.publisher.Mono.block(Mono.java:1707)
    at com.azure.storage.common.implementation.StorageImplUtils.blockWithOptionalTimeout(StorageImplUtils.java:191)
    at com.azure.storage.blob.BlobClient.uploadFromFileWithResponse(BlobClient.java:473)
    at com.microsoftopentechnologies.windowsazurestorage.service.UploadService$UploadThread.call(UploadService.java:426)
    at com.microsoftopentechnologies.windowsazurestorage.service.UploadService$UploadThread.call(UploadService.java:387)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
Caused: java.util.concurrent.ExecutionException
    at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
    at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
    at com.microsoftopentechnologies.windowsazurestorage.service.UploadService$UploadOnSlave.invoke(UploadService.java:323)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ec2-3-143-84-83.us-east-2.compute.amazonaws.com/3.143.84.83:33447
            at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
            at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
            at hudson.remoting.Channel.call(Channel.java:1000)
            at hudson.FilePath.act(FilePath.java:1192)
            at hudson.FilePath.act(FilePath.java:1181)
            at com.microsoftopentechnologies.windowsazurestorage.service.UploadToBlobService.uploadIndividuals(UploadToBlobService.java:133)
            at com.microsoftopentechnologies.windowsazurestorage.service.UploadService.execute(UploadService.java:527)
            at com.microsoft.jenkins.artifactmanager.AzureArtifactManager.stash(AzureArtifactManager.java:427)
            at org.jenkinsci.plugins.workflow.flow.StashManager.stash(StashManager.java:118)
            at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:119)
            at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:107)
            at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
            at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
            at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.base/java.lang.Thread.run(Thread.java:829)
Caused: java.io.IOException
    at com.microsoftopentechnologies.windowsazurestorage.service.UploadService$UploadOnSlave.invoke(UploadService.java:326)
    at com.microsoftopentechnologies.windowsazurestorage.service.UploadService$UploadOnSlave.invoke(UploadService.java:295)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3578)
    at hudson.remoting.UserRequest.perform(UserRequest.java:211)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:377)
    at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
    at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused: com.microsoftopentechnologies.windowsazurestorage.exceptions.WAStorageException: Fail to upload individual files to blob
    at com.microsoftopentechnologies.windowsazurestorage.service.UploadToBlobService.uploadIndividuals(UploadToBlobService.java:138)
    at com.microsoftopentechnologies.windowsazurestorage.service.UploadService.execute(UploadService.java:527)
    at com.microsoft.jenkins.artifactmanager.AzureArtifactManager.stash(AzureArtifactManager.java:427)
Caused: java.io.IOException
    at com.microsoft.jenkins.artifactmanager.AzureArtifactManager.stash(AzureArtifactManager.java:430)
    at org.jenkinsci.plugins.workflow.flow.StashManager.stash(StashManager.java:118)
    at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:119)
    at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:107)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
[2023-04-14T20:20:29.867Z] [GitHub Checks] GitHub check (name: Jenkins, status: completed) has been published.
[2023-04-14T20:20:29.881Z] Finished: FAILURE

It looks like that any non Azure agent has the same failure. But
- I can't reproduce from a local (to my computer) controller in my own ISP
- The blob endpoint URL is reachable (tested with a curl -v <url>) from the non Azure agent

Anything else?

Public Jenkins Infrastructure issue tracking the tentative on ci.jenkins.io: https://github.com/jenkins-infra/helpdesk/issues/3496
Example of failing build: https://ci.jenkins.io/job/Packaging/job/packaging/job/master/208/

It could be a misconfiguration of course, but I can't find any way to diagnose

timja commented 1 year ago

Is the same build always failing? i.e. I'll need some way to reproduce this to look at it.

I've never tried to run it from outside of Azure or my local machine but it should work just fine.

dduportal commented 1 year ago

Is the same build always failing? i.e. I'll need some way to reproduce this to look at it.

I've never tried to run it from outside of Azure or my local machine but it should work just fine.

Once a build starts to fail (bom builds are the easiest) then it always fail.

I can't reproduce because I can't even make it work locally (issue incoming)

dduportal commented 1 year ago

Good catch @timja : https://github.com/Azure/azure-sdk-for-java/issues/17903#issuecomment-742409657 \o/

timja commented 1 year ago

Is there anything in the server logs when this happens?

dduportal commented 1 year ago

Is there anything in the server logs when this happens?

Alas not, no issue as far as I can tell. We could (should) re-enable the setup (with an announce to developers) and check the threaddump and logs as soon as it happens again, WDYT?

timja commented 1 year ago

yeah sounds good.

jenkinsci / azure-artifact-manager-plugin