Azure / azure-acr-plugin

Jenkins Azure ACR plugin
MIT License
2 stars 10 forks source link

StackOverflowError when reading ACR Quick Task logs for a local code directory upload #44

Open jacobcsmith opened 4 years ago

jacobcsmith commented 4 years ago

Our builds are failing while waiting for the ACR quick task to complete. Below is an example stack trace of the failure:

java.lang.StackOverflowError
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.<init>(HttpURLConnection.java:876)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.<init>(AbstractDelegateHttpsURLConnection.java:55)
    at sun.net.www.protocol.https.DelegateHttpsURLConnection.<init>(DelegateHttpsURLConnection.java:64)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.<init>(HttpsURLConnectionImpl.java:100)
    at sun.net.www.protocol.https.Handler.openConnection(Handler.java:62)
    at sun.net.www.protocol.https.Handler.openConnection(Handler.java:57)
    at java.net.URL.openConnection(URL.java:1002)
    at com.microsoft.azure.storage.core.BaseRequest.createURLConnection(BaseRequest.java:185)
    at com.microsoft.azure.storage.core.BaseRequest.getProperties(BaseRequest.java:299)
    at com.microsoft.azure.storage.blob.BlobRequest.getProperties(BlobRequest.java:799)
    at com.microsoft.azure.storage.blob.BlobRequest.getBlobProperties(BlobRequest.java:578)
    at com.microsoft.azure.storage.blob.CloudBlob$8.buildRequest(CloudBlob.java:1256)
    at com.microsoft.azure.storage.blob.CloudBlob$8.buildRequest(CloudBlob.java:1246)
    at com.microsoft.azure.storage.core.ExecutionEngine.setupStorageRequest(ExecutionEngine.java:303)
    at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:80)
    at com.microsoft.azure.storage.blob.CloudBlob.downloadAttributes(CloudBlob.java:1239)
    at com.microsoft.azure.storage.blob.BlobInputStream.<init>(BlobInputStream.java:155)
    at com.microsoft.azure.storage.blob.CloudBlob.openInputStream(CloudBlob.java:2383)
    at com.microsoft.azure.storage.blob.CloudBlob.openInputStream(CloudBlob.java:2347)
    at com.microsoft.jenkins.acr.service.AzureStorageAppendBlob.getStream(AzureStorageAppendBlob.java:150)
    at com.microsoft.jenkins.acr.service.AzureStorageAppendBlob.createReader(AzureStorageAppendBlob.java:123)
    at com.microsoft.jenkins.acr.service.AzureStorageAppendBlob.readLine(AzureStorageAppendBlob.java:72)
...many more repeated lines

Of note, the quick task is still executing after the Jenkins job failure and additional quick task log file output is available when viewed in the Azure portal.

Below is some additional context and output for our environment.

Relevant line in Jenkinsfile: acrQuickTask azureCredentialsId : env.AZURE_CRED_ID, imageNames : [[image: "$env.ACR_NAME/$env.IMAGE_NAME:$env.BUILD_NUMBER"]], os: 'Windows', registryName: env.ACR_NAME, resourceGroupName: env.ACR_RESOURCE_GRP, variant: '', timeout: 2400

This project is building a legacy ASP.NET application. Since the code is sensitive, we utilize a private git server and use the local directory upload to submit the code to the ACR quick task. There are a large number of files inside the uploaded directory (~14,000) and I suspect that may be the reason for the SO error.

Dockerfile:

WORKDIR /app
COPY *.sln .
COPY APPNAME_Web/*.csproj ./APPNAME_Web/
COPY APPNAME_Web/*.config ./APPNAME_Web/
RUN nuget restore
COPY APPNAME_Web/. ./APPNAME_Web/ 
WORKDIR /app/
RUN msbuild /ds /p:Configuration=Release /v:d
FROM mcr.microsoft.com/dotnet/framework/aspnet:4.6.2 AS runtime
WORKDIR /inetpub/wwwroot 
COPY --from=build /app/APPNAME_Web/. ./

Additional Jenkins build log output:

Upload local directory to Azure Storage Blob [URL-REDACTED]
Build image with source code from source/202003120000/***-*****-*****.tar.gz.
Queue a task to ACR APP/APPCONTAINERREG - {"sourceType":"local","localDir":".","imageNames":["[APP]/[APPNAME]:32"],"buildArguments":[],"noCache":false,"timeout":1200,"platform":{"os":"Windows","architecture":"AMD64","variant":""},"dockerFilePath":"Dockerfile","canceled":false,"sourceUrl":"source/202003120000/****-*****-****.tar.gz"}
Finished queue an ACR Quick Task: ha15
Get log link: [URL-REDACTED]
2020/03/12 15:43:54 Downloading source code...
2020/03/12 15:44:08 Finished downloading source code
2020/03/12 15:44:10 Using acb_vol_******** as the home volume
2020/03/12 15:44:10 Setting up Docker configuration...
2020/03/12 15:44:20 Successfully set up Docker configuration
2020/03/12 15:44:20 Logging in to registry: [REDACTED]
2020/03/12 15:44:25 Successfully logged into [REDACTED]
2020/03/12 15:44:25 Executing step ID: build. Timeout(sec): 28800, Working directory: '', Network: ''
2020/03/12 15:44:25 Scanning for dependencies...
2020/03/12 15:44:29 Successfully scanned dependencies
2020/03/12 15:44:29 Launching container with name: build
Sending build context to Docker daemon  208.8MB

Step 1/15 : FROM mcr.microsoft.com/dotnet/framework/sdk:3.5 AS build
3.5: Pulling from dotnet/framework/sdk
9038b92872bc: Already exists
9bc3230fc78d: Pulling fs layer
c4e1b99d47db: Pulling fs layer
7e66007ae5f5: Pulling fs layer
a1b715772539: Pulling fs layer
05b1b1846f1b: Pulling fs layer
c577d5e5e968: Pulling fs layer
28aae10e81c3: Pulling fs layer
e3e37d592cba: Pulling fs layer
8a5675911bab: Pulling fs layer
a07840d11568: Pulling fs layer
e47d75bb6bf2: Pulling fs layer
3314250cfc3c: Pulling fs layer
d385150cb370: Pulling fs layer
5f5d2c3dd882: Pulling fs layer
a1b715772539: Waiting
05b1b1846f1b: Waiting
c577d5e5e968: Waiting
28aae10e81c3: Waiting
e3e37d592cba: Waiting
8a5675911bab: Waiting
a07840d11568: Waiting
e47d75bb6bf2: Waiting
3314250cfc3c: Waiting
d385150cb370: Waiting
5f5d2c3dd882: Waiting
c4e1b99d47db: Verifying Checksum
c4e1b99d47db: Download complete
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (QA test)
Stage "QA test" skipped due to earlier failure(s)

We are exploring ways of trimming down the number of files submitted to ACR using .dockerfileignore but I'm wondering if there is a way to prevent this error regardless of the number of files.

Thanks for your help.

omarquina-areuin commented 4 years ago

Hi, I have the same issue, but building a reactjs image. Do you have any clue about this issue?. Is it about the size of the image pushing to ACR?.