Intermittent AccessDeniedException on Windows moving file from AppData\Local\Temp to target\jib-cache\selectors

[jdoylei]

Environment:

• Jib version: 3.4.0
• Build tool: Apache Maven 3.8.4
• OS: windows 10

Description of the issue:

We have 20+ Jib Maven projects that normally build successfully on our Windows laptops, but intermittently (every 10 builds or less) fail with AccessDeniedException indicating a file move from the user's tmpdir to the Jib application cache. Please see the Maven log excerpt below with the stack trace (using mvn -X). The failure happens to us often when using Maven Jib from "skaffold dev", due to the large number of builds done as code is changed.

Expected behavior:

Builds succeed consistently.

Steps to reproduce:

Run mvn -X -l jib.log -pl :<child module name> -am -DskipTests package jib:build -Djib.containerize=:<child module name> up to 10 times in Maven root project dir.

jib-maven-plugin Configuration:

                <configuration>
                    <from>
                        <image>private.reg.io/ubi8/ubi-micro</image>
                    </from>
                    <to>
                        <image>private.reg.io/${project.artifactId}:${project.version}</image>
                    </to>
                    <container>
                        <entrypoint>INHERIT</entrypoint>
                        <appRoot>/deployments/${project.artifactId}.war</appRoot>
                    </container>
                    <skaffold>
                        <watch>
                            <includes>
                                <include>
                                    src/main/webapp
                                </include>
                            </includes>
                        </watch>
                    </skaffold>
                </configuration>

Log output:

Caused by: java.nio.file.AccessDeniedException: C:\Users\xxx\AppData\Local\Temp\13220281918294591275.tmp -> C:\Users\xxx\git\fork\prj-fork\prj-web\target\jib-cache\selectors\03c4a7469349f7d32e62b59980815fa56031a31f405e9490fd6ec3d28d51770c
    at sun.nio.fs.WindowsException.translateToIOException (WindowsException.java:89)
    at sun.nio.fs.WindowsException.rethrowAsIOException (WindowsException.java:103)
    at sun.nio.fs.WindowsFileCopy.move (WindowsFileCopy.java:317)
    at sun.nio.fs.WindowsFileSystemProvider.move (WindowsFileSystemProvider.java:293)
    at java.nio.file.Files.move (Files.java:1432)
    at com.google.cloud.tools.jib.cache.CacheStorageWriter.writeSelector (CacheStorageWriter.java:471)
    at com.google.cloud.tools.jib.cache.CacheStorageWriter.writeUncompressed (CacheStorageWriter.java:300)
    at com.google.cloud.tools.jib.cache.Cache.writeUncompressedLayer (Cache.java:145)
    at com.google.cloud.tools.jib.builder.steps.BuildAndCacheApplicationLayerStep.call (BuildAndCacheApplicationLayerStep.java:108)
    at com.google.cloud.tools.jib.builder.steps.BuildAndCacheApplicationLayerStep.call (BuildAndCacheApplicationLayerStep.java:38)
    at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:131)
    at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:75)
    at com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:82)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
    at java.lang.Thread.run (Thread.java:840)

Additional Information:

This looks like a combination of issue #1635 and #3572. It is happening to us with AccessDeniedException (rather than e.g. FileAlreadyExistsException) and in jib-cache/selectors rather than in jib-cache/layers.

We tried the following with no success:

• Changing the application cache location (the move destination)
• Changing the tmpdir location (the move source - MAVEN_OPTS=-Djava.io.tmpdir=...)
• Closing all apps and signing out and signing back in

I think that multiple places in the Jib code were updated to use moveIfDoesNotExist and its retry logic, e.g. the code for jib-cache/layers, but not writeSelector and jib-cache/selectors. Could writeSelector use the same moveIfDoesNotExist retry approach?

[chanseokoh]

writeSelector() could retry, but unlike moveIfDoesNotExist(), I think it should overwrite the content if the file already exists.

[jdoylei]

@chanseokoh - thanks for taking a look! I see you what you mean now - that writeSelector requires different logic from moveIfDoesNotExist. I'm thinking retry would help writeSelector, because we are only seeing the AccessDeniedException on the selectors directory and not anywhere else (like layers), so if retry is used for the layers directory, it's probably helping there.

[chanseokoh]

Yeah, I agree.

[jdoylei]

@chanseokoh, is there any chance this might be addressed in an upcoming release? We've started another round of development this month where the developers were hitting this issue frequently again.