GoogleContainerTools / jib

🏗 Build container images for your Java applications.
Apache License 2.0
13.6k stars 1.43k forks source link

Cannot invoke "com.google.auth.oauth2.AccessToken.getTokenValue()" because "accessToken" is null #4140

Open wleese opened 10 months ago

wleese commented 10 months ago

Environment:

Description of the issue: Every once in a while pushing a container fails with:

Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (default-cli) on project xxxx-app: Cannot invoke "com.google.auth.oauth2.AccessToken.getTokenValue()" because "accessToken" is null -> [Help 1]

This happens with multiple different applications, with no discernible pattern.

Full stack trace, with debug logging:

278403 [INFO] Getting manifest for base image europe-west4-docker.pkg.dev/xxxxx/images/xxx-runtime-jre...
278404 [DEBUG] TIMING   Authenticating push to europe-west4-docker.pkg.dev
278406 [DEBUG] TIMING   trying mirrors
278406 [DEBUG] TIMED    trying mirrors : 0.237 ms
278412 [DEBUG] TIMING   Preparing application layer builders
278412 [DEBUG] TIMED    Preparing application layer builders : 0.571 ms
278417 [INFO] Building dependencies layer...
278417 [DEBUG] TIMING   Building dependencies layer
278432 [INFO] Building resources layer...
278432 [DEBUG] TIMING   Building resources layer
278437 [INFO] Building classes layer...
278437 [DEBUG] TIMING   Building classes layer
278458 [INFO] Building jvm arg files layer...
278459 [DEBUG] TIMING   Building jvm arg files layer
278470 [INFO] Building axle-build-metrics layer...
278471 [DEBUG] TIMING   Building axle-build-metrics layer
278556 [DEBUG] Building axle-build-metrics layer built sha256:03de117a76e695557f8751bbfe0be5c5901dc75e0ca22c17921996f77cb0cfc6
278557 [DEBUG] TIMED    Building axle-build-metrics layer : 86.564 ms
278562 [DEBUG] Building jvm arg files layer built sha256:f3e46ee7fd2d25740654fb5ac57d12b1eff5b9b485a8a217e4f81295b7f27bdc
278562 [DEBUG] TIMED    Building jvm arg files layer : 103.838 ms
278665 [DEBUG] Google ADC found
278794 [INFO] The base image requires auth. Trying again for europe-west4-docker.pkg.dev/xxxx/images/xx-runtime-jre...
278801 [DEBUG] Google ADC found
278847 [INFO] Using Google Application Default Credentials for europe-west4-docker.pkg.dev/xxxx/images/xxx-runtime-jre
278847 [DEBUG] TIMED    Pulling base image manifest : 443.66 ms
278847 [INFO] Using Google Application Default Credentials for europe-west4-docker.pkg.dev/xxx/images/xxx-app:3100628-main-9eb25e86
278849 [DEBUG] TIMED    Building and pushing image : 634.652 ms
278849 [DEBUG] attempting bearer auth for europe-west4-docker.pkg.dev/xxx/images/xxx-app...
278849 [INFO] 
278851 [INFO] ------------------------------------------------------------------------
278851 [INFO] Reactor Summary for xxx 3100628-main-9eb25e86:
278851 [INFO] 
278852 [INFO] xxx ............................................. SUCCESS [  3.964 s]
278852 [INFO] xxx-app ......................................... FAILURE [04:32 min]
278852 [INFO] ------------------------------------------------------------------------
278852 [INFO] BUILD FAILURE
278852 [INFO] ------------------------------------------------------------------------
278852 [INFO] Total time:  04:37 min
278852 [INFO] Finished at: 2023-11-13T11:45:45+01:00
278852 [INFO] ------------------------------------------------------------------------
278853 [ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (default-cli) on project xxx-app: Cannot invoke "com.google.auth.oauth2.AccessToken.getTokenValue()" because "accessToken" is null -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (default-cli) on project xxx-app: Cannot invoke "com.google.auth.oauth2.AccessToken.getTokenValue()" because "accessToken" is null
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:333)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:52)
    at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:161)
    at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:73)
Caused by: org.apache.maven.plugin.MojoExecutionException: Cannot invoke "com.google.auth.oauth2.AccessToken.getTokenValue()" because "accessToken" is null
    at com.google.cloud.tools.jib.maven.BuildImageMojo.execute (BuildImageMojo.java:181)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:52)
    at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:161)
    at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:73)
Caused by: java.lang.NullPointerException: Cannot invoke "com.google.auth.oauth2.AccessToken.getTokenValue()" because "accessToken" is null
    at com.google.cloud.tools.jib.frontend.CredentialRetrieverFactory.lambda$googleApplicationDefaultCredentials$3 (CredentialRetrieverFactory.java:284)
    at com.google.cloud.tools.jib.builder.steps.RegistryCredentialRetriever.retrieve (RegistryCredentialRetriever.java:49)
    at com.google.cloud.tools.jib.builder.steps.RegistryCredentialRetriever.getBaseImageCredential (RegistryCredentialRetriever.java:36)
    at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call (PullBaseImageStep.java:167)
    at com.google.cloud.tools.jib.builder.steps.PullBaseImageStep.call (PullBaseImageStep.java:70)
    at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly (TrustedListenableFutureTask.java:131)
    at com.google.common.util.concurrent.InterruptibleTask.run (InterruptibleTask.java:75)
    at com.google.common.util.concurrent.TrustedListenableFutureTask.run (TrustedListenableFutureTask.java:82)
    at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1144)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
    at java.lang.Thread.run (Thread.java:1583)

Retrying usually results in success.

No changes were made to the build setup, or any changes known about the underlying infrastructure (GitLab Runners).

Expected behavior:

Container is published.

Steps to reproduce:

The problem happens every once in awhile.

Additional Information:

Looking at the code:

 public CredentialRetriever googleApplicationDefaultCredentials() {
    return () -> {
      try {
        if (imageReference.getRegistry().endsWith("gcr.io")
            || imageReference.getRegistry().endsWith("docker.pkg.dev")) {
          GoogleCredentials googleCredentials = googleCredentialsProvider.get();
          logger.accept(LogEvent.info("Google ADC found"));
          if (googleCredentials.createScopedRequired()) { // not scoped if service account
            // The short-lived OAuth2 access token to be generated from the service account with
            // refreshIfExpired() below will have one-hour expiry (as of Aug 2019). Instead of using
            // an access token, it is technically possible to use the service account private key to
            // auth with GCR, but it does not worth writing complex code to achieve that.
            logger.accept(LogEvent.info("ADC is a service account. Setting GCS read-write scope"));
            List<String> scope = Collections.singletonList(OAUTH_SCOPE_STORAGE_READ_WRITE);
            googleCredentials = googleCredentials.createScoped(scope);
          }
          googleCredentials.refreshIfExpired();

          logGotCredentialsFrom("Google Application Default Credentials");
          AccessToken accessToken = googleCredentials.getAccessToken();
          // https://cloud.google.com/container-registry/docs/advanced-authentication#access_token
          return Optional.of(Credential.from("oauth2accesstoken", accessToken.getTokenValue()));
        }

      } catch (IOException ex) { // Includes the case where ADC is simply not available.
        logger.accept(
            LogEvent.info("ADC not present or error fetching access token: " + ex.getMessage()));
      }
      return Optional.empty();
    };
  }

It seems that for some reason googleCredentials.getAccessToken(); returns null. I don't know why, but at the very least it's probably a good idea to catch the NPE.

We also have some logs of how the request for authentication / the access token, was made:

{
    "insertId": "1f2bp76dsb93",
    "logName": "projects/xxx/logs/cloudaudit.googleapis.com%2Fdata_access",
    "operation": {
        "first": true,
        "id": "4898467374120538801",
        "last": true,
        "producer": "iamcredentials.googleapis.com"
    },
    "protoPayload": {
        "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
        "authenticationInfo": {
            "principalSubject": "serviceAccount:xxx.svc.id.goog[xx/containerbuilder]",
            "serviceAccountDelegationInfo": [
                {}
            ]
        },
        "authorizationInfo": [
            {
                "granted": true,
                "permission": "iam.serviceAccounts.getAccessToken",
                "resourceAttributes": {}
            }
        ],
        "metadata": {
            "identityDelegationChain": [
                "projects/-/serviceAccounts/containerbuilder@xxx.iam.gserviceaccount.com"
            ]
        },
        "methodName": "GenerateAccessToken",
        "request": {
            "@type": "type.googleapis.com/google.iam.credentials.v1.GenerateAccessTokenRequest",
            "name": "projects/-/serviceAccounts/containerbuilder@xxxx.iam.gserviceaccount.com"
        },
        "requestMetadata": {
            "callerIp": "gce-internal-ip",
            "callerSuppliedUserAgent": "google-api-go-client/0.5 gke-metadata-server,gzip(gfe)",
            "destinationAttributes": {},
            "requestAttributes": {
                "auth": {},
                "time": "2023-11-13T10:45:45.816012913Z"
            }
        },
        "resourceName": "projects/-/serviceAccounts/xxx",
        "serviceName": "iamcredentials.googleapis.com",
        "status": {}
    },
    "receiveTimestamp": "2023-11-13T10:45:47.137195471Z",
    "resource": {
        "labels": {
            "email_id": "containerbuilder@xxx.iam.gserviceaccount.com",
            "project_id": "xxx",
            "unique_id": "xxx"
        },
        "type": "service_account"
    },
    "severity": "INFO",
    "timestamp": "2023-11-13T10:45:45.806510333Z"
}

And the response, while I don't have the entire payload, we do have some metadata:

     "authorizationInfo": [
        {
          "granted": true,
          "resourceAttributes": {},
          "permission": "iam.serviceAccounts.getAccessToken"
        }
      ],
      "requestMetadata": {
        "callerIp": "gce-internal-ip",
        "requestAttributes": {
          "auth": {},
          "time": "2023-11-13T10:45:45.816012913Z"
        },
        "destinationAttributes": {},
        "callerSuppliedUserAgent": "google-api-go-client/0.5 gke-metadata-server,gzip(gfe)"
      },
      "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
      "authenticationInfo": {
        "serviceAccountDelegationInfo": [
          {}
        ],
        "principalSubject": "serviceAccount:xxxx.svc.id.goog[xxxx/containerbuilder]"
      },
wleese commented 10 months ago

Looking at the log snippets I posted, specifically:

      "requestMetadata": {
        "callerIp": "gce-internal-ip",
        "requestAttributes": {
          "auth": {},
          "time": "2023-11-13T10:45:45.816012913Z"
        },
        "destinationAttributes": {},
        "callerSuppliedUserAgent": "google-api-go-client/0.5 gke-metadata-server,gzip(gfe)"
      },

.. in relation to jib-maven-plugin, I assume this is a transitive call triggered by jib authenticating, rather than a direct call, as evident from the callerSuppliedUserAgent. In other words, we have no clear record of the request jib-maven-plugin made.

Xander-Polishchuk commented 7 months ago

We're having exactly the same issue running on Github with authorization via google-github-actions/auth@v1 workload identity. It happens every once in a while.

Wallman commented 4 months ago

We're having exactly the same issue running on Github with authorization via google-github-actions/auth@v1 workload identity. It happens every once in a while.

+1 but for google-github-actions/auth@v2 and jib-maven-plugin:3.4.2

Wallman commented 4 months ago

@JoeWang1127 any timeline on this? Thanks

mpeddada1 commented 4 months ago

Thanks so much for the detailed explanation of the behavior! Trying to see if we can reproduce this locally. Do you happen to have a small reproducer or details on how you've configured you POM?

Wallman commented 4 months ago

@wleese @Xander-Polishchuk Are you using OIDC as auth mechanism to GCP? We are.

Xander-Polishchuk commented 4 months ago

@Wallman yep

Wallman commented 4 months ago

Thanks so much for the detailed explanation of the behavior! Trying to see if we can reproduce this locally. Do you happen to have a small reproducer or details on how you've configured you POM?

@mpeddada1 I created a simple project to replicate the issue, but I have not been able to replicate it. Since it is a flaky error, it's hard to know why. We are still experiencing the error from time to time in our company CI, but have currently solved it with an automatic retry.

Anyone feel free to add to the project if you have ideas of what could increase the chance of replicating the issue.

mpeddada1 commented 3 months ago

Thanks for creating the sample @Wallman. I will try this out to see how often this can be replicated. Additionally, at the very least, a better error message instead of throwing a NullPointerException for when accessToken is null may be something we want to do here (as mentioned in the original description).

For the time being, the workaround would be set up retries as you are currently doing.

Wallman commented 2 months ago

We are still experiencing this issue, both in GitHub Actions and CircleCI

maxmil commented 1 month ago

We are also experiencing this issue. Github actions with jib 3.4.3.

Note that we have a multi module build with two modules that are being containerised and pushed concurrently. We have examples of one of the pushes failing but the other succeeding.

svametcalf commented 2 weeks ago

We also just experienced this issue. For context, we were running a build on GitLabCI with a remote gradle cache in parallel, with the runner deployed in GKE using Workload Identity for auth.

Here is the stacktrace:

org.gradle.api.tasks.TaskExecutionException: Execution failed for task <redacted>.
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:148)
    at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:146)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:134)
    at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
    at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)
    at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
    at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:331)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:318)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.lambda$execute$0(DefaultTaskExecutionGraph.java:314)
    at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:80)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:314)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
    at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
Caused by: org.gradle.internal.UncheckedException: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Cannot invoke "com.google.auth.oauth2.AccessToken.getTokenValue()" because "accessToken" is null
    at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:68)
    at org.gradle.internal.UncheckedException.throwAsUncheckedException(UncheckedException.java:41)
    at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:128)
    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
    at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
    at org.gradle.api.internal.tasks.execution.TaskExecution$3.run(TaskExecution.java:248)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:47)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:68)
    at org.gradle.api.internal.tasks.execution.TaskExecution.executeAction(TaskExecution.java:233)
    at org.gradle.api.internal.tasks.execution.TaskExecution.executeActions(TaskExecution.java:216)
    at org.gradle.api.internal.tasks.execution.TaskExecution.executeWithPreviousOutputFiles(TaskExecution.java:199)
    at org.gradle.api.internal.tasks.execution.TaskExecution.execute(TaskExecution.java:166)
    at org.gradle.internal.execution.steps.ExecuteStep.executeInternal(ExecuteStep.java:105)
    at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:44)
    at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:59)
    at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:56)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
    at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:56)
    at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:44)
    at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:67)
    at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:37)
    at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:41)
    at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:74)
    at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:55)
    at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:50)
    at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:28)
    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.executeDelegateBroadcastingChanges(CaptureStateAfterExecutionStep.java:100)
    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:72)
    at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:50)
    at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:40)
    at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:29)
    at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:179)
    at org.gradle.internal.execution.steps.BuildCacheStep.lambda$execute$1(BuildCacheStep.java:70)
    at org.gradle.internal.Either$Right.fold(Either.java:175)
    at org.gradle.internal.execution.caching.CachingState.fold(CachingState.java:59)
    at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:68)
    at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:46)
    at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:36)
    at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:25)
    at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:36)
    at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:22)
    at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:91)
    at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$2(SkipUpToDateStep.java:55)
    at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)
    at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:37)
    at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:65)
    at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:36)
    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
    at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:76)
    at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:37)
    at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:108)
    at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:55)
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:71)
    at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:45)
    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.executeWithNonEmptySources(SkipEmptyWorkStep.java:177)
    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:81)
    at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:53)
    at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:32)
    at org.gradle.internal.execution.steps.RemoveUntrackedExecutionStateStep.execute(RemoveUntrackedExecutionStateStep.java:21)
    at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
    at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:36)
    at org.gradle.internal.execution.steps.LoadPreviousExecutionStateStep.execute(LoadPreviousExecutionStateStep.java:23)
    at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:75)
    at org.gradle.internal.execution.steps.CleanupStaleOutputsStep.execute(CleanupStaleOutputsStep.java:41)
    at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.lambda$execute$2(ExecuteWorkBuildOperationFiringStep.java:66)
    at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.execute(ExecuteWorkBuildOperationFiringStep.java:66)
    at org.gradle.internal.execution.steps.ExecuteWorkBuildOperationFiringStep.execute(ExecuteWorkBuildOperationFiringStep.java:38)
    at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:32)
    at org.gradle.api.internal.tasks.execution.TaskExecution$4.withWorkspace(TaskExecution.java:293)
    at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:30)
    at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:21)
    at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:37)
    at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:27)
    at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:47)
    at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:34)
    at org.gradle.internal.execution.impl.DefaultExecutionEngine$1.execute(DefaultExecutionEngine.java:64)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:145)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:134)
    at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
    at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74)
    at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66)
    at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59)
    at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53)
    at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73)
    at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
    at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:331)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:318)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.lambda$execute$0(DefaultTaskExecutionGraph.java:314)
    at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:80)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:314)
    at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463)
    at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380)
    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
    at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47)
Caused by: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Cannot invoke "com.google.auth.oauth2.AccessToken.getTokenValue()" because "accessToken" is null
    at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild(JibBuildRunner.java:285)
    at com.google.cloud.tools.jib.gradle.BuildImageTask.buildImage(BuildImageTask.java:122)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
    ... 118 more
Caused by: java.lang.NullPointerException: Cannot invoke "com.google.auth.oauth2.AccessToken.getTokenValue()" because "accessToken" is null
    at com.google.cloud.tools.jib.frontend.CredentialRetrieverFactory.lambda$googleApplicationDefaultCredentials$3(CredentialRetrieverFactory.java:284)
    at com.google.cloud.tools.jib.builder.steps.RegistryCredentialRetriever.retrieve(RegistryCredentialRetriever.java:49)
    at com.google.cloud.tools.jib.builder.steps.RegistryCredentialRetriever.getTargetImageCredential(RegistryCredentialRetriever.java:42)
    at com.google.cloud.tools.jib.builder.steps.AuthenticatePushStep.call(AuthenticatePushStep.java:58)
    at com.google.cloud.tools.jib.builder.steps.AuthenticatePushStep.call(AuthenticatePushStep.java:35)
    at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
    at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
    at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)

It did seem to show up when two containers were being pushed. After retrying the build a number of times, it finally passed. When it did fail, it did seem to be midway through the push and it was inconsistent on which layer / image it failed on.

If there is any other troubleshooting information that would be hefpful for the next time this happens to someone, please let us know.

Wallman commented 6 days ago

Even though we use three retries, the problem was too common to let be. So we have now resorted to using jib:dockerBuild and a separate docker push. Hopefully this will remove our issues.

maxmil commented 6 days ago

We changed the build to only push one container per jib task run and this worked around the problem.

Wallman commented 6 days ago

We changed the build to only push one container per jib task run and this worked around the problem.

Hmm could you elaborate a bit? Isn't pushing one image the standard?

maxmil commented 6 days ago

We have a multi module build where several modules have a jib task. Running gradlew jib ran these tasks concurrently. However running them sequentially by running gradlew :module1:jib, gradlew :module2:jib etc made the build a bit slower but avoided the problem.