Spring Cloud GCP not working with GKE Workload Identity

[devil47sid]

Description: Google Auth libraries unable to use credentials set by GKE workload identity. Version: Spring boot 2.3.3.RELEASE, Spring Cloud Hoxton.SR8

Steps to reproduce: Deploy any app using default google cloud credentials in Workload Identity enabled GKE cluster

Stacktrace:

should be used if your app uses services that 
  require credentials. java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. at 
  com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:134) at 
  com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:119) at 
  com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:91) at 
  com.google.api.gax.core.GoogleCredentialsProvider.getCredentials(GoogleCredentialsProvider.java:67) at 
  org.springframework.cloud.gcp.core.DefaultCredentialsProvider.<init>(DefaultCredentialsProvider.java:110) at 
  org.springframework.cloud.gcp.autoconfigure.core.GcpContextAutoConfiguration.googleCredentials(GcpContextAutoConfiguration.java:61) at 
  org.springframework.cloud.gcp.autoconfigure.core.GcpContextAutoConfiguration$$EnhancerBySpringCGLIB$$26345ff6.CGLIB$googleCredentials$1(<generated>) at 
  org.springframework.cloud.gcp.autoconfigure.core.GcpContextAutoConfiguration$$EnhancerBySpringCGLIB$$26345ff6$$FastClassBySpringCGLIB$$df41ff71.invoke(<generated>) at 
  org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) at 
  org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) at 
  org.springframework.cloud.gcp.autoconfigure.core.GcpContextAutoConfiguration$$EnhancerBySpringCGLIB$$26345ff6.googleCredentials(<generated>) at 
  java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
  java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at 
  java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at 
  java.base/java.lang.reflect.Method.invoke(Method.java:564) at 
  org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) at 
  org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:650) at 
  org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:483) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1176) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at 
  org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at 
  org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at 
  org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at 
  org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) at 
  org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:227) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1356) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1203) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at 
  org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at 
  org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:408) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1176) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at 
  org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at 
  org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at 
  org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at 
  org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) at 
  org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:538) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1176) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at 
  org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at 
  org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at 
  org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at 
  org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) at 
  org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:538) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1176) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at 
  org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at 
  org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at 
  org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) at 
  org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) at 
  org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:538) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1176) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at 
  org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at 
  org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) at 
  org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640) at 
  org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:130) at 
  org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1420) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) at 
  org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) at 
  org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) at 
  org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) at 
  org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) at 
  org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) at 
  org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) at 
  org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:62) at 
  org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) at 
  org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) at 
  org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) at 
  org.springframework.boot.SpringApplication.run(SpringApplication.java:315) at 
  org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) at 
  org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) at 
  com.globalpay.enterprise.integrations.frontend.FrontEndApplication.main(FrontEndApplication.java:11) at 
  java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
  java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at 
  java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at 
  java.base/java.lang.reflect.Method.invoke(Method.java:564) at 
  org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) at 
  org.springframework.boot.loader.Launcher.launch(Launcher.java:109) at 
  org.springframework.boot.loader.Launcher.launch(Launcher.java:58) at 
  org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

[saturnism]

Hi @devil47sid - Wanted to check if you can validate all the instructions needed to enable Workload Identity in the cluster ( https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#enable_on_cluster

I've ran into similar issues in the past and it most likely can be one of the following:

1. enable IAM credentials API (I missed this one in the past)
2. make sure there is a Google service account (GSA), Kubernetes service account (KSA), and the binding annotation in the KSA
3. Lastly, make sure the deployment/pod/workload is associated w/ the KSA.

To be comprehensive, pls see this doc for more troubleshooting steps: https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#troubleshooting

[elefeint]

I've validated workload identity with default credentials in GKE as part of doing some unrelated testing. I've encountered issues similar to Ray's point (3.) -- when running create deployment, the default namespace is used, unless the yaml file contains a namespace specification. This makes creating deployments with kubectl create deployment subtly different from using kubectl run as the workload identity tutorial does, since the latter takes a --serviceaccount parameter, while the former does not.

Closing this issue; please reopen if you are still having trouble.