Pub/Sub in Spring application is not able to refresh credentials on the fly

[jarekdrabek]

Hello

I followed the tutorial described here on google site: https://cloud.google.com/pubsub/docs/spring#using-spring-integration-channel-adapters

it is working fine, but what I need now is to rotate my pubsub credentials. For this I use Spring @RefreshScope annotation that will mark the CredentialsProvider bean as the one that needs to be refreshed:

@Bean @RefreshScope public CredentialsProvider gcpPubSubCredentialsProvider(ResourceLoader resourceLoader, @Value("${datacategorization.bigqueryResourceCreation.credentialsLocation}") String credentialsPath) { try { return FixedCredentialsProvider.create(ServiceAccountCredentials .fromStream(resourceLoader.getResource(credentialsPath).getInputStream())); } catch (IOException e) { throw new RuntimeException(String.format("IOException when reading %s file", credentialsPath), e); } }

The CredentialsProvider is renewed with the new credentials correctly. But unfortunately the renewed CredentialsProvider bean is not used on runtime after the subscriber is created. The Subscriber is using the old credentials. As far as I seen the code, while creating subscriber the Credentials instance is retrieved from CredentialsProvider and passed to subscriber. Looks like there is no mechanism for refreshing the credentials in subscriber in the fly. At least I could not find one. You can see it in com.google.api.gax.rpc.ClientContext class. When ClientContext is created the credentials are retrieved:

Credentials credentials = settings.getCredentialsProvider().getCredentials();

Is there a way of making this working? Because as for now, without the possibility of refreshing credentials, the spring integration for PubSub looks like useless for Production purposes.

Spring Cloud GCP Version: 1.2.6.RELEASE

Best Regards Jarek

[meltsufin]

For context, the CredentialsProvider is used by SubscriberFactory, which in turn is used to create the SubscriberStub. The SubscriberStub is what is actually used to pull messages. Alternatively, SubscriberFactory is used directly to create an asynchronous Subscriber. If you add @RefreshScope to SubscriberFactory, you may be able to make the subscribe() method use new credentials. However, I don't think it will work for pull*() methods, because the SubscriberStub is only created once in the constructor. Maybe we can make it a bean to allow refreshing. ...Just some thoughts.

[elefeint]

For the streaming pull, even if PubSubInboundChannelAdapter is instantiated with a refreshable delegate of PubSubTemplate, something managing the lifecycle would still need to call doStop() and doStart().

@artembilan Do Spring Integration MessageProducerSupport.doStart() and doStop() integrate with Spring Cloud Config refreshable context in any way? Or does the end-user application need to listen to context refresh event and manually restart the inbound adapter?

[elefeint]

@jarekdrabek The solution to this has two parts: 1) using spring.cloud.refresh.extra-refreshable property to turn Spring Cloud GCP beans involved into refresh-scoped beans.

spring.cloud.refresh.extra-refreshable=org.springframework.cloud.gcp.pubsub.support.SubscriberFactory,\
  org.springframework.cloud.gcp.pubsub.core.subscriber.PubSubSubscriberTemplate

2) Restarting the Spring Application adapter when RefreshScopeRefreshedEvent is detected.

  @Autowired
  private PubSubInboundChannelAdapter pubSubAdapter;

  @EventListener(RefreshScopeRefreshedEvent.class)
  public void onRefreshScope(RefreshScopeRefreshedEvent event) {
    this.pubSubAdapter.stop();
    this.pubSubAdapter.start();
  }

Sample application demonstrating these in combination: https://github.com/elefeint/examples/tree/master/pubsub-rotate-credentials

[meltsufin]

@elefeint Is this something we should integrate into the project core?

[elefeint]

@meltsufin I don't think so, since the most common scenario is to initialize adapters once and then leave them running. This would be the case for default credentials (deriving credentials from runtime environment) and service accounts in cases where it's okay to restart the app when credentials change.

However, it's also not a one-off -- credential rotation is common, too -- so this seems useful to document.