Opening multiple connections from a `ConnectionFactory` with `CredentialsRefresher` makes the second connection fail

agross commented 7 months ago

Describe the bug

I'm currently trying to get OAuth working against Keycloak. So far auth works, but as soon as I enable CredentialsRefresher according to the docs I can only open a single connection from the ConnectionFactory. The issue is that the messaging lib I want to use (Wolverine) opens 2 connections to RabbitMQ by default.

Reproduction steps

You probably need a working Keycloak instance prepared according to RabbitMQ's own example.

using RabbitMQ.Client;
using RabbitMQ.Client.OAuth2;

var tokenEndpointUri = new Uri("https://keycloak.home.therightstuff.de/realms/test/protocol/openid-connect/token");
var oAuth2Client = new OAuth2ClientBuilder("producer", "kbOFBXI9tANgKUq8vXHLhT6YhbivgXxn", tokenEndpointUri).Build();
var credentialProvider = new OAuth2ClientCredentialsProvider("prod-uaa-1", oAuth2Client);
var credentialsRefresher = new TimerBasedCredentialRefresher();

var cf = new ConnectionFactory
{
  HostName = "rabbitmq.home.therightstuff.de",
  VirtualHost = "/",
  Ssl = new SslOption("rabbitmq.home.therightstuff.de", null, true),
  CredentialsProvider = credentialProvider,

  CredentialsRefresher = credentialsRefresher,
};

var one = cf.CreateConnection();
var two = cf.CreateConnection(); // Fails as long as CredentialsRefresher is set above.

one.Dispose();
two.Dispose();

Exception:

Unhandled exception. RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
 ---> System.ArgumentException: An item with the same key has already been added. Key: RabbitMQ.Client.OAuth2.OAuth2ClientCredentialsProvider
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at RabbitMQ.Client.TimerBasedCredentialRefresher.Register(ICredentialsProvider provider, NotifyCredentialRefreshed callback)
   at RabbitMQ.Client.Framing.Impl.Connection.MaybeStartCredentialRefresher()
   at RabbitMQ.Client.Framing.Impl.Connection.StartAndTune()
   at RabbitMQ.Client.Framing.Impl.Connection.Open(Boolean insist)
   at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, String clientProvidedName)
   at RabbitMQ.Client.Framing.Impl.Connection..ctor(IConnectionFactory factory, Boolean insist, IFrameHandler frameHandler, ArrayPool`1 memoryPool, String clientProvidedName)
   at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IFrameHandler fh)
   at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IEndpointResolver endpoints)
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
   --- End of inner exception stack trace ---
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName)
   at RabbitMQ.Client.ConnectionFactory.CreateConnection(String clientProvidedName)
   at RabbitMQ.Client.ConnectionFactory.CreateConnection()
   at Program.<Main>$(String[] args) in /Users/agross/Downloads/wolverine/src/Transports/RabbitMQ/RabbitMqBootstrapping/Program.cs:line 28
   at Program.<Main>(String[] args)

Expected behavior

Second connection can be opened.

Additional context

I think the issue is that TimerBasedCredentialRefresher.Register is called for each connection being made. It tries to register the refresher per ICredentialsProvider (there is only one). At the time the second connection is made the first connection's provider is already registered for refreshing.

michaelklishin commented 7 months ago

Nevermind, the exception is sufficiently clear and suggests that the second connection fails due to an exception in TimerBasedCredentialRefresher.