Connecting to Google-hosted Redis with StackExchange not working with AuthString and TLS enabled CA Cert

[YarnUser31]

Hello!

We are using StackExchange.Redis version 2.6.70 in a .NET Core 6.0 project, to connect to a Google Cloud Platform Redis instance (version 5.0.0)(a.k.a Memorystore).

We face issues when we are trying to connect to our instance using the library, while with redis-cli the connection succeeds and we are able to see the contents.

The exception that we get in the code is: exceptionIt was not possible to connect to the redis server(s). There was an authentication failure; check that passwords (or client certificates) are configured correctly: (OpenSslCryptographicException) error:2006D002:BIO routines:BIO_new_file:system lib

P.S: If we disable authentication and TLS in GCP for our Redis instance, the connection through the library succeeds.

[mgravell]

Is the server cert going to be in the automatic trust chain? If it is a private CA, that can be tweaked in code (redis labs works the same, IIRC)

[YarnUser31]

We have single .pem file provided for redis for TLS but i m not aware of internal chain details Also Connection mode = Private service access

We are using below code snippet,but we are not able to connect to redis, Is there any sample code available and is there any setting that we might be missing.

      var configurationOptions = new ConfigurationOptions
        {
            EndPoints = { { "IPAdrress", 6378 } }
            , SslProtocols = System.Security.Authentication.SslProtocols.Tls12
            , Ssl = true,
            Password = "redispassowrd",
            AbortOnConnectFail = false
        };

        configurationOptions.CertificateSelection += delegate
        {
            var certPem = "certtext";                
            return new X509Certificate2(certPem);

        };
        configurationOptions.CertificateValidation += ValidateServerCertificate;
        try
        {                
            var redis = await ConnectionMultiplexer.ConnectAsync(configurationOptions);               
            var database = redis.GetDatabase();
            var cachekey = await database.StringSetAsync("testkey", "testvalue");
          }

    public static bool ValidateServerCertificate(object sender,
    X509Certificate certificate,
    X509Chain chain,
    SslPolicyErrors sslPolicyErrors)
    {
        if (sslPolicyErrors == SslPolicyErrors.None)
            return true;

        Console.WriteLine("Certificate error: {0}", sslPolicyErrors);

        return false;
    }

[YarnUser31]

We tried updating redis version to 6 and now we are getting below error

exceptionTimeout performing PING (5000ms), inst: 0, qu: 1, qs: 0, aw: False, bw: SpinningDown, serverEndpoint: IP:6378, mc: 1/1/0, mgr: 10 of 10 available, clientName: (SE.Redis-v2.6.70.49541), IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=1,Free=32766,Min=2,Max=32767), POOL: (Threads=5,QueuedItems=0,CompletedItems=54474), v: 2.6.70.49541 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

[mgravell]

Did your certificate error / validation code trigger at all?

[YarnUser31]

Nopes, this method ValidateServerCertificate is not getting called. Currently we are seeing timeout error when we try to set a key.

If cert is invalid, does it give timeout error? Is there any other possible reason?

[mgravell]

Very hard to guess I'm afraid. You could try passing a StringWriter into the optional log parameter on connect, to see if anything useful gets recorded, but I suspect I'm going to need to setup a Google endpoint and try connecting with a debugger attached

[YarnUser31]

Even after adding log to connect method, its showing only below log after connect with logs to redis11:16:17.2694: Connecting (async) on .NET 6.0.10 (StackExchange.Redis: v2.6.70.49541)

Can you please help in figuring out the issue with code . Are we missing any setting

[mgravell]

What I need is an endpoint to try connecting to locally. I'm trying to see if I can get any contacts inside Google Memorystore to do this.

On Mon, 14 Nov 2022 at 07:12, YarnUser31 @.***> wrote:

Even after adding log to connect method, its showing only below log after connect with logs to redis11:16:17.2694: Connecting (async) on .NET 6.0.10 (StackExchange.Redis: v2.6.70.49541)

Can you please help in figuring out the issue with code . Are we missing any setting

— Reply to this email directly, view it on GitHub https://github.com/StackExchange/StackExchange.Redis/issues/2299#issuecomment-1313195217, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAEHMEJXNNCY2BXXVBLEKDWIHQ43ANCNFSM6AAAAAAR3YR2YU . You are receiving this because you commented.Message ID: @.***>

-- Regards,

Marc

[mgravell]

Contact has been achieved; we'll see if we can get somewhere with an endpoint so that we can verify and/or advise - and if it is awkward: maybe improve the API a little! Will update as I know more.

[mgravell]

A possible lead, @YarnUser31 - are you using Lettuce here? If so, can you try disabling peer verification, and see if that helps?

[YarnUser31]

We tried adding this configurationOptions.CertificateValidation += ValidateServerCertificate; with always true to avoid cert check but this delegate doesnt get called.

Are you referring to something else with disabling peer verification? we arent using lettuce

[mgravell]

k, I'll feed that back; folks are working on trying to provision a GCP endpoint that I can use to investigate - it is basically impossible for me to do anything more than maybe shrug and grunt without access to something I can try connecting to.

[slorello89]

@YarnUser31 - some of our users had similar issues see: #2219 - this would manifest only on .NET 5+ apps running on linux servers - see issue for full details. This is a shot in the dark but it should be relatively easy to test. As of 2.6.66 you can now set up the ConfigurationOptions.SslClientAuthenticationOptions callback - which allows you to fully configure your SSL options.

var options = new ConfigurationOptions
{
    EndPoints = new EndPointCollection(){"hostname:port"},
    SslClientAuthenticationOptions = host => new SslClientAuthenticationOptions
    {
        EnabledSslProtocols = SslProtocols.Tls12,
        RemoteCertificateValidationCallback = delegate(object sender, X509Certificate? certificate, X509Chain? chain, SslPolicyErrors errors) { return true;},
        CipherSuitesPolicy = new CipherSuitesPolicy(Enum.GetValues<TlsCipherSuite>()),
        LocalCertificateSelectionCallback = delegate(object sender, string targetHost, X509CertificateCollection certificates, X509Certificate? certificate, string[] issuers) { return new X509Certificate2("certText"); }
    }
};

the above is an example of an extremely permissive ConfigurationOptions that I modeled more or less after what you enumerated above

[YarnUser31]

Not sure if we are missing anything but we tried above code and got below error

"It was not possible to connect to the redis server(s). There was an authentication failure; check that passwords (or client certificates) are configured correctly: (ArgumentNullException) Value cannot be null. (Parameter 'TargetHost') "

 var configurationOptions = new ConfigurationOptions
{
   EndPoints = new EndPointCollection() { "hostip:port" },
   Password = "pwds",
   Ssl = true,
   SslClientAuthenticationOptions = host => new SslClientAuthenticationOptions
   {
      EnabledSslProtocols = SslProtocols.Tls12,
      RemoteCertificateValidationCallback = delegate (object sender, X509Certificate? certificate, X509Chain? chain, SslPolicyErrors errors) { return true; },
      CipherSuitesPolicy = new CipherSuitesPolicy(Enum.GetValues<TlsCipherSuite>()),
      LocalCertificateSelectionCallback = delegate (object sender, string targetHost, X509CertificateCollection certificates, X509Certificate? certificate, string[] issuers) { return new X509Certificate2("XUFUDHSDJSHGJexample"); }
   }
};

[slorello89]

Might need to set the TargetHost in the SslClientAuthenticationOptions:

 var configurationOptions = new ConfigurationOptions
{
   EndPoints = new EndPointCollection() { "hostip:port" },
   Password = "pwds",
   Ssl = true,
   SslClientAuthenticationOptions = host => new SslClientAuthenticationOptions
   {
      EnabledSslProtocols = SslProtocols.Tls12,
      RemoteCertificateValidationCallback = delegate (object sender, X509Certificate? certificate, X509Chain? chain, SslPolicyErrors errors) { return true; },
      CipherSuitesPolicy = new CipherSuitesPolicy(Enum.GetValues<TlsCipherSuite>()),
      LocalCertificateSelectionCallback = delegate (object sender, string targetHost, X509CertificateCollection certificates, X509Certificate? certificate, string[] issuers) { return new X509Certificate2("XUFUDHSDJSHGJexample"); },
      TargetHost = host
   }
};

sorry - meant to be an approximation of what you'd need with very permissive parameters, not an exact representation.

[YarnUser31]

Hi, we tried this option and few other things also but we keep on getting timeout error. We contacted google support also and they have also asked to troubleshoot with stackexchange team only. As per them client should support TLS and stackexchange library supports TLS. Were you able to test connecting to redis from GKE?

[YarnUser31]

Hi we tried same code with .net core application hosted on Virtual machine in GCP and we are seeing same timeout issue. We are completely blocked because of this issue and we are not able to use Redis with our application code. Will you be able to help with some working sample for dotnet core with GCP Redis

[mgravell]

I still don't have access to any GCP endpoints myself, so everything here is 3rd hand, however: I received the following example a few days ago. Obviously replace the endpoint and host where specified. Completely untested, because no endpoint!

---

Users have two options:

1. Install the CA Certificate to the X509Store for their host (See https://learn.microsoft.com/en-us/dotnet/api/system.security.cryptography.x509certificates.x509store?view=net-7.0 for details)
2. Override the certificate validation and manually validate the certificate chain.

See below for an example of 2:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using StackExchange.Redis;
using System.Security.Cryptography.X509Certificates;
using System.Net.Security;

namespace StackExchangeRedisDemo
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var configurationOptions = new ConfigurationOptions
            {
                EndPoints = { { "127.0.0.1", 6378 } },
                SslProtocols = System.Security.Authentication.SslProtocols.Tls12,
                Ssl = true,
                SslHost = "127.0.0.1",
            };
            configurationOptions.CertificateValidation += ValidateServerCertificate;
            var redis = ConnectionMultiplexer.Connect(configurationOptions);
            var database = redis.GetDatabase();
            var cachekey = database.StringSet("testkey", "testvalue");
        }

        public static bool ValidateServerCertificate(object sender,
            X509Certificate certificate,
            X509Chain chain,
            SslPolicyErrors sslPolicyErrors)
        {
                if (sslPolicyErrors == SslPolicyErrors.None) {
                    // This will happen if the CA is added to the host's X509Store
                    Console.WriteLine("No policy errors!");
                    return true;
                }
                if (sslPolicyErrors == SslPolicyErrors.RemoteCertificateChainErrors)
                {
                    // If the CA is not in the X509Store we can still do manual validation
                    var certPemFile = "/path/to/the/ca/cert/rootCA.crt";
                    var ca =  new X509Certificate2(certPemFile);
                    chain.ChainPolicy.ExtraStore.Add(ca); // add CA cert for verification, Memorystore certificate chain excludes root by default
                    chain.Build(new X509Certificate2(certificate));
                    var valid = chain.ChainElements[chain.ChainElements.Count - 1].Certificate.Thumbprint == ca.Thumbprint;
                    if (!valid) {
                        Console.WriteLine("Did not match expected CA!");
                    }
                    return valid;
                }

                Console.WriteLine("Certificate error: {0}", sslPolicyErrors);

                return false;
        }
    }
}

[EricStG]

Ran into this with GCP memory store.

We couldn't load the pem on the host's certificate store, so based on the last comment, this is what ended up working (with the latest version of the library)

redisConfiguration.Ssl = true;
redisConfiguration.SslHost = "Redis' IP address" // Needed since we use a custom DNS record to connect
redisConfiguration.SslProtocols = SslProtocols.Tls12 | SslProtocols.Tls13;

var certificate = X509Certificate2.CreateFromPem("pem file from GCP");
redisConfiguration.CertificateSelection += (_, _, _, _, _) => certificate;
redisConfiguration.CertificateValidation += (_, _, chain, errors) =>
{
    if (errors == SslPolicyErrors.None)
    {
        // This will happen if the CA is added to the host's X509Store
        Console.WriteLine("No policy errors!");
        return true;
    }

    if (errors == SslPolicyErrors.RemoteCertificateChainErrors && chain is not null)
    {
        // If the CA is not in the X509Store we can still do manual validation
        chain.ChainPolicy.ExtraStore.Add(certificate); // add CA cert for verification, Memorystore certificate chain excludes root by default
        chain.Build(new X509Certificate2(certificate));
        var valid = chain.ChainElements[^1].Certificate.Thumbprint == certificate.Thumbprint;
        if (!valid)
        {
            Console.WriteLine("Did not match expected CA");
        }
        return valid;
    }

    Console.WriteLine("Certificate error: {0}", errors);
    return false;
};

Note: I'm not 100% sure if CertificateSelection is needed here, but I pull that from https://docs.redis.com/latest/rs/references/client_references/client_csharp/

[johnkors]

Same issue on "Heroku Data for Redis".

How do I configure verify_mode ?

Some details on the Heroku side (or AWS really): https://devcenter.heroku.com/articles/connecting-heroku-redis#connection-permissions