The library doesn't seems to take consideration of HTTP_PROXY or HTTPS_PROXY

[ygalbel]

These two environment variables were added to our server, but traffic did not pass through the proxy. Is there a way (perhaps another environment variable) to tell the library that traffic should go through an HTTP proxy?

Environment details

• OS:
• Windows Server 2020
• .NET version:
  • 6.0
• Package name and version:
  • Google.Cloud.PubSub.V1 2.10.0

Steps to reproduce

1. Close the internet on a server and allow only to pass via a proxy
2. Try to publish a message

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[jskeet]

but traffic did not pass through the proxy

So did you get an exception? Did something else happen? I would expect Grpc.Core (which that version of Google.Cloud.PubSub.V1 uses) to take note of those environment variables - but probably not the auth library. Whether or not that's involved will depend on the kind of credential you're using, which we don't know about at the moment.

If you can provide more information, we're more likely to be able to help you. Additionally, you might want to try updating to Google.Cloud.PubSub.V1 version 3.0.0, which used Grpc.Net.Client by default - so should use whatever proxy HttpClient uses.

[ygalbel]

Thanks for your anwer. We will check.

[ygalbel]

Hi, We upgraded to Google.Cloud.PubSub.V1 version 3.0.0 But still, the traffic doesn't pass over the proxy.

I was able to pass the proxy as HttpClientFactory but it worked only for the authentication part. The data messages were still ignoring the proxy.

     var factory = HttpClientFactory.ForProxy(new WebProxy(new Uri("*******")));

            var cfg = _config().For(destination);
            var gcpCredentials = GoogleCredential.FromJson(googleJson)
                .CreateScoped(PublisherServiceApiClient.DefaultScopes)
                .CreateWithHttpClientFactory(factory);

            var clientCreationSettings = new PublisherClient.ClientCreationSettings(
                serviceEndpoint: cfg.ServiceEndpoint,
                credentials: gcpCredentials.ToChannelCredentials());

I added every environment variable that could be related to http/grpc proxy. HTTP_PROXY, HTTPS_PROXY, GRPC_PROXY

How can I be sure that I use the new Grpc.Net.Client transport and not the old one?

[amanda-tarafa]

Can I ask if the server's system has a proxy configured? HttpClient should, by default, pick up whatever proxy configuration exists in the system, and that should have happend for both the authentication code and the gRPC calls when using Grpc.Net.Client. If you had to specifically set a proxy for the authentication part then I'm guessing that your system doesn't have proxy settings. Please let us know.

If for some reason you cannot set a system wide proxy, we can look into this further.

[ygalbel]

Hi thanks for your answer.

When I set a system proxy, Both the authentication and the data don't pass via the proxy.

When I add the proxy as I did in the code, You can see only the pubsub data bypass it

How can I am sure I am using Grpc.Net.Client? I see both Grpc.Net.Client and Google.Core.Api in my bin folder.

[amanda-tarafa]

Which framework are you targeting? In particular if you are targetting .NET Core 3.1 or higher, including .NET 5+ you are most likely using Grpc.Net.Client, in turn if you are targetting .NET Framework you are likely using Grpc.Core.

You can read more on transport selection, including the description of default behaviour and how to force your application to use one of the two implementations. If you could momentarily force your application to use Grpc.Net.Client and try again?

Also, a note on your latest response: I don't fully understand what you mean by the calls "bypassing" the proxy. Do you mean that the calls fail? Or do you have multiple proxies enabled? Are you certain that the proxy you have enabled will match the route these calls are taking, i.e. HTTP vs HTTPS (SSL), etc.? Can you set up just one proxy for all traffic and try again?

[ygalbel]

We are targeting .net 6.0. I made some checks and we are running Grpc.Net.Client. I set a proxy in windows level, but still the client create his own connection and don't pass by the proxy.

My dev machine is open to internet so the data is just published. On our server internet is closed so it fails there without any traced exception.

[amanda-tarafa]

Let me try and reproduce and I'll get back here when I know more.

[ygalbel]

Not sure it will help you, But I was able to made him work with changing a bit your code

       var builder = new PublisherServiceApiClientBuilder
                {
                    EmulatorDetection = clientCreationSettings?.EmulatorDetection ?? EmulatorDetection.None,
                    Endpoint = clientCreationSettings?.ServiceEndpoint,
                    ChannelCredentials = clientCreationSettings?.Credentials,
                    Settings = clientCreationSettings?.PublisherServiceApiSettings,
                    ChannelOptions = grpcChannelOptions,
                    GrpcAdapter = GrpcNetClientAdapter.Default.
                    WithAdditionalOptions((option) =>
                    {
                        option.HttpHandler = CreateHandler();
                    })
                };
                var channel = isAsync ?
                    await builder.CreateChannelAsync(cancellationToken: default).ConfigureAwait(false) :
                    builder.CreateChannel();

In CreateHandler we have this code:

 public static HttpMessageHandler CreateHandler()
        {
            var proxyAddress = "http://" + Environment.GetEnvironmentVariable("HTTP_PROXY");

            var proxy = new WebProxy(proxyAddress, true, null, null);
            var webRequestHandler = new HttpClientHandler()
            {
                UseProxy = true,
                Proxy = proxy,
                UseCookies = false
            };
            return webRequestHandler;
        }

Not sure it how it should be but it definitely works. The problem it's we need to integration the whole Google.Cloud.PubSub.V1 code in our project

[amanda-tarafa]

I can't reproduce what you are describing. In my case, both Google.Apis.Auth and Google.Cloud.PubSub.V1 always pick up proxy settings from the system. Take a look at the code I've been testing with. The HttpClient test is for control.

• First, with no system proxy settings at all. All three operations were successful.
• Then I've set a system setting with a dummy proxy: host 1.2.3.4 and port 1, all three operations failed as expected.
• Then I've set a system setting with a good proxy and all three operations succeeded and I was able to verify that they had gone through the good proxy.

Basically both the Auth and Pub/Sub code do, proxy wise, whatever System.Net.Http.HttpClient does, which is the expected behaviour. I'm not sure why you are not seeing this behaviour but without a way to reproduce, there's little we'll be able to do. What you are seeing is very likely to be related to how the proxies are being configured, or something else in your environment. Figuring out what is happening might be the best way to make your code work behind the proxy. If you manage to get System.Net.Http.HttpClient to behave differently from Google.Apis.Auth and Google.Cloud.PubSub.V1 please share a repro with us so we can continue to look into it further.

That said, for Google.Cloud.PubSub.V1 when using Grcp.Net.Client, there's no way to set a specific proxy programatically. I'll discuss with the team what are our options and how we can best support it. I'm marking this issues as a feature request for that reason.

In the meantime, and if you can't figure out why your code is not picking up system proxy settings, the only workaround I can think of is for you to force the use of Grpc.Core and set the Grpc.Core environment variables for proxy detection.

[amanda-tarafa]

Not sure it will help you, But I was able to made him work with changing a bit your code ...

Yes, as I've said in my last reply Google.Cloud.PubSub.V1 when using Grcp.Net.Client does not support setting a proxy, either programmatically or through environment variables. We'll evaluate what's the best way to add that support etc. I'll get back here when we know more about that. I must say, it's likely that we won't support the Grcp.Core env variables which is what you are doing in the modified code.

[ygalbel]

Thanks again for your answers.

Just one point I don't understand, Why the proxy environment variables are not supported built in as you are using the Grcp.Net.Client and it's already support them?

[amanda-tarafa]

Just one point I don't understand, Why the proxy environment variables are not supported built in as you are using the Grcp.Net.Client and it's already support them?

That's not the documentation for Grcp.Net.Client, that's the documentation for Grpc.Core.

Grpc.Net.Client is now the recommended gRPC implementation for .NET. You can read more about that here: https://grpc.io/blog/grpc-csharp-future/

[amanda-tarafa]

We've merged #8849 that will allow you to set a proxy for both PublisherClient and SubscriberClient. This changed is yet to be released but when it is (in the next couple of weeks) I'll come back here and add a small code sample of how you can set the Proxy. I'll be closing this issue now.

[ygalbel]

Thanks for your help.

[odin568]

Hi @amanda-tarafa I have the same issue in a corporate environment where we have a small on-prem application publishing pub/sub events. Before, until version 2.10.0, we specified the env variable grpc_proxy and this was all that needed to be done - no programatic change. This was nice as from our local machines, no proxy was required, only in deployed state. Now this does not work anymore with 3.0.0. I learned about the Grpc.Net change, which sounds good to me and then stumbled over this issue. So I am looking for a solution to set the proxy, in best case without any programmatic change, but by setting env variables. I tried HTTP_PROXY and HTTPS_PROXY along the grpc_proxy but that does not work. I see with the upcoming change, I might do that programatically but yeah - does not feel better as it was until now.

Any chance you can support again environment variables or am I seeing something wrong here?

[jskeet]

@odin568: I'd expect that if the system is set up to use a proxy in a way that HttpClient will take note of, it should just work. There should be less need for any environment variables than before. (The simplest way to test that is with a console app that just creates an HttpClient and makes a request to some arbitrary web site. You can use the example code that Amanda provided earlier) That's the way I'd recommend going - Grpc.Core will eventually not be supported any more.

If you do want to continue to use Grpc.Core though, you should:

• Add a direct dependency to Grpc.Core in your project
• Set the GRPC_DEFAULT_ADAPTER_OVERRIDE environment variable to Grpc.Core as described in https://cloud.google.com/dotnet/docs/reference/help/transports#default-selection

[odin568]

@jskeet thanks for prompt reply! Well yeah in fact I would like to use the .NET based one. Reading here: https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient.defaultproxy?view=net-6.0 it seems I also do not anything wrong. I specify HTTP_PROXY and HTTPS_PROXY but nothing changes - added it as upper case variant and lowercase one as we struggled with that in the past (unix container). Yes will try the variants out directly with HttpClient - just need a bit more time. So it looks it simply gets ignored :(

[odin568]

So, did some ugly hack in the startup method to basically make the HttpClient calls in deployed state.

private static async Task DebugGrpc()
        {
            Log.Information("HttpClient test");
            HttpClient client = new HttpClient();

            Log.Information("Proxy test: " + HttpClient.DefaultProxy.GetProxy(new Uri("https://www.google.com")));

            Log.Information("Pinging Google");
            try
            {
                var response = await client.GetAsync("https://www.google.com");
                if (response.IsSuccessStatusCode)
                {
                    Log.Information("Google found");
                }
                else
                {
                    Log.Error("Pinging Google failed" + response.StatusCode);
                }
            }
            catch (HttpRequestException ex)
            {
                Log.Error(ex ,"Pinging Google failed");
            }

        }

You see it succeeds and grabbed the right proxy I specified via HTTP_PROXY/HTTPS_PROXY and succeeded. But then other stuff fails as it does not pick up, even a health check, where I simply try to get the topic to test connectivity.

The code herefor is mainly:

public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context,
            CancellationToken cancellationToken = default)
        {
            try
            {
                string apiKey = _configuration["GCP:ApiKey"];
                string projectId = _configuration["GCP:ProjectId"];
                string topicId = _configuration["GCP:TopicId"];

                PublisherServiceApiClient client = ConnectionFactory.CreatePublisherClient(apiKey);
                var topicName = TopicName.FromProjectTopic(projectId, topicId);
                await client.GetTopicAsync(topicName); // needs PubSub Viewer rights!
                return HealthCheckResult.Healthy($"Successfully reached topic '{topicId}'");
            }
            catch (Exception ex)
            {
                return new HealthCheckResult(context.Registration.FailureStatus, ex.Message, ex);
            }
        }

public static PublisherServiceApiClient CreatePublisherClient(string gcpApiKey)
        {
            if (MockApiClient != null)
                return MockApiClient;

            var credential = GoogleCredential.FromJson(gcpApiKey);

            var callSettings = CallSettings
                .FromRetry(
                    RetrySettings.FromExponentialBackoff(
                        3, 
                        TimeSpan.FromMilliseconds(110), 
                        TimeSpan.FromSeconds(70), 
                        1.3, 
                        RetrySettings.FilterForStatusCodes(StatusCode.Unavailable)
                    )
                )
                .WithTimeout(TimeSpan.FromSeconds(100));

            var publishSettings = new PublisherServiceApiSettings
            {
                PublishSettings = callSettings
            };

            return new PublisherServiceApiClientBuilder
            {
                Settings = publishSettings,
                ChannelCredentials = credential.ToChannelCredentials()
            }.Build();
        }

[jskeet]

One aspect I note is that your credential isn't scoped. I don't know whether that's relevant here, but it's probably worth trying:

var credential = GoogleCredential.FromJson(gcpApiKey).CreateScoped(PublisherServiceApiClient.DefaultScopes);

Or if you're only creating a PublisherServiceApiClient you can simplify the code by not creating the ChannelCredentials yourself:

return new PublisherServiceApiClientBuilder
{
    Settings = publishSettings,
    JsonCredentials = gcpApiKey
}.Build();

It's interesting that it's failing with a timeout at the moment, after 100 seconds. Could you remove your retry settings from CreatePublisherClient to see if that changes the error?

[odin568]

Creating credentials scoped does not change anything. Thanks for the hint about the simplification!

Interestingly when I remove the timeout/retry settings completly - exactly nothing changes.

So in essence I only need to do return new PublisherServiceApiClientBuilder { JsonCredentials = gcpApiKey }.Build(); now and get basically the same log messages. Might be because DeadlineExceeded is != Unavailable status code.

But the code worked perfect until library change so don't think we have an issue there... Infrastructure is fine. When I deploy downgraded version everything is fine.

[amanda-tarafa]

A couple other things to try, to see if we can figure out what's happening, as this might not be related to the proxy not being found but to something else.

• Can you call the CheckHealthAsync action from a browser and see what happens there? Environment-triggered continuos health checks will usually reply "unhealthy" if the check does not respond "quickly" and there's a consistent 60 second timeout there that might be because of that. If something were hanging the health check would be terminated and instead of getting the actual error we would just see the timeout we are seing there.
• Try the HttpClient test as part of the health check.
• Try obtaining the health check topic as part of Startup.

Just as FYI, I just ran again my sample code locally and I can confirm that HttpClient, Google.Apis.Auth and Google.Cloud.PubSub.V1 all pick up the same proxy configuration, whether is specified through system settings or through the environment variables described for HttpClient.

[odin568]

• From browser this gets killed with nginx timeout finally - so no "unhealthy". Perhaps it is not properly configured on my end anyhow, but seems at least after 3min it does not return from browser.
• Doing the HttpClient test as part of the healtch check succeeds with same result as in image above.
• PubSub in Startup. Get the same error as before after < 1min

Exception 01 ===================================
Type: Grpc.Core.RpcException
Source: System.Runtime.ExceptionServices.ExceptionDispatchInfo, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e
Message: Status(StatusCode="DeadlineExceeded", Detail="")
Trace:    at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at xxxxxxxxxxxxxxxxxx.Program.DebugGrpc() in /builds/xxxxxxxxxxxxxxxxxx/app/xxxxxxxxxxxxxxxxxx/Program.cs:line 65
Location: 
Method: Void Throw() (0, 0)

Code producing directly in startup:

private static async Task DebugGrpc()
        {
            string apiKey = Environment.GetEnvironmentVariable("GCP__APIKEY");
            string projectId = Environment.GetEnvironmentVariable("GCP__PROJECTID");
            string topicId = Environment.GetEnvironmentVariable("GCP__TOPICID");

            Log.Information($"Checking for topic {projectId}/{topicId} ...");

            try
            {
                PublisherServiceApiClient client = new PublisherServiceApiClientBuilder
                {
                    JsonCredentials = apiKey
                }.Build();
                var topicName = TopicName.FromProjectTopic(projectId, topicId);
                await client.GetTopicAsync(topicName); // needs PubSub Viewer rights!
                Log.Information($"topic {projectId}/{topicId} exists!");
            }
            catch (Exception ex)
            {
                Log.Error(ex, "failed: " + ex.Message);
            }

        }

[jskeet]

If you're able to reproduce the failure directly in Startup, does that mean it also fails in a simple console app which doesn't do anything with ASP.NET Core?

[odin568]

I expect so. it is hard to just spawn up console app as it all runs in kubernetes environment. But if you have a look at the main, I just initialize a bootstrap logger to have proper logging for opensearch and then I do the tests before bootstrapping the application itself: From my local machine it works all fine, but in corporate environment, proxy is always a pain. But it works perfectly right now with old version of the library, so no other issue like firewalling, etc.

[amanda-tarafa]

I'll get back at this tomorrow morning London time.

[odin568]

Allright, thank you very much for your support!

[amanda-tarafa]

A few more things to try, and I'm sorry, but since we cannot reproduce, we'll have to continue this back and forth until we know more.

1. First, a question, by any chance, are you using a regional endpoint for Pub/Sub that you have ommited in your sample code? The regional endpoint would look something like this: us-east1-pubsub.googleapis.com:443.
2. Otherwise, to see if this really is the proxy, can you modify your DebugGrpc method as follows?

private static async Task DebugGrpc()
{
    string apiKey = Environment.GetEnvironmentVariable("GCP__APIKEY");
    string projectId = Environment.GetEnvironmentVariable("GCP__PROJECTID");
    string topicId = Environment.GetEnvironmentVariable("GCP__TOPICID");

    var proxy = new WebProxy(new Uri("YOUR_PROXY"));

    // Manually set the proxy for credential operations.
    var credential = GoogleCredential.FromJson(apiKey)
                .CreateScoped(PublisherServiceApiClient.DefaultScopes)
                .CreateWithHttpClientFactory(HttpClientFactory.ForProxy(proxy));

    Log.Information($"Checking for topic {projectId}/{topicId} ...");

    try
    {
        PublisherServiceApiClient publisherClient = new PublisherServiceApiClientBuilder
        {
            // Manually set the proxy on the HttpClient used by gRPC.
            GrpcAdapter = GrpcNetClientAdapter.Default.WithAdditionalOptions(options => 
                options.HttpHandler = new HttpClientHandler 
                { 
                    Proxy = proxy, 
                    UseProxy = true
                }),
                // Use here the credential with the proxy set.
                Credential = credential
        }.Build();
        var topicName = TopicName.FromProjectTopic(projectId, topicId);
        await client.GetTopicAsync(topicName); // needs PubSub Viewer rights!
        Log.Information($"topic {projectId}/{topicId} exists!");
    }
    catch (Exception ex)
    {
        Log.Error(ex, "failed: " + ex.Message);
    }
}

[odin568]

So tested it again and it works like that. I adjusted the code slightly to get the proxy from the desired env variable so that you see it is definitly available: (I have both HTTP_PROXY and HTTPS_PROXY defined FYI)

        private static async Task DebugGrpc()
        {
            string apiKey = Environment.GetEnvironmentVariable("GCP__APIKEY");
            string projectId = Environment.GetEnvironmentVariable("GCP__PROJECTID");
            string topicId = Environment.GetEnvironmentVariable("GCP__TOPICID");

            string proxyFromEnv = Environment.GetEnvironmentVariable("HTTPS_PROXY");

            var proxy = new WebProxy(new Uri(proxyFromEnv));
            Log.Information($"Using Proxy {proxyFromEnv}");

            // Manually set the proxy for credential operations.
            var credential = GoogleCredential.FromJson(apiKey)
                .CreateScoped(PublisherServiceApiClient.DefaultScopes)
                .CreateWithHttpClientFactory(HttpClientFactory.ForProxy(proxy));

            Log.Information($"Checking for topic {projectId}/{topicId} ...");

            try
            {
                PublisherServiceApiClient publisherClient = new PublisherServiceApiClientBuilder
                {
                    // Manually set the proxy on the HttpClient used by gRPC.
                    GrpcAdapter = GrpcNetClientAdapter.Default.WithAdditionalOptions(options => 
                        options.HttpHandler = new HttpClientHandler 
                        { 
                            Proxy = proxy, 
                            UseProxy = true
                        }),
                    // Use here the credential with the proxy set.
                    Credential = credential
                }.Build();
                var topicName = TopicName.FromProjectTopic(projectId, topicId);
                await publisherClient.GetTopicAsync(topicName); // needs PubSub Viewer rights!
                Log.Information($"topic {projectId}/{topicId} exists!");
            }
            catch (Exception ex)
            {
                Log.Error(ex, "failed: " + ex.Message);
            }
        }

Outcome:

So this helped to set it explicitly...

[amanda-tarafa]

OK, so now let's see if we have trouble with the credential or with Pub/Sub or possibly with both. If you could:

• Revert the DebugGrpc to what you had in this comment. This one should continue to fail as before.
• Write a DebugCredential method, and call it before the other Debug... methods:

        private static async Task DebugCredential()
        {
            string apiKey = Environment.GetEnvironmentVariable("GCP__APIKEY");

            // Just create the credential.
            // Let's scope it so that we are certain it doesn't use locally signed JWTs.
            var credential = GoogleCredential.FromJson(apiKey)
                .CreateScoped(PublisherServiceApiClient.DefaultScopes);

            Log.Information("Checking for access token ...");
            try
            {
                await ((ICredential)credential).GetAccessTokenForRequestAsync("uri");
                Log.Information("Access token fetched!");
            }
            catch (Exception ex)
            {
                Log.Error(ex, "failed: " + ex.Message);
            }
        }

[odin568]

This seems successful and GRPC test is failing again.

[jskeet]

Could you log the full exception instead of just the message in the failing DebugGrpc method? That would help us to make sure we know which gRPC adapter is being used.

[odin568]

Sure, it is always the one from this comment: https://github.com/googleapis/google-cloud-dotnet/issues/8834#issuecomment-1194134250

[jskeet]

Humbug, with no further inner exception. Rats. Even logging the underlying gRPC client wouldn't help, as it's the CallInvoker that we really need.

One option is to specify the gRPC adapter explicitly, but without any extra options:

PublisherServiceApiClient client = new PublisherServiceApiClientBuilder
{
    JsonCredentials = apiKey,
    GrpcAdapter = GrpcNetClientAdapter.Default
}.Build();

I'm expecting that to fail in exactly the same way, but it would just give me slightly more confidence that by default we really are using Grpc.Net.Client.

I'm going to have a little investigation into the source code for Grpc.Net.Client and see if I can find anything out...

[odin568]

Yes, it fails in the very same manner with same exception.

[jskeet]

Thanks for the confirmation. Will comment again once I've done a bit of digging.

[jskeet]

Quick question: which version of .NET are you using? (I'm guessing it's probably 6, but I don't think we know for sure so far in this thread, and diving into the gRPC code there are definite "pre-.NET 5" and "post-.NET 5" paths.

[odin568]

Sure, never stated that explicitly, we are on the latest edge of .NET 6, latest patch level always. It finally runs in a derived container from here: https://hub.docker.com/_/microsoft-dotnet-aspnet Full Tag currently 6.0.7-jammy-amd64

[jskeet]

Good to know, thanks. I've got another test to try :) (Thanks so much for being so response - it makes all the difference.)

This is a test that removes the Google libraries from the picture entirely:

using Grpc.Core;
using Grpc.Net.Client;
using System.Text;
...
// In a DebugGrpcChannel method or similar
var channel = GrpcChannel.ForAddress("https://pubsub.googleapis.com");
var callInvoker = channel.CreateCallInvoker();
var marshaller = new Marshaller<string>(Encoding.UTF8.GetBytes, Encoding.UTF8.GetString);
var method = new Method<string, string>(MethodType.Unary,
    "test-service", "test-method",
    marshaller, marshaller);
try
{
    var response = callInvoker.BlockingUnaryCall(method, null, default, "test-request");
    // Log the response (it would be really unexpected to get here!)
}
catch (Exception e)
{
    // Log the exception
}

I would expect an exception like this:

Unhandled exception. Grpc.Core.RpcException: Status(StatusCode="Unimplemented", Detail="Bad gRPC response. HTTP status code: 404")
   at Grpc.Net.Client.Internal.HttpClientCallInvoker.BlockingUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
  ...

That's basically "I managed to get to the server, but you've made a bogus request."

I suspect the method will either hang or throw an exception in your environment. But it would be good to know. (If it behaves with the expected error, I'm even more confused...)

[odin568]

It is blocked since 10min in "BlockingUnaryCall(...)" and not returning at all, no exception :(

[jskeet]

Right. Well, that's good news in terms of this repository, in that I don't think we have a bug. It's bad news in terms of us not being able to help much more, either.

It feels like this is something that should be fairly easily reproducible, but I can't reproduce it locally... for example, if I run that code with HTTPS_PROXY=nowhere.com then I get an entirely different exception. I might try it with the Docker container you mentioned...

[jskeet]

@odin568: I'm assuming that you're running it with the HTTPS_PROXY environment variable set? That's the right environment variable to use here.

[odin568]

yes, to be safe I also set HTTP_PROXY. Both filled with same value. I also used that variable in the other debug methods to show it is correctly set: https://github.com/googleapis/google-cloud-dotnet/issues/8834#issuecomment-1196448526

[odin568]

Ha, now it returned after ~30 minutes 🎆

[odin568]

FROM mcr.microsoft.com/dotnet/aspnet:6.0.7-focal-amd64

# Add package for GCP-Libraries to work
RUN apt-get update && apt-get install -y libc-dev && \
    rm -rf /var/lib/apt/lists/*

# Create a group and user (uid/gid required for K8s)
RUN groupadd --gid 3000 appgroup && \
    useradd -rm -d /home/appuser -s /bin/bash -g appgroup -G sudo -u 1000 appuser

# Tell docker that all future commands should run as the appuser user
USER 1000

# add application
COPY ${PUBLISH_PATH} App/
WORKDIR /App
EXPOSE 80
ENTRYPOINT ["dotnet", "myapp.dll"]

That's basically my container definition. Manually merged and simplified to give you an insight.

Our testsuite container has the same issue and runs with container mcr.microsoft.com/dotnet/sdk:6.0-jammy on gitlab k8s runners.

[jskeet]

That's very, very helpful, thanks.

[jskeet]

Unfortunately, I've had no joy reproducing this even with Docker.

Here's my Dockerfile:

# syntax=docker/dockerfile:1
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
WORKDIR /app

# Copy csproj and restore as distinct layers
COPY *.csproj ./
RUN dotnet restore

# Copy everything else and build
COPY *.cs ./
RUN dotnet publish -c Release -o out

FROM mcr.microsoft.com/dotnet/aspnet:6.0.7-focal-amd64

# Add package for GCP-Libraries to work
RUN apt-get update && apt-get install -y libc-dev && \
    rm -rf /var/lib/apt/lists/*

# Create a group and user (uid/gid required for K8s)
RUN groupadd --gid 3000 appgroup && \
    useradd -rm -d /home/appuser -s /bin/bash -g appgroup -G sudo -u 1000 appuser

# Tell docker that all future commands should run as the appuser user
USER 1000

# add application
COPY --from=build-env /app/out App/
WORKDIR /App
EXPOSE 80
ENTRYPOINT ["dotnet", "Issue8834.dll"]

... and my project file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Grpc.Net.Client" Version="2.47.0" />
  </ItemGroup>

</Project>

... and the code:

using Grpc.Core;
using Grpc.Net.Client;
using System.Text;

var channel = GrpcChannel.ForAddress("https://pubsub.googleapis.com");
var callInvoker = channel.CreateCallInvoker();
var marshaller = new Marshaller<string>(Encoding.UTF8.GetBytes,
Encoding.UTF8.GetString);
var method = new Method<string, string>(MethodType.Unary,
    "test-service", "test-method",
    marshaller, marshaller);
try
{
    Console.WriteLine("Starting request");
    var response = callInvoker.BlockingUnaryCall(method, null,
        default, "test-request");
    Console.WriteLine($"Got response {response}");
}
catch (Exception e)
{
    Console.WriteLine(e);
}

When I run it with just docker run, it gets the expected 404 failure. When I run it with docker run --env HTTPS_PROXY=nowhere.com it shows that it tried to connect to nowhere.com and failed.

I have no idea why it would behave differently in your k8s cluster :(

[odin568]

Only difference I see is that I have v2.46.0 implicitly instead of 2.47.0... At least something around the deadlineexceeded they changed - even if I don't expect something really.... https://github.com/grpc/grpc-dotnet/compare/v2.46.0...v2.47.0

[jskeet]

Trying 2.46.0 now just in case...

[jskeet]

Nope, exactly the same behavior :(