Grpc exporter doesn't follow dns change.

[pengweiqhca]

Bug Report

List of all OpenTelemetry NuGet packages and version that you are using (e.g. OpenTelemetry 1.0.2):

• OpenTelemetry.Exporter.OpenTelemetryProtocol 1.5.1

Runtime version (e.g. net462, net48, netcoreapp3.1, net6.0 etc. You can find this information from the *.csproj file):

• net6.0

Symptom

When i changed dns ip, grpc exporter still connected the old ip.

What is the expected behavior?

Grpc connect to new ip.

What is the actual behavior?

Grpc still connected the old ip long after i change dns.

Reproduce

var builder = WebApplication.CreateBuilder(args);

var meter = new Meter("Test");

var counter = meter.CreateObservableUpDownCounter("test.counter", () =>
{
    Console.WriteLine(DateTimeOffset.Now);

    return DateTimeOffset.Now.ToUnixTimeMilliseconds();
});

builder.Services
    .Configure<MetricReaderOptions>(options => options.PeriodicExportingMetricReaderOptions.ExportIntervalMilliseconds = 10000)
    .AddOpenTelemetry()
    .WithMetrics(mpb => mpb.AddMeter(meter.Name)
        .AddOtlpExporter(options => options.Endpoint = new Uri("http://collector.foo.com:4317")));

var app = builder.Build();

app.Run();

1. Set collector.foo.com = ip1 in local host file.
2. Open wireshark 1, start capturing TCP packet, set filter ip.addr == ip1 && tcp.port = 4317.
3. Open wireshark 2, start capturing TCP packet, set filter ip.addr == ip2 && tcp.port = 4317.
4. Run test code.
5. TCP data will appear in wireshark 1.
6. Modify collector.foo.com = ip2 in local host file.
7. TCP data still in wireshark 1 rather than wireshark 2.

Additional Context

Grpc.Net use SocketsHttpHandler in .net6.0 and SocketsHttpHandler.PooledConnectionLifetime is InfiniteTimeSpan.

GrpcChannel.ForAddress(options.Endpoint) have no any options, so SocketsHttpHandler will use default settings. If grpc exporter is used every 1 minute, the http connection will never be disposed and always use the old connection, so there need to use the GrpcClientFactory instead.

Http exporter already uses HttpClientFactory.

[vishweshbankwar]

@pengweiqhca I am not able to repro this behavior you are noticing (I do not see any interruption). looking at the doc as well, it should continue to work.

@JamesNK Could you please confirm? Reference: Grpc Channel Config for otlp exporter.

[JamesNK]

I'm not an expert at reacting to DNS changes.

Changing the code to use Grpc.Net.Client.ClientFactory is one option.

A simpler change would be to specify a SocketsHttpHandler with GrpcChannelOptions.HttpHandler. Then you can specify SocketsHttpHandler.PooledConnectionLifetime

[vishweshbankwar]

@JamesNK Thanks! The behavior I am noticing is there is no interruption in connection during ip changes with this Grpc Channel Config (different than what reported on this issue). If I interpret this doc correctly, we dont have to do anything and it should automatically refresh. is that correct?

[JamesNK]

Once a connection is up and running, SocketsHttpHandler controls the connection lifetime. I think https://learn.microsoft.com/en-us/dotnet/api/system.net.http.socketshttphandler.pooledconnectionlifetime?view=net-7.0#system-net-http-socketshttphandler-pooledconnectionlifetime is what you want to set.

we dont have to do anything and it should automatically refresh. is that correct?

I don't know. As I said, I'm not an expert in this area.