[Bug Report] [v2] [IoTSDK] ModuleClient.SendMessageToRoute() not firing IncomingMessageCallback when routing M2M

[mitchell-burley-orica]

Context

• OS, version, SKU and CPU architecture used: Ubuntu 20.04.2 LTS
• Application's .NET Target Framework : .NET8.0-rc1
• Device: PC
• SDK version used: 2.0.0-preview007

Description of the issue

As a simplified version, we have created 2 bare-bones modules that are simply to send messages on an output and to receive messages on an input. When calling The SendMessageToRoute() method on the ModuleClient class, it seemingly succeeds without exception at sending the message, however the IotEdgeModule2.OnMessageReceivedAsync callback is never fired. We have verified that the routing is setup correctly between the modules. Using the pre-migrated methods in v1 of the SDK, this works fine. Following the sample code provided here: EdgeModuleMessageSample it seems very similar to what I've outlined below. Any insight would be greatly appreciated

Code sample exhibiting the issue

IotEdgeModule1 -> Responsible for periodically sending a message on output1

public sealed class IotEdgeModule1: BackgroundService
{
    private IotHubModuleClient _client;
    protected override async Task ExecuteAsync(CancellationToken cancellationToken)
    {
        var connectionString = "removed";

        _client = new IotHubModuleClient(connectionString);

        await _client.OpenAsync(cancellationToken);

        var sleep = TimeSpan.FromSeconds(15);
        while (!cancellationToken.IsCancellationRequested)
        {
            var message = new TelemetryMessage(Encoding.ASCII.GetBytes("Sample message"));
            await _client.SendMessageToRouteAsync("output1", message);
            await Task.Delay(sleep, cancellationToken);
        }
  }

IotEdgeModule2 -> Responsible for logging the messages it receives from output1 of IotEdgeModule1

public sealed class IotEdgeModule2 : BackgroundService
{
    private IotHubModuleClient _client;
    protected override async Task ExecuteAsync(CancellationToken cancellationToken)
    {
        var connectionString = "removed";

        _client = new IotHubModuleClient(connectionString);

        await _client.OpenAsync(cancellationToken);

        await _client.SetIncomingMessageCallbackAsync(OnMessageReceivedAsync);

        var sleep = TimeSpan.FromSeconds(15);
        while (!cancellationToken.IsCancellationRequested)
        {
            await Task.Delay(sleep, cancellationToken);
        }
    }

    private async Task<MessageAcknowledgement> OnMessageReceivedAsync(IncomingMessage receivedMessage)
    {
        Console.WriteLine("Received message on {Input} input.", receivedMessage.InputName);

        return await Task.FromResult(MessageAcknowledgement.Complete);
    }
}

Routing: FROM /messages/modules/IotEdgeModule1/outputs/output1 INTO BrokeredEndpoint("/modules/IotEdgeModule2/inputs/input1")