fhoutr01 commented 5 years ago

DotNetCore 2.2.6 on linux takes a long time to make connections and publish messages to AWS AMQ

General

We run dotnetcore in containers on our OpenShift cluster. Most recently they failed on all environments because they can no longer connect to AWS AMQ. It feels like it's running against a timeout of something. This issue only occurs when running dotnetcore on linux.

My linux guest system is an ubuntu 18.04 ( see dotnet --info below ). It also occurs on RHEL 7 containers

I eliminated the network as culprit:

I can connect from my windows machine, but it takes a long time from my linux guest OS in the same machine.
My linux guest OS can connect and post messages to AWS AMQ from a java program ( fast ).
I asked our network guy to allow outgoing network traffic from my machine to that specific instance.
I used wireshark to monitor the network traffic from my PC to see if it connected somewhere else ( which it didn't ).

I eliminated AWS AMQ as the culprit:

The AWS logs show no sign of something wrong
The same queue works in a java
We set up a different queue from scratch and the issue still occurs.

Wireshark shows pauses of ~20seconds at certain intervals. It is consistently around 20seconds, so it does feel like a timeout of some sort... Wireshark shows dotnetcore connecting with TLS 1.0 connection in both windows & linux using the same cipher suite ( TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA ). When executing the same code on windows it can connect just fine. When trying to connect to the AWS AMQ in java it is able to connect fine as well.

The only remaining "culprit" I can think of is something going wrong in the dotnetcore runtime itself unless I am missing something.

Below you have a C# code sample ( taken from the amazon documentation ). Building and running on windows - it works perfect. Build and running on Linux - It takes forever to connect and post messages ( > 50s ).

Note: I just noticed there is a slight version difference between my linux & windows regarding dotnetcore. I will update it and try that as well and add it as output.

System Information

Host System: Windows 10 Guest System: Linux 18.04

Dotnet SDK was installed through https://dotnet.microsoft.com/download/linux-package-manager/ubuntu18-04/sdk-current

AWS AMQ Information

Instance Type: mq.t2.micro Deployment mode: single-instance broker Broker Engine: ActiveMQ Broker engine version: 5.15.9

Build command

dotnet build --force -c Release dotnet publish -c Release -o ../publish/exploded

Run command

dotnet AWSMQSend.dll $@

Application output on linux: ( notice gaps in timestamps )

9/9/19 11:54:56 AM :: Creating Factory
9/9/19 11:54:56 AM :: Factory Created
9/9/19 11:54:56 AM :: Connection...
9/9/19 11:54:56 AM :: Done
9/9/19 11:54:56 AM :: Creating Session.....
9/9/19 11:55:27 AM :: Done
9/9/19 11:55:27 AM :: Using destination: queue://queue_filip
9/9/19 11:55:57 AM :: Start the connection so that messages will be processed
9/9/19 11:55:57 AM :: Sending message....
9/9/19 11:56:17 AM :: Received message with ID:   ID:fluxbox-34467-637033634349736262-1:0:1:1:1
9/9/19 11:56:17 AM :: Received message with text: Hello World!

Application output on windows:

9-9-2019 12:01:39 :: Creating Factory
9-9-2019 12:01:39 :: Factory Created
9-9-2019 12:01:39 :: Connection...
9-9-2019 12:01:40 :: Done
9-9-2019 12:01:40 :: Creating Session.....
9-9-2019 12:01:40 :: Done
9-9-2019 12:01:40 :: Using destination: queue://queue_filip
9-9-2019 12:01:40 :: Start the connection so that messages will be processed
9-9-2019 12:01:40 :: Sending message....
9-9-2019 12:01:40 :: Received message with ID:   ID:BEZAV1CL13911-53543-637036270142492136-1:0:1:1:1
9-9-2019 12:01:40 :: Received message with text: Hello World!

Application Code C# code

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

using System;
using Apache.NMS;
using Apache.NMS.Util;

namespace Apache.NMS.ActiveMQ.Test
{
public class Program
{
    public static void Main(string[] args)
    {
        string host = "<your-host>.mq.eu-central-1.amazonaws.com";
        string port = "61617";
        string userName = "<queue-username>";
        string password = "<queue-password>";

        Uri connecturi = new Uri("activemq:ssl://" + host + ":" + port);

        Console.WriteLine( DateTime.Now + " :: Creating Factory" );

        // NOTE: ensure the nmsprovider-activemq.config file exists in the executable folder.
        IConnectionFactory factory = new NMSConnectionFactory(connecturi);

        Console.WriteLine( DateTime.Now + " :: Factory Created" );

        Console.WriteLine( DateTime.Now + " :: Connection...");
        IConnection connection = factory.CreateConnection( userName, password);
        Console.WriteLine( DateTime.Now + " :: Done");
        Console.WriteLine( DateTime.Now + " :: Creating Session.....");
        //The line below takes 20 seconds to complete!
        ISession session = connection.CreateSession( AcknowledgementMode.AutoAcknowledge );
        Console.WriteLine( DateTime.Now + " :: Done");

        IDestination destination = SessionUtil.GetDestination(session, "queue://queue_filip");
        Console.WriteLine( DateTime.Now + " :: Using destination: " + destination);

        // Create a consumer and producer
        using(IMessageConsumer consumer = session.CreateConsumer(destination))
        using(IMessageProducer producer = session.CreateProducer(destination))
        {
            // Start the connection so that messages will be processed.
            Console.WriteLine( DateTime.Now + " :: Start the connection so that messages will be processed" );
            connection.Start();
            producer.DeliveryMode = MsgDeliveryMode.Persistent;

            // Send a message
            ITextMessage request = session.CreateTextMessage("Hello World!");
            request.NMSCorrelationID = "abc";
            request.Properties["NMSXGroupID"] = "cheese";
            request.Properties["myHeader"] = "Cheddar";

            Console.WriteLine( DateTime.Now + " :: Sending message...." );

            producer.Send(request);

            // Consume a message
            ITextMessage message = consumer.Receive() as ITextMessage;
            if(message == null)
            {
                Console.WriteLine( DateTime.Now + " :: No message received!");
            }
            else
            {
                Console.WriteLine( DateTime.Now + " :: Received message with ID:   " + message.NMSMessageId);
                Console.WriteLine( DateTime.Now + " :: Received message with text: " + message.Text);
            }
        }
    }
}
}

dotnet --info [Linux]

$ dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   2.2.401
 Commit:    729b316c13

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.2.401/

Host (useful for support):
  Version: 2.2.6
  Commit:  7dac9b1b51

.NET Core SDKs installed:
  2.2.401 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

dotnet --info [Windows]

$ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.104
 Commit:    73f036d4ac

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.16299
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.104\

Host (useful for support):
  Version: 2.2.2
  Commit:  a4fd7b2c84

.NET Core SDKs installed:
  2.2.104 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Updates

Update: Execution on linux is fast against a local AMQ docker container, without SSL:

9/9/19 2:00:18 PM :: Creating Factory
9/9/19 2:00:18 PM :: Factory Created
9/9/19 2:00:18 PM :: Connection...
9/9/19 2:00:19 PM :: Done
9/9/19 2:00:19 PM :: Creating Session.....
9/9/19 2:00:19 PM :: Done
9/9/19 2:00:19 PM :: Using destination: queue://queue_filip
9/9/19 2:00:19 PM :: Start the connection so that messages will be processed
9/9/19 2:00:19 PM :: Sending message....
9/9/19 2:00:19 PM :: Received message with ID:   ID:fluxbox-44221-637036344189323303-1:0:1:1:1
9/9/19 2:00:19 PM :: Received message with text: Hello World!

Edit: Spelling

fhoutr01 commented 5 years ago

Note: Updated to the latest version of dotnetcore SDK on windows and the result is the same. It's still fast on Windows. This leads me to believe there is something wrong when running dotnetcore on linux when connecting to AWS ActiveMQ.


.NET Core SDK (reflecting any global.json):
 Version:   2.2.401
 Commit:    729b316c13

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.16299
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.401\

Host (useful for support):
  Version: 2.2.6
  Commit:  7dac9b1b51

.NET Core SDKs installed:
  2.2.104 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

fhoutr01 commented 5 years ago

Updated original post with ActiveMQ information

scalablecory commented 5 years ago

Hi @fhoutr01, are you able to try with .NET Core 3.0? We can give you some tracing commands that might help determine if there's something obvious. If nothing comes up there, we will need a minimal repro that doesn't go through an AMQ library.

davidsh commented 5 years ago

Update: Execution on linux is fast against a local AMQ docker container, without SSL

We've had various issues already reported here in this repo about Linux SSL performance. Many of those issues have been resolved for .NET Core 3.0. So, it would be very good to try out the latest .NET Core 3.0 preview: https://devblogs.microsoft.com/dotnet/announcing-net-core-3-0-preview-9/

to see if the performance issues are resolved for you.

fhoutr01 commented 5 years ago

Hey,

I installed dotnetcore 3.0 preview 9 and updated the csproj file to build it against a dotnet 3.0 app:

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>netcoreapp3.0</TargetFramework>
        <LangVersion>latest</LangVersion>
    </PropertyGroup>

No luck unfortunately :(.

$ dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   3.0.100-preview9-014004
 Commit:    8e7ef240a5

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /home/filip/temp/dotnetcore/sdk/3.0.100-preview9-014004/

Host (useful for support):
  Version: 3.0.0-preview9-19423-09
  Commit:  2be172345a

.NET Core SDKs installed:
  3.0.100-preview9-014004 [/home/filip/temp/dotnetcore/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.0.0-preview9.19424.4 [/home/filip/temp/dotnetcore/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.0.0-preview9-19423-09 [/home/filip/temp/dotnetcore/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

9/10/2019 8:41:54 AM :: Creating Factory
9/10/2019 8:41:54 AM :: Factory Created
9/10/2019 8:41:54 AM :: Connection...
9/10/2019 8:41:54 AM :: Done
9/10/2019 8:41:54 AM :: Creating Session.....
9/10/2019 8:42:25 AM :: Done
9/10/2019 8:42:25 AM :: Using destination: queue://queue_filip
9/10/2019 8:42:55 AM :: Start the connection so that messages will be processed
9/10/2019 8:42:55 AM :: Sending message....
9/10/2019 8:43:15 AM :: Received message with ID:   ID:BEZAV1CL13911-59547-637036404994873106-1:0:1:1:1
9/10/2019 8:43:15 AM :: Received message with text: Hello World!

Crozin commented 5 years ago

Got the exact same issue using latest version (netcore3.0-preview9) as well as 2.1:

dotnet3:

  It was not possible to find any installed .NET Core SDKs
  Did you mean to run .NET Core SDK commands? Install a .NET Core SDK from:
      https://aka.ms/dotnet-download

Host (useful for support):
  Version: 3.0.0-preview9-19423-09
  Commit:  2be172345a

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.0.0-preview9.19424.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.0.0-preview9-19423-09 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

dotnet2.1:



Host (useful for support):
  Version: 2.1.12
  Commit:  ccea2e606d

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.12 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.12 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.12 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download```

wfurt commented 5 years ago

can you get packet captures @fhoutr01. From your description it feels like every request is slow, not just all together e.g. aggregate performance, right? As quick test, can you try to connect to specific IP instead of using name?

scalablecory commented 5 years ago

Some tracing would be helpful here as well:

export COMPlus_EnableEventPipe=1
COMPlus_EventPipeConfig=Microsoft-System-Net-Sockets:0xFFFFFFFFFFFFFFFF:5,Microsoft-System-Net-Security:0xFFFFFFFFFFFFFFFF:5,Microsoft-System-Net-Http:0xFFFFFFFFFFFFFFFF:5,Microsoft-System-Net-NameResolution:0xFFFFFFFFFFFFFFFF:5
dotnet foo.dll

fhoutr01 commented 5 years ago

I'm currently abroad for work so I'll have a look at this on monday!

@wfurt I can attach wireshark logs when i'm back at the office next monday. @scalablecory I'll get on that next monday!

fhoutr01 commented 5 years ago

Hello,

I ran the tests with the latest dotnetcore version (2.2.7 )

.NET Core SDK (reflecting any global.json):
 Version:   2.2.402
 Commit:    c7f2f96116

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.2.402/

Host (useful for support):
  Version: 2.2.7
  Commit:  b1e29ae826

.NET Core SDKs installed:
  2.2.402 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.2.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.2.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.2.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Results remain the same. I attached a tar compressed archive containing the build, publish and execution of the code with the parameters set that you provided. Inside the archive is also a wireshark of the whole connection towards the AWS AMQ. There are a few NTP calls in there as well (unfortunately ).

dotnet.tar.gz

fhoutr01 commented 5 years ago

@wfurt directly connecting is unfortunately not possible because of HTTPS certificates. If I connect over IP, it can't verify the certificate because it does not match the hostname. And since it's not HTTP protocol, I cannot specify a header to add.

What I can tell is that the resolving of the host is really fast, setting up a connection to the host is also fast. It's slow when it's trying to bind the session.

Output below is from directly connecting to the IP instead of the hostname.

9/17/19 11:58:52 AM :: Creating Factory
9/17/19 11:58:52 AM :: Factory Created
9/17/19 11:58:52 AM :: Connection...
9/17/19 11:58:52 AM :: Done
9/17/19 11:58:52 AM :: Creating Session.....

Unhandled Exception: System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at Apache.NMS.ActiveMQ.Transport.Tcp.SslTransport.CreateSocketStream()
   at Apache.NMS.ActiveMQ.Transport.Tcp.TcpTransport.Start()
   at Apache.NMS.ActiveMQ.Transport.TransportFilter.Start()
   at Apache.NMS.ActiveMQ.Transport.TransportFilter.Start()
   at Apache.NMS.ActiveMQ.Transport.WireFormatNegotiator.Start()
   at Apache.NMS.ActiveMQ.Transport.TransportFilter.Start()
   at Apache.NMS.ActiveMQ.Transport.TransportFilter.Start()
   at Apache.NMS.ActiveMQ.Connection.CheckConnected()
   at Apache.NMS.ActiveMQ.Connection.CreateActiveMQSession(AcknowledgementMode ackMode)
   at Apache.NMS.ActiveMQ.Connection.CreateSession(AcknowledgementMode sessionAcknowledgementMode)

Crozin commented 5 years ago

@fhoutr01 IIRC you can specify IP address as an endpoint and still pass ssl.ServerName=CERT-HOSTNAME (or transport.ServerName=CERT-HOSTNAME)

https://github.com/apache/activemq-nms-openwire/blob/master/src/main/csharp/Transport/Tcp/SslTransportFactory.cs#L41 - you can actually setup any of those properties with query parameters of connection URI.

fhoutr01 commented 5 years ago

@Crozin No cigar! :(

When connecting directly to the IP and specifying the cert-hostname through transport.ServerName=CERT-HOSTNAME I get the same results.

Thanks for the suggestion by the way. I'm not a C# developer so your suggestion definitely helped!

wfurt commented 5 years ago

I look at the packet capture @fhoutr01 and the grouping around 10s boundary seems suspicious. It is hard to know exactly what is going on but if I assume basic request/response communication then client sends request in packet 20 at 9.56s. Server sends back something almost immediately (packet 22) and then 5s later more - packet 24. And then it takes another 10s for client to send anything back. Similarly "request" at 20.61s packet 30, response at 30.69s packet 38.

This is already established SSL connection so DNS or certificate validation should not matter any more. I don't know anything about AMQ but it feels like the server does not like the requests for some reason.

Aside from cracking inside of SSL or setting some separate instance for investigation I don't see anything obviously wrong.

The AWSMQSend.3787.netperf trace looks like from windows. I did not see any use of HTTP. But if MQ would use HTTP under the cover you can also try:

export DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=false
dotnet AWSMQSend.dll $@

if that is also slow, do

export CURLHANDLER_DEBUG_VERBOSE=true
dotnet AWSMQSend.dll $@

BTW do you also use AMQ @Crozin? If so did anybody tried to post issue to AWSSDK? What version of AWSSDK.MQ do you guys use?

Crozin commented 5 years ago

I did not see any use of HTTP. But if MQ would use HTTP under the cover you can also try:

MQ does not use HTTP using OpenWire/ActiveMQ transport (TCP/SSL).

It should also not matter if we use IP-based host or a hostname. As AWS AMQ DNS resolves to IPv4 address only according to this https://github.com/apache/activemq-nms-openwire/blob/master/src/main/csharp/Transport/Tcp/TcpTransportFactory.cs#L322 it should basically connect the very same way as if we provide IP in the first place.

@wfurt Yeah, I also wanted to give a chance to AMQ. From my local workstation (Windows based) it works like a harm on .NET Core 2.2.x as well as 3.x preview versions. But from Docker container running on Linux host I observe very same behavior as @fhoutr01.

fhoutr01 commented 5 years ago

The following code did not impact the performance

export DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=false
dotnet AWSMQSend.dll $@

I'm not sure where to find the verbose logging export CURLHANDLER_DEBUG_VERBOSE=true drops said verbose logging?

I can confirm that the delay only happens on linux ( the distro's i've tried so far ) and not windows as Crozin mentioned. From my windows machine there is no delay in connecting and posting/reading meassages from Amazon MQ.

wfurt commented 5 years ago

So it seems lile AMQ is using SslStream directly. With that , there is very little we do at .Net after handshake. That is essentially pass-through. And based on the flow, it seems like the delay may come from server. I look at the link @Crozin and noticed references to Logging. Perhaps next step is to debug this at MQ layer.

Crozin commented 5 years ago

Hmmm... it seems it might not be related to dotnet/linux after all. Switching from Apache.NMS.ActiveMQ.Core to more recent Apache.NMS.ActiveMQ.NetStd (which claims to be exact copy with dropped support for SSL2/3 and TLS left only) causes everything to work like a harm.

fhoutr01 commented 5 years ago

@Crozin I can confirm that using the Apache.NMS.ActiveMQ.NetStd as well as using the latest version of MassTransit ( 5.5.6-develop.2174 ) resolves the issue as well.

Thanks a lot for everybody their assistance!

dotnet / runtime

DotNetCore 2.2.6 is slow on connecting and publish messages to AWS AMQ #30806

General

System Information

AWS AMQ Information

Build command

Run command

Application output on linux: ( notice gaps in timestamps )

Application output on windows:

Application Code C# code

dotnet --info [Linux]

dotnet --info [Windows]

Updates