Socket Exception when idle

[ghostofcain]

Hi,

First, thank you very much for this library.

After upgrading to the latest version (3.4), I've been having problems with the socket exceptions. Basically, when I start my app, it creates a new cluster and hold on to it for all it's future usages (unless it's unexpectedly shut down at which point it recreates the cluster).

This used to work fine with the previous version but as of the latest version, when I start the service and create the cluster, after about a minute, I get a socket exception in

StreamingFrameReader::StreamingFrameReader

line SocketReadOnlyStream.SocketReceiveBuffer(socket, _tempBuffer, 0, 8);

The exception is: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

It then proceeds to move on to LongRunningConnection::Close which hangs on this line: ExceptionExtensions.SafeExecute(() => _responseWorker.Wait());

(I would imagine it's because the responseWorker is already dead but can't be sure).

Have you ever experienced this before? I'm not great with C# Sockets and my google searches have been unfruitful so far. Since it was ok before and it's not now, I would imagine it's something that was introduced between 3.2 and 3.4 but can't find out what it is.

Any help would be appreciated.

Many thanks,

Edit: If I set the receive timeout to 0, it will not fail after the timeout has passed. Maybe that's the problem?

[ghostofcain]

In case it helps, here are my settings:

     var cassandraSharpConfig = new CassandraSharpConfig
        {
            Clusters = new[]
            {
                new ClusterConfig
                {

                    Transport = new TransportConfig
                    {
                        DefaultExecutionFlags = ExecutionFlags.None,
                        DefaultConsistencyLevel = ConsistencyLevel.ONE,
                        Port = int.Parse(_configService.GetValue("CassandraSettings", "Port")),
                        User = _configService.GetValue("CassandraSettings", "Username"),
                        Password = _configService.GetValue("CassandraSettings", "Password"),
                        KeepAlive = true,
                        ReceiveBuffering = true,
                        Recoverable = true,
                        KeepAliveTime = 1000000000, // set for testing
                        SendTimeout = 2000,
                        ReceiveTimeout = 600000, // set for testing
                    },
                    Endpoints = new EndpointsConfig
                    {
                        Servers = new[] {_configService.GetValue("CassandraSettings", "Url")},
                    },
                    Name = _configService.GetValue("CassandraSettings", "ClusterName")
                }
            },
            Endpoints = new EndpointsConfig
                    {
                        Servers = new[] {_configService.GetValue("CassandraSettings", "Url")},
                    },
                    Name = _configService.GetValue("CassandraSettings", "ClusterName")

ClusterManager.Configure(cassandraSharpConfig); _cluster = ClusterManager.GetCluster(_clusterName);

[pchalamet]

Regarding errors, before some of them were not correctly reported/handled and shutdown was not that clean. Something went wrong there while refactoring this...

I've attempt a fix regarding Dispose() handling. There are 2 tasks (writer / reader) to handle socket multiplexing and I'm pretty sure a thread is not getting notify of shutdown. It works like this now:

• mark connection for shutdown
• wake up all threads waiting for lock
• cleanup everything

Let me know if this solve your deadlock problem.

[ghostofcain]

Thank you for your response. I fixed my problem with the deadlock and everything works fine in Windows (7) but in OpenSuSe (mono 3.4), i'm getting a type initialization exception in the LongRunningConnectionFactory because of these two lines in ExceptionExtensions:

    static ExceptionExtensions()
    {
        MethodInfo preserveStackTrace = typeof(Exception).GetMethod("InternalPreserveStackTrace", BindingFlags.Instance | BindingFlags.NonPublic);
        _preserveInternalException = (Action<Exception>) Delegate.CreateDelegate(typeof(Action<Exception>), preserveStackTrace);
    }

The exact error is: 2014-08-17 17:22:02,438 [1] ERROR xxx - Error:Failed to create new connection: System.TypeInitializationException: An exception was thrown by the type initializer for CassandraSharp.Utils.ExceptionExtensions ---> System.ArgumentNullException: Argument cannot be null.

I can't quite see the purpose of that extension (RethrowPreserveStackTrace) method since it's not used anywhere. I will remove it from my internal build for now and later on try it on my Debian server I have which is running a more cutting edge mono (3.6) to see if it's a problem there.

I'll keep you updated.

[ghostofcain]

Ok that was a red herring (although still a problem in Mono which is why I disabled it). TCP Keep alive is not enabled in Mono (as per https://github.com/zeromq/netmq/issues/7). I disabled it and everything is back to normal.

I should probably start a very long blog on the headaches Mono is giving me and ways to circumvent them :). I'm closing this issue since it's not really an issue with the code but with Mono.

Regards,

[pchalamet]

Keep it opened please. This is supposed to work on Linux/mono. I will look at it back from vacations :-)

• Pierre

-----Original Message----- From: "ghostofcain" notifications@github.com Sent: ‎17/‎08/‎2014 18:46 To: "pchalamet/cassandra-sharp" cassandra-sharp@noreply.github.com Cc: "Pierre Chalamet" pierre@chalamet.net Subject: Re: [cassandra-sharp] Socket Exception when idle (#81)

Ok that was a red herring (although still a problem in Mono which is why I disabled it). TCP Keep alive is not enabled in Mono (as per zeromq/netmq#7). I disabled it and everything is back to normal. I should probably start a very long blog on the headaches Mono is giving me and ways to circumvent them :). I'm closing this issue since it's not really an issue with the code but with Mono. Regards, — Reply to this email directly or view it on GitHub.

[ghostofcain]

Ok sure. Reopened.

Let me know if you need any help with reproducing the problem.

[pchalamet]

Removed useless RethrowPreserveStackTrace Do not force keep-alive when running under Mono.

I've runned MoMA and looks like there are some other partially implemented methods hanging around but should work anyway.

[ghostofcain]

Thanks Pierre. There's another bug for which I'll raise another JIRA.