ArgumentException with a hint that the Stream was not readable is thrown when multiple endpoints are started simultaneously

[sanilpaul]

Who's affected

• Any user which is starting multiple endpoints simultaneously for the first time

  Symptoms

When multiple endpoints are starting simultaneously for the first time, All endpoints except one crashes with "Stream was not readable exception ".

Steps to reproduce

I have tried with 3 endpoints. We can reproduce it on multiple machines consistently. This probably is related : https://groups.google.com/forum/#!topic/ravendb/IrMqmgPyrGo

RavenDB - Version 2.5 Build-2956 NServiceBus" version="5.2.0" NServiceBus.Host" version="6.0.0" NServiceBus.RavenDB" version="2.1.0" RavenDB.Client" version="2.5.2916"

public class EndpointConfig : IConfigureThisEndpoint
    {
        public void Customize(BusConfiguration configuration)
        {
            configuration.UsePersistence<RavenDBPersistence>();
        }
    }

Steps to Repro

1. pull down Ravendb. Start RavenServer
2. Create two end points. All I have is the above Code for each endpoints
3. Set multiple start up projects in Visual studio. Set both Endpoints to start.
4. F5

System.ArgumentException was unhandled by user code
  HResult=-2147024809
  Message=Stream was not readable.
  Source=mscorlib
  StackTrace:
       at System.IO.StreamReader..ctor(Stream stream, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean leaveOpen)
       at System.IO.StreamReader..ctor(Stream stream, Boolean detectEncodingFromByteOrderMarks)
       at System.IO.StreamReader..ctor(Stream stream)
       at Raven.Abstractions.Extensions.ExceptionExtensions.TryReadResponseIfWebException(Exception ex)
       at Raven.Abstractions.Extensions.ExceptionExtensions.TryReadErrorResponseObject[T](Exception ex, T protoTypeObject)
       at Raven.Client.Connection.ServerClient.ShouldRethrowIndexException(WebException e, Exception& newEx)
       at Raven.Client.Connection.ServerClient.DirectPutIndex(String name, OperationMetadata operationMetadata, Boolean overwrite, IndexDefinition definition)
       at Raven.Client.Connection.ServerClient.<>c__DisplayClass5e.<PutIndex>b__5d(OperationMetadata u)
       at Raven.Client.Connection.ReplicationInformer.TryOperation[T](Func`2 operation, OperationMetadata operationMetadata, OperationMetadata primaryOperationMetadata, Boolean avoidThrowing, T& result, Boolean& wasTimeout)
       at Raven.Client.Connection.ReplicationInformer.ExecuteWithReplication[T](String method, String primaryUrl, OperationCredentials primaryCredentials, Int32 currentRequest, Int32 currentReadStripingBase, Func`2 operation)
       at Raven.Client.Connection.ServerClient.ExecuteWithReplication[T](String method, Func`2 operation)
       at Raven.Client.Connection.ServerClient.PutIndex(String name, IndexDefinition definition, Boolean overwrite)
       at Raven.Client.Indexes.AbstractIndexCreationTask.Execute(IDatabaseCommands databaseCommands, DocumentConvention documentConvention)
       at Raven.Client.DocumentStoreBase.ExecuteIndex(AbstractIndexCreationTask indexCreationTask)
       at Raven.Client.Indexes.AbstractIndexCreationTask.Execute(IDocumentStore store)
       at NServiceBus.Features.RavenDbTimeoutStorage.Setup(FeatureConfigurationContext context) in c:\BuildAgent\work\c4d62ce02b983878\src\NServiceBus.RavenDB\Timeouts\RavenDbTimeoutStorage.cs:line 48
       at NServiceBus.Features.Feature.SetupFeature(FeatureConfigurationContext config) in c:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Features\Feature.cs:line 203
       at NServiceBus.Features.FeatureActivator.ActivateFeature(FeatureState featureState, IEnumerable`1 featuresToActivate, FeatureConfigurationContext context) in c:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Features\FeatureActivator.cs:line 233
       at NServiceBus.Features.FeatureActivator.SetupFeatures(FeatureConfigurationContext context) in c:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Features\FeatureActivator.cs:line 118
       at NServiceBus.Configure.Initialize() in c:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Configure.cs:line 115
       at NServiceBus.Bus.Create(BusConfiguration configuration) in c:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\Bus.cs:line 21
       at NServiceBus.GenericHost.PerformConfiguration(Action`1 moreConfiguration) in c:\BuildAgent\work\a3de8759ee491634\src\NServiceBus.Hosting.Windows\GenericHost.cs:line 121
       at NServiceBus.GenericHost.Start() in c:\BuildAgent\work\a3de8759ee491634\src\NServiceBus.Hosting.Windows\GenericHost.cs:line 70
       at NServiceBus.Hosting.Windows.WindowsHost.Start() in c:\BuildAgent\work\a3de8759ee491634\src\NServiceBus.Hosting.Windows\WindowsHost.cs:line 30
       at NServiceBus.Hosting.Windows.Program.<>c__DisplayClasse.<Main>b__6(WindowsHost service) in c:\BuildAgent\work\a3de8759ee491634\src\NServiceBus.Hosting.Windows\Program.cs:line 80
       at Topshelf.Internal.ControllerDelegates`1.StartActionObject(Object obj) in c:\Projects\TopShelfForNSB\src\Topshelf\Internal\ControllerDelegates.cs:line 18
       at Topshelf.Internal.IsolatedServiceControllerWrapper`1.<>c__DisplayClass2.<set_StartAction>b__1(TService service) in c:\Projects\TopShelfForNSB\src\Topshelf\Internal\IsolatedServiceControllerWrapper.cs:line 65
       at Topshelf.Internal.ServiceController`1.<.cctor>b__1(ServiceController`1 sc) in c:\Projects\TopShelfForNSB\src\Topshelf\Internal\ServiceController.cs:line 35
       at Magnum.StateMachine.LambdaAction`1.Execute(T instance, Event event, Object parameter) in :line 0
       at Magnum.StateMachine.EventActionList`1.Execute(T stateMachine, Event event, Object parameter) in :line 0

[synhershko]

Thanks for reporting, I know what's going on but need to find a reliable way for us to launch and make sure index definitions were put. I'll have a fix in later this week.

[sanilpaul]

@synhershko just checking back. Any success with this issue?

[synhershko]

Yes, I'll push this early next week

[sanilpaul]

@synhershko Thanks

[synhershko]

@sanilpaul pushed a fix for this on NSB's end, and opened an issue with RavenDB (although we don't rely on their fix) http://issues.hibernatingrhinos.com/issue/RavenDB-3311

Can you please take this for a spin?

Code: https://github.com/Particular/NServiceBus.RavenDB/tree/hotfix-2.1.1

Artifacts: http://builds.particular.net/viewLog.html?buildId=100180&tab=artifacts&buildTypeId=NServiceBusRavenDB_1Buil

[sanilpaul]

@synhershko It looks fixed on my machine. Thank you !

[synhershko]

@johnsimons @danielmarbach @andreasohlund does this warrant a hotfix release or should I just merge with develop so its included in future releases?

[danielmarbach]

I'd say this does not qualify for a hotfix because we are not loosing messages.

On Tue, Mar 17, 2015 at 1:16 PM, Itamar Syn-Hershko < notifications@github.com> wrote:

@johnsimons https://github.com/johnsimons @danielmarbach https://github.com/danielmarbach @andreasohlund https://github.com/andreasohlund does this warrant a hotfix release or should I just merge with develop so its included in future releases?

— Reply to this email directly or view it on GitHub https://github.com/Particular/NServiceBus.RavenDB/issues/65#issuecomment-82315598 .

Daniel Marbach

Solution Architect, Particular Software

Email: daniel.marbach@particular.net

Web: www.particular.net http://t.signauxneuf.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJN7t5XX4RzDjlVS9QB88pTc_dW3Ljg7x56dWyBdgBXs402?t=http%3A%2F%2Fwww.particular.net%2F&si=4896787407044608&pi=33cc6202-1f31-4040-8d6c-e70641f44bba

[sanilpaul]

@synhershko @danielmarbach Agree that messages are not lost. But starting from a clean state is always painful as the failed ones have to be restarted or endpoints has to be staggered. if you guys don't plan to put out a hotfix, now that its fixed, Is there an estimate on when we can expect next release?

[synhershko]

It seems like we are going to have a release very soon, so I'd say no need for hotfixing this one as it mostly only affects development mode

[sanilpaul]

@synhershko Thanks

[synhershko]

The underlying issue was just fixed on RavenDB itself, and the next stable release should have it: https://github.com/ayende/ravendb/commit/c3a827cd19b79dc6075a977e423b8d68e37ab204 (on the 3.0 branch)

[SimonCropp]

hmm a lock. assuming that is not in the fast path?

[synhershko]

No, it should only be accessed on startup in our scenario

[SimonCropp]

fair enough