search

TheCodeKing / XDMessaging.Net

Easy-to-use, zero configuration solution for PubSub messaging across application and network boundaries.
http://thecodeking.co.uk/project/xdmessaging/
Other
61 stars 21 forks source link

Occasional Unhandled IOException Crashing Application #12

Closed NimbusHex closed 7 years ago

NimbusHex commented 7 years ago

I'm getting an occasional unhandled exception from a couple machines out of a ton that are running in production using XDMessaging.Lite with XDTransportMode.Compatibility.

System.IO.IOException: There was an unexpected IO error binding to a channel. Ensure the process is unable to read/write to directory 'C:\ProgramData\XDMessagingv4\APSSCommChannel\c47a0002-832d-4c34-9d00-bc8310cbb18b.msg'. ---> System.IO.IOException: The process cannot access the file 'C:\ProgramData\XDMessagingv4\APSSCommChannel\c47a0002-832d-4c34-9d00-bc8310cbb18b.msg' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at XDMessaging.Transport.IOStream.XDIOStreamListener.ProcessMessage(String fullPath)
   --- End of inner exception stack trace ---

Server stack trace: 
   at XDMessaging.Transport.IOStream.XDIOStreamListener.ProcessMessage(String fullPath)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
   at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)
   at System.Action`1.EndInvoke(IAsyncResult result)
   at System.Runtime.Remoting.Messaging.AsyncResult.SyncProcessMessage(IMessage msg)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
   at System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.ThreadPoolCallBack(Object o)
   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

When doing send operations, I'm not doing any kind of thread safety checks, do I need to make sure SendToChannel gets called in a thread-safe way?

Thanks for this awesome library!

TheCodeKing commented 7 years ago

Hi,

Glad you like it. Sounds like the messages might be getting cleaned up before all threads have finished processing (perhaps cause under so much load), or there is an issue they threads are trying to read a message before the lock is released from a write. This shouldn't happen as it uses Windows events to trigger when a new message is ready for processing.

I'll have a dig through and see if I can improve the error handling or find a cause. You shouldn't need to worry about thread safety.

Thanks, Mike

On 9 May 2017 at 22:00, NimbusHex notifications@github.com wrote:

I'm getting an occasional unhandled exception from a couple machines out of a ton that are running in production using XDMessaging.Lite with XDTransportMode.Compatibility.

System.IO.IOException: There was an unexpected IO error binding to a channel. Ensure the process is unable to read/write to directory 'C:\ProgramData\XDMessagingv4\APSSCommChannel\c47a0002-832d-4c34-9d00-bc8310cbb18b.msg'. ---> System.IO.IOException: The process cannot access the file 'C:\ProgramData\XDMessagingv4\APSSCommChannel\c47a0002-832d-4c34-9d00-bc8310cbb18b.msg' because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share) at XDMessaging.Transport.IOStream.XDIOStreamListener.ProcessMessage(String fullPath) --- End of inner exception stack trace ---

Server stack trace: at XDMessaging.Transport.IOStream.XDIOStreamListener.ProcessMessage(String fullPath) at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs) at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)

Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase) at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData) at System.Action`1.EndInvoke(IAsyncResult result) at System.Runtime.Remoting.Messaging.AsyncResult.SyncProcessMessage(IMessage msg) at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink) at System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.ThreadPoolCallBack(Object o) at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

When doing send operations, I'm not doing any kind of thread safety checks, do I need to make sure SendToChannel gets called in a thread-safe way?

Thanks for this awesome library!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/TheCodeKing/XDMessaging.Net/issues/12, or mute the thread https://github.com/notifications/unsubscribe-auth/AA4Hju2mz661a7zr-oWi4i4wVVTfxEPzks5r4NQEgaJpZM4NV3_w .

TheCodeKing commented 7 years ago

I reproduced the issue under heavy load. In IOStream mode messages were deleted after 5 seconds. If a process is too busy, there can be a race condition whereby the message is deleted by a cleanup task just as another thread tries to read the message.

I've increased the default timeout to 10 seconds in 5.0.5 to allow more room for processing. This can also be overridden with an AppSetting called "IoStreamMessageTimeoutInMilliseconds".

NimbusHex commented 7 years ago

Fantastic, thank you!