search

sciserver / graywulf

Database cluster federation toolkit
2 stars 3 forks source link

Jobs throwing exceptions in Cancel might cause scheduler to crash #88

Open dobos opened 6 years ago

dobos commented 6 years ago

2017-12-06T17:05:00 Unhandled Exception: System.ServiceModel.CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

Server stack trace: at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen() at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Jhu.Graywulf.Tasks.ICancelableTask.Cancel() at Jhu.Graywulf.Tasks.CancelableCollection.Cancel() in C:\Data\dobos\project\skyquery-all-v1.3\graywulf\dll\Jhu.Graywulf.RemoteService\Tasks\CancelableCollection.cs:line 86 at Jhu.Graywulf.Activities.JobAsyncCodeActivity.Cancel(AsyncCodeActivityContext activityContext) in C:\Data\dobos\project\skyquery-all-v1.3\graywulf\dll\Jhu.Graywulf.Activities\Activities\JobAsyncCodeActivity.cs:line 132 at System.Activities.AsyncCodeActivity.InternalCancel(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager) at System.Activities.AsyncCodeActivity.InternalAbort(ActivityInstance instance, ActivityExecutor executor, Exception terminationReason) at System.Activities.ActivityInstance.Abort(ActivityExecutor executor, BookmarkManager bookmarkManager, Exception terminationReason, Boolean isTerminate) at System.Activities.Runtime.ActivityExecutor.Abort(Exception terminationException, Boolean isTerminate) at System.Activities.Runtime.ActivityExecutor.Abort(Exception reason) at System.Activities.Hosting.WorkflowInstance.WorkflowInstanceControl.Abort(Exception reason) at System.Activities.WorkflowApplication.OnAbortWaitComplete(Object state, TimeoutException exception) at System.Activities.WorkflowApplication.OnWaitAsyncComplete(Object state, TimeoutException exception) at System.Runtime.AsyncWaitHandle.AsyncWaiter.Invoke() at System.Runtime.ActionItem.CallbackHelper.InvokeWithoutContext(Object state) at System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped nativeOverlapped) at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped nativeOverlapped) at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)

dobos commented 6 years ago

[System.InvalidOperationException] The activity 'CopyRemoteTable' with ID 493 threw or propagated an exception while being canceled.

[System.ServiceModel.CommunicationObjectFaultedException] The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

Server stack trace: at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen() at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Jhu.Graywulf.Tasks.ICancelableTask.Cancel() at Jhu.Graywulf.Tasks.CancelableCollection.Cancel() in C:\Data\dobos\project\skyquery-all-v1.3\graywulf\dll\Jhu.Graywulf.RemoteService\Tasks\CancelableCollection.cs:line 86 at Jhu.Graywulf.Activities.JobAsyncCodeActivity.Cancel(AsyncCodeActivityContext activityContext) in C:\Data\dobos\project\skyquery-all-v1.3\graywulf\dll\Jhu.Graywulf.Activities\Activities\JobAsyncCodeActivity.cs:line 132 at System.Activities.AsyncCodeActivity.InternalCancel(ActivityInstance instance, ActivityExecutor executor, BookmarkManager bookmarkManager) at System.Activities.Runtime.ActivityExecutor.CancelActivityWorkItem.Execute(ActivityExecutor executor, BookmarkManager bookmarkManager)