yuzd / Hangfire.HttpJob

httpjob for Hangfire,restful api for Hangfire,job调度与业务分离
https://github.com/yuzd/Hangfire.HttpJob/wiki
MIT License
628 stars 185 forks source link

job按每分钟执行一次,开多个线程推送大量数据就会报异常什么原因呢,直接调用这个接口不会报错? #63

Closed arvinws closed 4 years ago

arvinws commented 4 years ago

Failed An exception occurred during performance of the job.

System.Threading.Tasks.TaskCanceledException The operation was canceled.

System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: 由于线程退出或应用程序请求,已中止 I/O 操作。. ---> System.Net.Sockets.SocketException (995): 由于线程退出或应用程序请求,已中止 I/O 操作。 --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Hangfire.HttpJob.Server.HttpJob.Run(HttpJobItem item, PerformContext context, List1 logList, HttpJobItem parentJob) at Hangfire.HttpJob.Server.HttpJob.Excute(HttpJobItem item, String jobName, String queuename, Boolean isretry, PerformContext context) +64msProcessing Server: W01:88320 Worker: c29faf1b +78ms任务开始执行,执行时间:2020-07-25 19:58:13 +108ms任务名称:PushAccessControlInfos|队列名称:DEFAULT01 +133ms参数:【{"Success":null,"Fail":null,"Url":"http://localhost:5000/api/SzApi/PushAccessControlInfos","Method":"POST","Data":"","ContentType":"application/json","Timeout":5000,"DelayFromMinutes":15,"Cron":" *","JobName":"PushAccessControlInfos","QueueName":"DEFAULT01","AgentClass":"","SendSuccess":false,"SendFail":true,"Mail":"","EnableRetry":false,"RetryDelaysInSeconds":"20,30,60","RetryTimes":3,"BasicUserName":"","BasicPassword":"","Headers":{},"CallbackEL":"","TimeZone":"","DingTalk":null}】 +6.960s【HttpJob Timeout】:5000ms +7.018sSystem.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: 由于线程退出或应用程序请求,已中止 I/O 操作。. ---> System.Net.Sockets.SocketException (995): 由于线程退出或应用程序请求,已中止 I/O 操作。 --- End of inner exception stack trace --- at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token) at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Hangfire.HttpJob.Server.HttpJob.Run(HttpJobItem item, PerformContext context, List1 logList, HttpJobItem parentJob) at Hangfire.HttpJob.Server.HttpJob.Excute(HttpJobItem item, String jobName, String queuename, Boolean isretry, PerformContext context)

arvinws commented 4 years ago

image

arvinws commented 4 years ago

根据你提供的dmeo测试几次,捕获到异常的地方 https://github.com/yuzd/Hangfire.HttpJob/blob/master/Hangfire.HttpJob/Server/HttpJob.cs image

yuzd commented 4 years ago

这个错误是请求出现异常导致的。 需要确认下 你的这个接口 如果直接调用会存在这个问题吗?需要配合记录下日志排查。

你描述说有并发问题?并发是从哪里来的呢, hangfire是调度执行,同一个job的并发应该不存在才对。

arvinws commented 4 years ago

这个错误是请求出现异常导致的。

需要确认下 你的这个接口 如果直接调用会存在这个问题吗?需要配合记录下日志排查。

你描述说有并发问题?并发是从哪里来的呢,

hangfire是调度执行,同一个job的并发应该不存在才对。

直接调用接口不会出现这种情况,只有调度执行的时候会出现。我的接口里面有开多线程运行,用的task,我昨天测试了,如果接口只是开几个线程,httpjob调度执行并不会出现异常。但是如果开启的线程达到几十个,就会报异常。这个错误是你的httpjob类里面方法报出来的,并不是我接口报错。

arvinws commented 4 years ago

这个错误是请求出现异常导致的。 需要确认下 你的这个接口 如果直接调用会存在这个问题吗?需要配合记录下日志排查。

你描述说有并发问题?并发是从哪里来的呢, hangfire是调度执行,同一个job的并发应该不存在才对。

nlog-all-2020-07-26.log 这个是刚刚测试的日志

yuzd commented 4 years ago

你接口里面开多线程 应该和调度无关啊。调度只是去请求你的接口而已。除非你的接口超时了才会报那个错误信息。要么就是你的接口所在的服务的线程已经用完。 建议你在你的接口进入和返回都加上log 进行排查。 因为你只看调度这边报的错 其实就是http请求报的错。我认为原因不在调度,所以希望你这边能配合查下你的接口。

arvinws commented 4 years ago

你接口里面开多线程 应该和调度无关啊。调度只是去请求你的接口而已。除非你的接口超时了才会报那个错误信息。要么就是你的接口所在的服务的线程已经用完。 建议你在你的接口进入和返回都加上log 进行排查。 因为你只看调度这边报的错 其实就是http请求报的错。我认为原因不在调度,所以希望你这边能配合查下你的接口。

接口所在的服务线程已经用完是什么意思?我的接口要加上哪些日志来排查原因呢,因为我这边直接调用接口 一切都运行正常,而且就算是httpjob来调度我的接口 我的接口也没有报错,也是正常走完整个流程。所以还请大佬指点下

arvinws commented 4 years ago

image 这里的超时时间我也改过了 还是报异常。

arvinws commented 4 years ago

image image 这里两个超时时间,分别有什么效果呢?我刚刚又把单独接口里的json参数配置 TimeOut的时间加长了一点,好像没有问题不报错了。

yuzd commented 4 years ago

下面的超时时间才是控制访问接口超时