Open MindHunter86 opened 2 years ago
concurrent errors. need grpc refactoring or migration on ws(s)
time.Sleep in this block before redeploy()
not fixed
4:10PM DBG running 1min jobs... func=app.(*cron).runCronTasks
4:10PM DBG reannounces is not locked; running job... func=app.(*cron).reannounce
4:10PM DBG running 5min jobs... func=app.(*cron).runCronTasks
4:10PM WRN abnormal response from worker service error_code=DeadlineExceeded error_message="context deadline exceeded" func=master.(*worker).getRPCErrors
4:10PM ERR got an error in force reannounce request error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" func=app.(*cron).reannounce.func1
4:10PM WRN abnormal response from worker service error_code=DeadlineExceeded error_message="context deadline exceeded" func=master.(*worker).getRPCErrors
4:10PM ERR could not get torrents from the given worker id; skipping... error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" func=app.(*deploy).getWorkersTorrentsV2 worker_id=70e40133-4c0c-4859-a04d-e1c7e5f2c74c
4:10PM ERR got an error in deploy failed_announces request error="could not continue the delpoy process because one of workers errors" func=app.(*cron).reannounce.func1
4:10PM DBG deploy updates is not locked; running job... func=app.(*cron).redeploy
4:10PM DBG mask after switch cron_tasks=1 func=app.(*cron).runCronTasks
4:10PM DBG Called getResponse. func=anilibria.(*ApiClient).getApiResponse
4:10PM INF Correct response api_method=/getUpdates func=anilibria.(*ApiClient).getApiResponse
4:10PM ERR ATTENTION!!! Ignoring title with id 3996 or 8452; please, fix this shit immediately func=app.(*deploy).getAnilibriaUpdatesTorrents
4:10PM DBG worker reply accepted, authorizing... func=master.(*worker).authorizeSerivceReply worker_id=70e40133-4c0c-4859-a04d-e1c7e5f2c74c
4:10PM DBG the worker's reply has been authorized func=master.(*worker).authorizeSerivceReply worker_id=70e40133-4c0c-4859-a04d-e1c7e5f2c74c
4:10PM DBG got reply from the worker with torrents list func=master.(*worker).getTorrents torrents_count=45
4:11PM DBG mask before switch cron_tasks=0 func=app.(*cron).runCronTasks
4:11PM DBG running 1min jobs... func=app.(*cron).runCronTasks
4:11PM DBG reannounces is not locked; running job... func=app.(*cron).reannounce
4:11PM DBG mask after switch cron_tasks=2 func=app.(*cron).runCronTasks
4:11PM DBG worker reply accepted, authorizing... func=master.(*worker).authorizeSerivceReply worker_id=70e40133-4c0c-4859-a04d-e1c7e5f2c74c
4:11PM DBG the worker's reply has been authorized func=master.(*worker).authorizeSerivceReply worker_id=70e40133-4c0c-4859-a04d-e1c7e5f2c74c
4:11PM DBG worker reply accepted, authorizing... func=master.(*worker).authorizeSerivceReply worker_id=70e40133-4c0c-4859-a04d-e1c7e5f2c74c
4:11PM DBG the worker's reply has been authorized func=master.(*worker).authorizeSerivceReply worker_id=70e40133-4c0c-4859-a04d-e1c7e5f2c74c
4:11PM DBG got reply from the worker with torrents list func=master.(*worker).getTorrents torrents_count=45
4:11PM DBG torrent marked as failed func=app.(*deploy).searchFailedTitles torrents_hash=06f93ea97
filter=id%2Ccode%2Cnames%2Cupdated%2Clast_change%2Cstatus%2Ctype%2Ctorrents&search=Golden+Kamuy+4
4:11PM DBG api reqiest has been completed with response 200 OK api_method=/searchTitles func=anilibria.(*ApiClient).getApiResponseV2
4:11PM WRN got a problem in searching failed titles; there are none, two or more titles in the result; manual search required func=app.(*deploy).searchFailedTitles title_name="Golden Kamuy 4" titles_count=4 torrent_hash=06f93ea97
4:11PM WRN failed torrent will be skipped and not deleted func=app.(*deploy).searchFailedTitles title_name="Golden Kamuy 4" torrent_hash=06f93ea97
4:12PM DBG mask before switch cron_tasks=0 func=app.(*cron).runCronTasks
4:12PM DBG running 1min jobs... func=app.(*cron).runCronTasks
4:12PM DBG reannounces is not locked; running job... func=app.(*cron).reannounce
4:12PM DBG mask after switch cron_tasks=2 func=app.(*cron).runCronTasks
4:12PM WRN abnormal response from worker service error_code=DeadlineExceeded error_message="context deadline exceeded" func=master.(*worker).getRPCErrors
4:12PM ERR got an error in force reannounce request error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" func=app.(*cron).reannounce.func1
4:12PM WRN abnormal response from worker service error_code=Internal error_message="found protocol version 107 but expected 1" func=master.(*worker).getRPCErrors
4:12PM ERR could not get torrents from the given worker id; skipping... error="rpc error: code = Internal desc = found protocol version 107 but expected 1" func=app.(*deploy).getWorkersTorrentsV2 worker_id=70e40133-4c0c-4859-a04d-e1c7e5f2c74c
4:12PM ERR got an error in deploy failed_announces request error="could not continue the delpoy process because one of workers errors" func=app.(*cron).reannounce.func1
the problem occurs when the eth0 link is 100% busy
when reannounce job in cron is enabled, grpc connection breaks with error "found protocol version 21 but expected 1"
Logs master - https://hastebin.com/ucacifipan.apache worker - https://hastebin.com/omaqekodop.http