search

crawlab-team / crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
https://www.crawlab.cn
BSD 3-Clause "New" or "Revised" License
11.37k stars 1.79k forks source link

Pro Version random GRPC connection errors in the middle of task execution. #1391

Open Voyewodayer opened 1 year ago

Voyewodayer commented 1 year ago

Describe the bug While executing a task that runs for quite a long time (>1 day) docker logs tend to show various errors related to GRPC connection such as rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp <MASTER_NODE_IP_ADDRESS>:<GRPC_PORT>: connect: connection refused or connection reset by peer. Both worker node on which task runs and master node machines show continuous, uninterrupted availability network and hardware wise. Sometimes it is accompanied by errors related to failure of verifying license error verify license error: Post \"https://license.crawlab.cn/release/license/verify\": net/http: TLS handshake timeout. retry in 5 seconds.

Expected behavior Nodes don't lose GRPC connection randomly

tikazyq commented 1 year ago

Can you dump the master node docker logs?

Voyewodayer commented 1 year ago

Below there are logs from master node from most recent such occurence. Logs are timestamped between 1 minute before it happened to 1 minute after. Crawlab container did rebuild itself at exactly 2023-09-13T16:30:47.886726955Z. Around the same time, as we can see on line 15 of the logs, panic: runtime error: index out of range [243] with length 62 error happened. Afterwards service did reset.

Logs > ```2023-09-13T16:30:00.001195402Z 2023/09/14 00:30:00 info [SpiderAdminService] finished sync git > 2023-09-13T16:30:00.679442141Z [GIN] 2023/09/14 - 00:30:00 | 200 | 2.745628ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" > 2023-09-13T16:30:02.539992260Z [GIN] 2023/09/14 - 00:30:02 | 200 | 371.551614ms | 127.0.0.1 | GET "/metrics" > 2023-09-13T16:30:05.679892343Z [GIN] 2023/09/14 - 00:30:05 | 200 | 2.698525ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" > 2023-09-13T16:30:10.682191045Z [GIN] 2023/09/14 - 00:30:10 | 200 | 2.474229ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" > 2023-09-13T16:30:15.700731204Z [GIN] 2023/09/14 - 00:30:15 | 200 | 2.06508ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" > 2023-09-13T16:30:17.530844865Z [GIN] 2023/09/14 - 00:30:17 | 200 | 363.273375ms | 127.0.0.1 | GET "/metrics" > 2023-09-13T16:30:20.683237154Z [GIN] 2023/09/14 - 00:30:20 | 200 | 2.886519ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" > 2023-09-13T16:30:25.693064357Z [GIN] 2023/09/14 - 00:30:25 | 200 | 3.350451ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" > 2023-09-13T16:30:30.683004760Z [GIN] 2023/09/14 - 00:30:30 | 200 | 2.76009ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" > 2023-09-13T16:30:32.540234105Z [GIN] 2023/09/14 - 00:30:32 | 200 | 372.425448ms | 127.0.0.1 | GET "/metrics" > 2023-09-13T16:30:35.683029499Z [GIN] 2023/09/14 - 00:30:35 | 200 | 2.692134ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" > 2023-09-13T16:30:40.683991587Z [GIN] 2023/09/14 - 00:30:40 | 200 | 2.399953ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" > 2023-09-13T16:30:45.703194226Z [GIN] 2023/09/14 - 00:30:45 | 200 | 2.537904ms | 172.21.0.3 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" > 2023-09-13T16:30:47.499802207Z panic: runtime error: index out of range [243] with length 62 > 2023-09-13T16:30:47.499842700Z > 2023-09-13T16:30:47.499846920Z goroutine 2559365 [running]: > 2023-09-13T16:30:47.499850850Z github.com/emirpasic/gods/lists/arraylist.(*List).Add(...) > 2023-09-13T16:30:47.499854241Z /go/pkg/mod/github.com/emirpasic/gods@v1.18.1/lists/arraylist/arraylist.go:47 > 2023-09-13T16:30:47.499858001Z github.com/crawlab-team/crawlab-pro/core/metric.(*Service).getAllNodeMetrics.func1.1({0xc0024f2000?, 0xd4, 0xc001f65e70?}) > 2023-09-13T16:30:47.499861631Z /go/src/app/core/metric/service.go:226 +0x22d > 2023-09-13T16:30:47.499864811Z github.com/crawlab-team/crawlab-pro/core/metric.(*Service).requestRemoteNodeMetrics(0xc000a1b3b0, {0x229e148, 0xc001077500}, {0x22aaca0, 0xc001f71ba0}, 0xc0000fc870?, 0xc00119bee8) > 2023-09-13T16:30:47.499868422Z /go/src/app/core/metric/service.go:334 +0x4da > 2023-09-13T16:30:47.499871842Z github.com/crawlab-team/crawlab-pro/core/metric.(*Service).getAllNodeMetrics.func1({{0x64, 0xcb, 0x50, 0xd5, 0x59, 0xbf, 0x44, 0x6b, 0x80, 0x9c, ...}, ...}) > 2023-09-13T16:30:47.499875282Z /go/src/app/core/metric/service.go:224 +0xfa > 2023-09-13T16:30:47.499878852Z created by github.com/crawlab-team/crawlab-pro/core/metric.(*Service).getAllNodeMetrics > 2023-09-13T16:30:47.499892744Z /go/src/app/core/metric/service.go:223 +0x40c > 2023-09-13T16:30:48.028047289Z 2023/09/14 00:30:48 info [NotificationService] start handling events > 2023-09-13T16:30:48.028150446Z 2023/09/14 00:30:48 info [NotificationService] attempt to obtain grpc stream... > 2023-09-13T16:30:48.035790910Z 2023/09/14 00:30:48 info initialized controllers successfully > 2023-09-13T16:30:48.035805620Z 2023/09/14 00:30:48 info initialized middlewares successfully > 2023-09-13T16:30:48.036093351Z 2023/09/14 00:30:48 info initialized routes successfully > 2023-09-13T16:30:48.050108956Z 2023/09/14 00:30:48 info current node type: master > 2023-09-13T16:30:48.050120687Z 2023/09/14 00:30:48 info running in docker container > 2023-09-13T16:30:48.056160645Z 2023/09/14 00:30:48 info process created > 2023-09-13T16:30:48.073410993Z 2023/09/14 00:30:48 info process started > 2023-09-13T16:30:48.094045077Z [nginx] * Starting nginx nginx > 2023-09-13T16:30:48.112639794Z [nginx] ...done. > 2023-09-13T16:30:48.124166549Z ts=2023-09-13T16:30:48.124Z caller=main.go:535 level=info msg="Starting Prometheus Server" mode=server version="(version=2.37.0, branch=HEAD, revision=b41e0750abf5cc18d8233161560731de05199330)" > 2023-09-13T16:30:48.124265346Z ts=2023-09-13T16:30:48.124Z caller=main.go:540 level=info build_context="(go=go1.18.4, user=root@0ebb6827e27f, date=20220714-15:13:18)" > 2023-09-13T16:30:48.124347572Z ts=2023-09-13T16:30:48.124Z caller=main.go:541 level=info host_details="(Linux 6.2.0-1011-aws #11~22.04.1-Ubuntu SMP Mon Aug 21 16:27:59 UTC 2023 x86_64 d94895de835b (none))" > 2023-09-13T16:30:48.124430108Z ts=2023-09-13T16:30:48.124Z caller=main.go:542 level=info fd_limits="(soft=1048576, hard=1048576)" > 2023-09-13T16:30:48.124506303Z ts=2023-09-13T16:30:48.124Z caller=main.go:543 level=info vm_limits="(soft=unlimited, hard=unlimited)" > 2023-09-13T16:30:48.128543475Z ts=2023-09-13T16:30:48.128Z caller=web.go:553 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090 > 2023-09-13T16:30:48.128969857Z ts=2023-09-13T16:30:48.128Z caller=main.go:972 level=info msg="Starting TSDB ..." > 2023-09-13T16:30:48.129318592Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1691863202169 maxt=1692057600000 ulid=01H7VS25N0N02EQSXQHX1HX3MY > 2023-09-13T16:30:48.129418209Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1692057602167 maxt=1692252000000 ulid=01H81JERK0MNXN208KY4CD1QSE > 2023-09-13T16:30:48.129508896Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1692252002167 maxt=1692446400000 ulid=01H87BTXQNTQ1QK6A23D9J1X4E > 2023-09-13T16:30:48.129593812Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1692446402167 maxt=1692640800000 ulid=01H8D57Q3452QTS4CKV817MFP2 > 2023-09-13T16:30:48.129682358Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1692640802172 maxt=1692835200000 ulid=01H8JYMEHHPCFY0KJYECPSEMB7 > 2023-09-13T16:30:48.129793816Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1692835200000 maxt=1693029600000 ulid=01H8RH4RZFEZNPNV2E0706WPSM > 2023-09-13T16:30:48.129895274Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693029600000 maxt=1693224000000 ulid=01H8YAHCSRB190BC7GX09ZCERJ > 2023-09-13T16:30:48.129972059Z ts=2023-09-13T16:30:48.129Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693224002168 maxt=1693418400000 ulid=01H94AT8JEBMTGPG395XDB4D9Q > 2023-09-13T16:30:48.130060405Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693418402167 maxt=1693612800000 ulid=01H9A46Y5GEB9MBYYR73R4ZNAF > 2023-09-13T16:30:48.130138811Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693612802167 maxt=1693807200000 ulid=01H9FXKFJ7S9M4HZZGPCNHHW27 > 2023-09-13T16:30:48.130234048Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1693807202167 maxt=1694001600000 ulid=01H9NQ01NRA4TJJYRQ80KV6FRH > 2023-09-13T16:30:48.130321995Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694001602169 maxt=1694196000000 ulid=01H9VGCN18QM0GXZJ7099R5PA2 > 2023-09-13T16:30:48.130407651Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694196002172 maxt=1694390400000 ulid=01HA19S8CB3XBZ3S86GBDE1Q02 > 2023-09-13T16:30:48.130492657Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694584802167 maxt=1694592000000 ulid=01HA6WA4QYSWDD6EGQQ5R97QPW > 2023-09-13T16:30:48.130559062Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694592000000 maxt=1694599200000 ulid=01HA735DBF53ZW8F1HKW2DFFD0 > 2023-09-13T16:30:48.130673510Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694390402167 maxt=1694584800000 ulid=01HA735FM7S727FVXTC0B7EVKH > 2023-09-13T16:30:48.130732974Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694599202167 maxt=1694606400000 ulid=01HA7A1K7ZS8676T3HXJC9Y6QW > 2023-09-13T16:30:48.130818120Z ts=2023-09-13T16:30:48.130Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1694606402167 maxt=1694613600000 ulid=01HA7GXAGEA7C2EHET3NAAK42M > 2023-09-13T16:30:48.130988453Z ts=2023-09-13T16:30:48.130Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/data/prometheus/lock > 2023-09-13T16:30:48.131119902Z ts=2023-09-13T16:30:48.131Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false > 2023-09-13T16:30:48.164358649Z 2023/09/14 00:30:48 info grpc server listens to 0.0.0.0:9666 > 2023-09-13T16:30:48.164842774Z 2023/09/14 00:30:48 info master[4abd561c-3138-11ee-a3dd-0242ac120002] exists in db > 2023-09-13T16:30:48.166868181Z 2023/09/14 00:30:48 info updated master[4abd561c-3138-11ee-a3dd-0242ac120002] in db. id: 64cb50d559bf446b809c6a62 > 2023-09-13T16:30:48.166956257Z 2023/09/14 00:30:48 info master[4abd561c-3138-11ee-a3dd-0242ac120002] monitoring started > 2023-09-13T16:30:48.169513262Z 2023/09/14 00:30:48 info [GrpcClient] grpc client connected to localhost:9666 > 2023-09-13T16:30:48.169524753Z 2023/09/14 00:30:48 info [GrpcClient] grpc client registered client services > 2023-09-13T16:30:48.169592698Z 2023/09/14 00:30:48 info cron: start > 2023-09-13T16:30:48.170579110Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.170493093 +0800 CST entry 1 next 2023-09-15 09:00:00 +0800 CST > 2023-09-13T16:30:48.170662996Z 2023/09/14 00:30:48 info [GrpcClient] grpc client connected to localhost:9666 > 2023-09-13T16:30:48.170671566Z 2023/09/14 00:30:48 info [GrpcClient] grpc client registered client services > 2023-09-13T16:30:48.175231716Z 2023/09/14 00:30:48 info [GrpcClient] grpc client subscribed to remote server > 2023-09-13T16:30:48.175360726Z 2023/09/14 00:30:48 info subscribe events > 2023-09-13T16:30:48.175465383Z 2023/09/14 00:30:48 info [GrpcClient] grpc client subscribed to remote server > 2023-09-13T16:30:48.175553690Z 2023/09/14 00:30:48 info [GrpcClient] start handling stream message... > 2023-09-13T16:30:48.175621995Z 2023/09/14 00:30:48 info [GrpcClient] start handling stream message... > 2023-09-13T16:30:48.175852781Z 2023/09/14 00:30:48 info [NodeServer] master received subscribe request from node[4abd561c-3138-11ee-a3dd-0242ac120002] > 2023-09-13T16:30:48.175861732Z 2023/09/14 00:30:48 info [NodeServer] master subscribed node[4abd561c-3138-11ee-a3dd-0242ac120002] > 2023-09-13T16:30:48.194122604Z 2023/09/14 00:30:48 info [NodeServer] master received subscribe request from node[4abd561c-3138-11ee-a3dd-0242ac120002] > 2023-09-13T16:30:48.194135145Z 2023/09/14 00:30:48 info [NodeServer] master subscribed node[4abd561c-3138-11ee-a3dd-0242ac120002] > 2023-09-13T16:30:48.194996458Z 2023/09/14 00:30:48 info [MessageServer] received connect request from node[4abd561c-3138-11ee-a3dd-0242ac120002], key: plugin:notification:4abd561c-3138-11ee-a3dd-0242ac120002 > 2023-09-13T16:30:48.211657454Z mongo: no documents in result > 2023-09-13T16:30:48.211668385Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.211672266Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() > 2023-09-13T16:30:48.211675766Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() > 2023-09-13T16:30:48.211678956Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:48.211682256Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() > 2023-09-13T16:30:48.211685387Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:48.211688507Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() > 2023-09-13T16:30:48.211691657Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() > 2023-09-13T16:30:48.211694857Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() > 2023-09-13T16:30:48.211698057Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() > 2023-09-13T16:30:48.211700938Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1616 google.golang.org/grpc.(*Server).handleStream() > 2023-09-13T16:30:48.211703838Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() > 2023-09-13T16:30:48.211716639Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.234837623Z rpc error: code = Unknown desc = mongo: no documents in result > 2023-09-13T16:30:48.234851094Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.234855574Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() > 2023-09-13T16:30:48.234859325Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() > 2023-09-13T16:30:48.234862935Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() > 2023-09-13T16:30:48.234866855Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:28 github.com/cenkalti/backoff/v4.Retry() > 2023-09-13T16:30:48.234870885Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() > 2023-09-13T16:30:48.234874686Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() > 2023-09-13T16:30:48.234911768Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.238151413Z grpc error: stream not found > 2023-09-13T16:30:48.238162364Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.238166834Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() > 2023-09-13T16:30:48.238170654Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() > 2023-09-13T16:30:48.238174324Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() > 2023-09-13T16:30:48.238177514Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() > 2023-09-13T16:30:48.238180725Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.238190115Z 2023/09/14 00:30:48 error cannot subscribe worker node[dbfb8369-3c07-11ee-be05-0242ac130002]: grpc error: stream not found > 2023-09-13T16:30:48.239289515Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.239197668 +0800 CST entry 2 next 2023-09-15 09:00:00 +0800 CST > 2023-09-13T16:30:48.246282942Z 2023/09/14 00:30:48 info [MessageServer] received connect request from node[4abd561c-3138-11ee-a3dd-0242ac120002], key: plugin:dependency:4abd561c-3138-11ee-a3dd-0242ac120002 > 2023-09-13T16:30:48.258706171Z grpc error: stream not found > 2023-09-13T16:30:48.258717372Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.258721532Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() > 2023-09-13T16:30:48.258724873Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() > 2023-09-13T16:30:48.258728193Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() > 2023-09-13T16:30:48.258731293Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() > 2023-09-13T16:30:48.258734453Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.258823160Z grpc error: stream not found > 2023-09-13T16:30:48.258831140Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.258835210Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() > 2023-09-13T16:30:48.258839331Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() > 2023-09-13T16:30:48.258843071Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() > 2023-09-13T16:30:48.258846921Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() > 2023-09-13T16:30:48.258850722Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.258869503Z 2023/09/14 00:30:48 error cannot subscribe worker node[0f85d06d-3c08-11ee-8713-0242ac130002]: grpc error: stream not found > 2023-09-13T16:30:48.259326676Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.259267142 +0800 CST entry 3 next 2023-09-16 09:00:00 +0800 CST > 2023-09-13T16:30:48.261264696Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.261208792 +0800 CST entry 4 next 2023-09-16 09:00:00 +0800 CST > 2023-09-13T16:30:48.261341982Z grpc error: stream not found > 2023-09-13T16:30:48.261349122Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.261353263Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() > 2023-09-13T16:30:48.261357133Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() > 2023-09-13T16:30:48.261368774Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() > 2023-09-13T16:30:48.261373334Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() > 2023-09-13T16:30:48.261377284Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.261427718Z grpc error: stream not found > 2023-09-13T16:30:48.261434058Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.261437919Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() > 2023-09-13T16:30:48.261441779Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() > 2023-09-13T16:30:48.261446029Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() > 2023-09-13T16:30:48.261449739Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() > 2023-09-13T16:30:48.261453690Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.261618682Z 2023/09/14 00:30:48 error cannot subscribe worker node[f1c6d46d-3cd5-11ee-9861-0242ac130002]: grpc error: stream not found > 2023-09-13T16:30:48.263339686Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.263284292 +0800 CST entry 5 next 2023-09-16 09:00:00 +0800 CST > 2023-09-13T16:30:48.269075362Z grpc error: stream not found > 2023-09-13T16:30:48.269085343Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.269089203Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() > 2023-09-13T16:30:48.269092663Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() > 2023-09-13T16:30:48.269095944Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() > 2023-09-13T16:30:48.269099204Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() > 2023-09-13T16:30:48.269109774Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.269186280Z grpc error: stream not found > 2023-09-13T16:30:48.269191990Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.269195231Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() > 2023-09-13T16:30:48.269198281Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() > 2023-09-13T16:30:48.269201561Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() > 2023-09-13T16:30:48.269204611Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() > 2023-09-13T16:30:48.269207661Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.269246914Z 2023/09/14 00:30:48 error cannot subscribe worker node[9cef89e1-508b-11ee-a134-0242ac130002]: grpc error: stream not found > 2023-09-13T16:30:48.270702010Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.270644596 +0800 CST entry 6 next 2023-09-16 09:00:00 +0800 CST > 2023-09-13T16:30:48.271406231Z grpc error: stream not found > 2023-09-13T16:30:48.271416941Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.271421572Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/server.go:133 github.com/crawlab-team/crawlab-core/grpc/server.(*Server).GetSubscribe() > 2023-09-13T16:30:48.271426312Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:272 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).subscribeNode() > 2023-09-13T16:30:48.271430152Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:217 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() > 2023-09-13T16:30:48.271433533Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() > 2023-09-13T16:30:48.271436803Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.271458524Z node error: monitor error > 2023-09-13T16:30:48.271463805Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.271467425Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:236 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() > 2023-09-13T16:30:48.271471445Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() > 2023-09-13T16:30:48.271483646Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.271616576Z node error: monitor error > 2023-09-13T16:30:48.271624767Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.271627927Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:236 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).monitor() > 2023-09-13T16:30:48.271631227Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/node/service/master_service.go:110 github.com/crawlab-team/crawlab-core/node/service.(*MasterService).Monitor() > 2023-09-13T16:30:48.271634317Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.272588666Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.272536603 +0800 CST entry 7 next 2023-09-16 09:00:00 +0800 CST > 2023-09-13T16:30:48.274388327Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.274329453 +0800 CST entry 8 next 2023-09-16 09:00:00 +0800 CST > 2023-09-13T16:30:48.276172956Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.276112632 +0800 CST entry 9 next 2023-09-16 09:00:00 +0800 CST > 2023-09-13T16:30:48.277968866Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.277911702 +0800 CST entry 10 next 2023-09-16 15:00:00 +0800 CST > 2023-09-13T16:30:48.279803769Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.279744484 +0800 CST entry 11 next 2023-09-16 15:00:00 +0800 CST > 2023-09-13T16:30:48.281613940Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.281555846 +0800 CST entry 12 next 2023-09-16 15:00:00 +0800 CST > 2023-09-13T16:30:48.283546860Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.283487825 +0800 CST entry 13 next 2023-09-16 15:00:00 +0800 CST > 2023-09-13T16:30:48.285316878Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.285259664 +0800 CST entry 14 next 2023-09-16 15:00:00 +0800 CST > 2023-09-13T16:30:48.287169462Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.287080446 +0800 CST entry 15 next 2023-09-16 15:00:00 +0800 CST > 2023-09-13T16:30:48.289052438Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.288971663 +0800 CST entry 16 next 2023-09-16 15:00:00 +0800 CST > 2023-09-13T16:30:48.290922164Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.29086329 +0800 CST entry 17 next 2023-09-16 15:00:00 +0800 CST > 2023-09-13T16:30:48.292724124Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.29266683 +0800 CST entry 18 next 2023-09-17 09:00:00 +0800 CST > 2023-09-13T16:30:48.294465891Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.294409167 +0800 CST entry 19 next 2023-09-17 09:00:00 +0800 CST > 2023-09-13T16:30:48.296359038Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.296303924 +0800 CST entry 20 next 2023-09-17 09:00:00 +0800 CST > 2023-09-13T16:30:48.298385725Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.29832902 +0800 CST entry 21 next 2023-09-17 09:00:00 +0800 CST > 2023-09-13T16:30:48.300268811Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.300216007 +0800 CST entry 22 next 2023-09-17 09:00:00 +0800 CST > 2023-09-13T16:30:48.302334860Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.302275956 +0800 CST entry 23 next 2023-09-17 09:00:00 +0800 CST > 2023-09-13T16:30:48.304317414Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.304257159 +0800 CST entry 24 next 2023-09-17 09:00:00 +0800 CST > 2023-09-13T16:30:48.306258495Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.30620042 +0800 CST entry 25 next 2023-09-17 09:00:00 +0800 CST > 2023-09-13T16:30:48.311538557Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.311486213 +0800 CST entry 26 next 2023-09-17 15:00:00 +0800 CST > 2023-09-13T16:30:48.316619995Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.31655729 +0800 CST entry 27 next 2023-09-17 15:00:00 +0800 CST > 2023-09-13T16:30:48.319157229Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.319054161 +0800 CST entry 28 next 2023-09-17 15:00:00 +0800 CST > 2023-09-13T16:30:48.323228483Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.323135586 +0800 CST entry 29 next 2023-09-19 09:00:00 +0800 CST > 2023-09-13T16:30:48.326576826Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.326512721 +0800 CST entry 30 next 2023-09-15 16:00:00 +0800 CST > 2023-09-13T16:30:48.336786435Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.3367168 +0800 CST entry 31 next 2023-09-19 10:00:00 +0800 CST > 2023-09-13T16:30:48.338617938Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.338559374 +0800 CST entry 32 next 2023-09-19 10:00:00 +0800 CST > 2023-09-13T16:30:48.340433219Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.340369525 +0800 CST entry 33 next 2023-09-19 10:00:00 +0800 CST > 2023-09-13T16:30:48.342137293Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.342081329 +0800 CST entry 34 next 2023-09-19 10:00:00 +0800 CST > 2023-09-13T16:30:48.343908811Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.343845176 +0800 CST entry 35 next 2023-09-19 10:00:00 +0800 CST > 2023-09-13T16:30:48.345669669Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.345602994 +0800 CST entry 36 next 2023-10-02 09:00:00 +0800 CST > 2023-09-13T16:30:48.347481690Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.347416715 +0800 CST entry 37 next 2023-10-03 09:00:00 +0800 CST > 2023-09-13T16:30:48.349201034Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.34913614 +0800 CST entry 38 next 2023-10-02 16:00:00 +0800 CST > 2023-09-13T16:30:48.350922719Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.350867145 +0800 CST entry 39 next 2023-09-19 13:00:00 +0800 CST > 2023-09-13T16:30:48.352625532Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.352568008 +0800 CST entry 40 next 2023-09-19 13:00:00 +0800 CST > 2023-09-13T16:30:48.354303654Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.354246139 +0800 CST entry 41 next 2023-09-19 13:00:00 +0800 CST > 2023-09-13T16:30:48.355980365Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.355922581 +0800 CST entry 42 next 2023-09-19 13:00:00 +0800 CST > 2023-09-13T16:30:48.364167488Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.364100293 +0800 CST entry 43 next 2023-09-19 13:00:00 +0800 CST > 2023-09-13T16:30:48.365823988Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.365763603 +0800 CST entry 44 next 2023-09-19 16:00:00 +0800 CST > 2023-09-13T16:30:48.367579845Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.36751782 +0800 CST entry 45 next 2023-09-19 16:00:00 +0800 CST > 2023-09-13T16:30:48.369297279Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.369237985 +0800 CST entry 46 next 2023-09-19 16:00:00 +0800 CST > 2023-09-13T16:30:48.370967450Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.370910826 +0800 CST entry 47 next 2023-09-18 09:00:00 +0800 CST > 2023-09-13T16:30:48.372716277Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.372650622 +0800 CST entry 48 next 2023-10-05 09:00:00 +0800 CST > 2023-09-13T16:30:48.374416750Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.374349725 +0800 CST entry 49 next 2023-10-06 19:00:00 +0800 CST > 2023-09-13T16:30:48.376166837Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.376099792 +0800 CST entry 50 next 2023-10-06 19:00:00 +0800 CST > 2023-09-13T16:30:48.377973608Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.377914203 +0800 CST entry 51 next 2023-10-06 19:00:00 +0800 CST > 2023-09-13T16:30:48.379894037Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.379817621 +0800 CST entry 52 next 2023-10-06 19:00:00 +0800 CST > 2023-09-13T16:30:48.381571658Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.381505673 +0800 CST entry 53 next 2023-10-06 19:00:00 +0800 CST > 2023-09-13T16:30:48.383283142Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.383218447 +0800 CST entry 54 next 2023-10-07 03:00:00 +0800 CST > 2023-09-13T16:30:48.385002346Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.384939622 +0800 CST entry 55 next 2023-10-07 03:00:00 +0800 CST > 2023-09-13T16:30:48.386824198Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.386765214 +0800 CST entry 56 next 2023-10-07 03:00:00 +0800 CST > 2023-09-13T16:30:48.388849745Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.388791781 +0800 CST entry 57 next 2023-10-07 03:00:00 +0800 CST > 2023-09-13T16:30:48.390883252Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.390821998 +0800 CST entry 58 next 2023-10-07 03:00:00 +0800 CST > 2023-09-13T16:30:48.392864636Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.392806252 +0800 CST entry 59 next 2023-10-07 10:00:00 +0800 CST > 2023-09-13T16:30:48.394832358Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.394774094 +0800 CST entry 60 next 2023-10-07 10:00:00 +0800 CST > 2023-09-13T16:30:48.396829313Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.396769179 +0800 CST entry 61 next 2023-10-07 10:00:00 +0800 CST > 2023-09-13T16:30:48.400890897Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.40079376 +0800 CST entry 62 next 2023-10-05 16:00:00 +0800 CST > 2023-09-13T16:30:48.407137769Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.407066024 +0800 CST entry 63 next 2023-09-17 00:00:00 +0800 CST > 2023-09-13T16:30:48.409093171Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.409034047 +0800 CST entry 64 next 2023-09-17 00:10:00 +0800 CST > 2023-09-13T16:30:48.410776713Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.410716749 +0800 CST entry 65 next 2023-09-17 00:20:00 +0800 CST > 2023-09-13T16:30:48.412455405Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.41239779 +0800 CST entry 66 next 2023-09-19 03:00:00 +0800 CST > 2023-09-13T16:30:48.414206571Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.414149237 +0800 CST entry 67 next 2023-09-19 03:10:00 +0800 CST > 2023-09-13T16:30:48.416003121Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.415944257 +0800 CST entry 68 next 2023-09-19 03:20:00 +0800 CST > 2023-09-13T16:30:48.417769439Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.417710335 +0800 CST entry 69 next 2023-09-19 03:30:00 +0800 CST > 2023-09-13T16:30:48.419579280Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.419524866 +0800 CST entry 70 next 2023-09-19 03:40:00 +0800 CST > 2023-09-13T16:30:48.421472447Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.421418394 +0800 CST entry 71 next 2023-09-19 03:50:00 +0800 CST > 2023-09-13T16:30:48.423543088Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.423492984 +0800 CST entry 72 next 2023-09-19 04:00:00 +0800 CST > 2023-09-13T16:30:48.425464476Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.425415463 +0800 CST entry 73 next 2023-09-19 04:10:00 +0800 CST > 2023-09-13T16:30:48.427472282Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.427421058 +0800 CST entry 74 next 2023-09-19 04:20:00 +0800 CST > 2023-09-13T16:30:48.429523250Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.429471677 +0800 CST entry 75 next 2023-09-19 04:30:00 +0800 CST > 2023-09-13T16:30:48.431584180Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.431533386 +0800 CST entry 76 next 2023-09-20 12:00:00 +0800 CST > 2023-09-13T16:30:48.433670461Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.433611617 +0800 CST entry 77 next 2023-10-02 08:00:00 +0800 CST > 2023-09-13T16:30:48.435718919Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.435661165 +0800 CST entry 78 next 2023-10-05 08:00:00 +0800 CST > 2023-09-13T16:30:48.437821861Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.437761857 +0800 CST entry 79 next 2023-10-06 20:00:00 +0800 CST > 2023-09-13T16:30:48.439831377Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.439773043 +0800 CST entry 80 next 2023-10-06 20:00:00 +0800 CST > 2023-09-13T16:30:48.441888976Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.441830612 +0800 CST entry 81 next 2023-10-06 20:00:00 +0800 CST > 2023-09-13T16:30:48.456727700Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.456643184 +0800 CST entry 82 next 2023-10-06 20:00:00 +0800 CST > 2023-09-13T16:30:48.459072970Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.458999365 +0800 CST entry 83 next 2023-10-06 20:00:00 +0800 CST > 2023-09-13T16:30:48.460879591Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.460804575 +0800 CST entry 84 next 2023-10-06 20:00:00 +0800 CST > 2023-09-13T16:30:48.462746306Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.462677541 +0800 CST entry 85 next 2023-10-07 04:00:00 +0800 CST > 2023-09-13T16:30:48.464530156Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.46446041 +0800 CST entry 86 next 2023-10-07 04:00:00 +0800 CST > 2023-09-13T16:30:48.466201666Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.466135572 +0800 CST entry 87 next 2023-10-07 04:00:00 +0800 CST > 2023-09-13T16:30:48.467935222Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.467855396 +0800 CST entry 88 next 2023-10-07 04:00:00 +0800 CST > 2023-09-13T16:30:48.469661507Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.469591202 +0800 CST entry 89 next 2023-10-07 04:00:00 +0800 CST > 2023-09-13T16:30:48.471407703Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.471335958 +0800 CST entry 90 next 2023-10-07 04:00:00 +0800 CST > 2023-09-13T16:30:48.473188602Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.473130798 +0800 CST entry 91 next 2023-09-17 00:10:00 +0800 CST > 2023-09-13T16:30:48.474946260Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.474889495 +0800 CST entry 92 next 2023-09-18 00:10:00 +0800 CST > 2023-09-13T16:30:48.476723078Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.476662274 +0800 CST entry 93 next 2023-09-18 00:40:00 +0800 CST > 2023-09-13T16:30:48.478456494Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.47840208 +0800 CST entry 94 next 2023-09-15 08:00:00 +0800 CST > 2023-09-13T16:30:48.480227672Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.480159697 +0800 CST entry 95 next 2023-10-12 03:00:00 +0800 CST > 2023-09-13T16:30:48.481984820Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.481917195 +0800 CST entry 96 next 2023-10-12 03:00:00 +0800 CST > 2023-09-13T16:30:48.483730656Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.483662961 +0800 CST entry 97 next 2023-10-12 03:00:00 +0800 CST > 2023-09-13T16:30:48.485460671Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.485394366 +0800 CST entry 98 next 2023-10-12 03:00:00 +0800 CST > 2023-09-13T16:30:48.487238190Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.487170855 +0800 CST entry 99 next 2023-10-12 06:00:00 +0800 CST > 2023-09-13T16:30:48.489009338Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.488941773 +0800 CST entry 100 next 2023-10-12 06:00:00 +0800 CST > 2023-09-13T16:30:48.490747003Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.490680199 +0800 CST entry 101 next 2023-10-12 06:00:00 +0800 CST > 2023-09-13T16:30:48.492567766Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.4924972 +0800 CST entry 102 next 2023-10-12 06:00:00 +0800 CST > 2023-09-13T16:30:48.494323243Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.494270439 +0800 CST entry 103 next 2023-09-14 01:00:00 +0800 CST > 2023-09-13T16:30:48.496157876Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.496103572 +0800 CST entry 104 next 2023-09-14 01:00:00 +0800 CST > 2023-09-13T16:30:48.498083635Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.49802129 +0800 CST entry 105 next 2023-09-14 01:00:00 +0800 CST > 2023-09-13T16:30:48.500041877Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.499988243 +0800 CST entry 106 next 2023-09-14 04:00:00 +0800 CST > 2023-09-13T16:30:48.501919772Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.501855618 +0800 CST entry 107 next 2023-09-14 04:00:00 +0800 CST > 2023-09-13T16:30:48.507186574Z 2023/09/14 00:30:48 info [MessageServer] received connect request from node[9cef89e1-508b-11ee-a134-0242ac130002], key: plugin:dependency:9cef89e1-508b-11ee-a134-0242ac130002 > 2023-09-13T16:30:48.507313743Z 2023/09/14 00:30:48 info [MessageServer] received connect request from node[9cef89e1-508b-11ee-a134-0242ac130002], key: metric:9cef89e1-508b-11ee-a134-0242ac130002 > 2023-09-13T16:30:48.507406250Z 2023/09/14 00:30:48 info [NodeServer] master received subscribe request from node[9cef89e1-508b-11ee-a134-0242ac130002] > 2023-09-13T16:30:48.507447313Z 2023/09/14 00:30:48 info [NodeServer] master subscribed node[9cef89e1-508b-11ee-a134-0242ac130002] > 2023-09-13T16:30:48.507718253Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.507673829 +0800 CST entry 108 next 2023-09-14 04:00:00 +0800 CST > 2023-09-13T16:30:48.511682350Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.511632456 +0800 CST entry 109 next 2023-09-14 07:00:00 +0800 CST > 2023-09-13T16:30:48.513652393Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.513598349 +0800 CST entry 110 next 2023-09-14 07:00:00 +0800 CST > 2023-09-13T16:30:48.515647877Z 2023/09/14 00:30:48 info cron: added now 2023-09-14 00:30:48.515601054 +0800 CST entry 111 next 2023-09-14 07:00:00 +0800 CST > 2023-09-13T16:30:48.579386273Z ts=2023-09-13T16:30:48.579Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" > 2023-09-13T16:30:48.580266966Z ts=2023-09-13T16:30:48.580Z caller=head.go:536 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=888.794µs > 2023-09-13T16:30:48.580279887Z ts=2023-09-13T16:30:48.580Z caller=head.go:542 level=info component=tsdb msg="Replaying WAL, this may take a while" > 2023-09-13T16:30:48.704699707Z 2023/09/14 00:30:48 info subscribe events > 2023-09-13T16:30:48.705826868Z mongo: no documents in result > 2023-09-13T16:30:48.705837899Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.705842569Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() > 2023-09-13T16:30:48.705846360Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() > 2023-09-13T16:30:48.705849610Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:48.705852780Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() > 2023-09-13T16:30:48.705868561Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:48.705872401Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() > 2023-09-13T16:30:48.705875842Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() > 2023-09-13T16:30:48.705879312Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() > 2023-09-13T16:30:48.705882922Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() > 2023-09-13T16:30:48.705886202Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1616 google.golang.org/grpc.(*Server).handleStream() > 2023-09-13T16:30:48.705892433Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() > 2023-09-13T16:30:48.705896743Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:48.706077136Z rpc error: code = Unknown desc = mongo: no documents in result > 2023-09-13T16:30:48.706085917Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:48.706089757Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() > 2023-09-13T16:30:48.706093248Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() > 2023-09-13T16:30:48.706096398Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() > 2023-09-13T16:30:48.706099728Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:28 github.com/cenkalti/backoff/v4.Retry() > 2023-09-13T16:30:48.706103348Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() > 2023-09-13T16:30:48.706107078Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() > 2023-09-13T16:30:48.706111319Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:49.028973368Z 2023/09/14 00:30:49 info [NotificationService] obtained grpc stream, start receiving messages... > 2023-09-13T16:30:49.101417084Z 2023/09/14 00:30:49 info [NodeServer] master received subscribe request from node[f1c6d46d-3cd5-11ee-9861-0242ac130002] > 2023-09-13T16:30:49.101586436Z 2023/09/14 00:30:49 info [NodeServer] master subscribed node[f1c6d46d-3cd5-11ee-9861-0242ac130002] > 2023-09-13T16:30:49.151207909Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[0f85d06d-3c08-11ee-8713-0242ac130002], key: metric:0f85d06d-3c08-11ee-8713-0242ac130002 > 2023-09-13T16:30:49.207353705Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[f1c6d46d-3cd5-11ee-9861-0242ac130002], key: metric:f1c6d46d-3cd5-11ee-9861-0242ac130002 > 2023-09-13T16:30:49.220476805Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[f1c6d46d-3cd5-11ee-9861-0242ac130002], key: plugin:dependency:f1c6d46d-3cd5-11ee-9861-0242ac130002 > 2023-09-13T16:30:49.246155205Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[dbfb8369-3c07-11ee-be05-0242ac130002], key: plugin:dependency:dbfb8369-3c07-11ee-be05-0242ac130002 > 2023-09-13T16:30:49.251243753Z 2023/09/14 00:30:49 info [NodeServer] master received subscribe request from node[f1c6d46d-3cd5-11ee-9861-0242ac130002] > 2023-09-13T16:30:49.251327519Z 2023/09/14 00:30:49 info [NodeServer] master subscribed node[f1c6d46d-3cd5-11ee-9861-0242ac130002] > 2023-09-13T16:30:49.400263954Z 2023/09/14 00:30:49 info subscribe events > 2023-09-13T16:30:49.401600281Z mongo: no documents in result > 2023-09-13T16:30:49.401615312Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:49.401619832Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() > 2023-09-13T16:30:49.401623842Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() > 2023-09-13T16:30:49.401628293Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:49.401632913Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() > 2023-09-13T16:30:49.401637594Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:49.401641704Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() > 2023-09-13T16:30:49.401645764Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() > 2023-09-13T16:30:49.401649744Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() > 2023-09-13T16:30:49.401654095Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() > 2023-09-13T16:30:49.401657995Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1616 google.golang.org/grpc.(*Server).handleStream() > 2023-09-13T16:30:49.401674066Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() > 2023-09-13T16:30:49.401678337Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:49.401811316Z rpc error: code = Unknown desc = mongo: no documents in result > 2023-09-13T16:30:49.401820267Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:49.401823857Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() > 2023-09-13T16:30:49.401827527Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() > 2023-09-13T16:30:49.401830588Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() > 2023-09-13T16:30:49.401833648Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:28 github.com/cenkalti/backoff/v4.Retry() > 2023-09-13T16:30:49.401837208Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() > 2023-09-13T16:30:49.401840928Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() > 2023-09-13T16:30:49.401844629Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:49.425924592Z 2023/09/14 00:30:49 info [NodeServer] master received subscribe request from node[dbfb8369-3c07-11ee-be05-0242ac130002] > 2023-09-13T16:30:49.425945734Z 2023/09/14 00:30:49 info [NodeServer] master subscribed node[dbfb8369-3c07-11ee-be05-0242ac130002] > 2023-09-13T16:30:49.536712495Z 2023/09/14 00:30:49 info [NodeServer] master received subscribe request from node[0f85d06d-3c08-11ee-8713-0242ac130002] > 2023-09-13T16:30:49.536843734Z 2023/09/14 00:30:49 info [NodeServer] master subscribed node[0f85d06d-3c08-11ee-8713-0242ac130002] > 2023-09-13T16:30:49.538279638Z 2023/09/14 00:30:49 info license activated > 2023-09-13T16:30:49.550948655Z 2023/09/14 00:30:49 info [NodeServer] master received subscribe request from node[f1c6d46d-3cd5-11ee-9861-0242ac130002] > 2023-09-13T16:30:49.551122128Z 2023/09/14 00:30:49 info [NodeServer] master subscribed node[f1c6d46d-3cd5-11ee-9861-0242ac130002] > 2023-09-13T16:30:49.685311155Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[dbfb8369-3c07-11ee-be05-0242ac130002], key: metric:dbfb8369-3c07-11ee-be05-0242ac130002 > 2023-09-13T16:30:49.934377751Z 2023/09/14 00:30:49 info [MessageServer] received connect request from node[0f85d06d-3c08-11ee-8713-0242ac130002], key: plugin:dependency:0f85d06d-3c08-11ee-8713-0242ac130002 > 2023-09-13T16:30:50.643634660Z ts=2023-09-13T16:30:50.643Z caller=head.go:578 level=info component=tsdb msg="WAL checkpoint loaded" > 2023-09-13T16:30:50.703857920Z ts=2023-09-13T16:30:50.703Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=619 maxSegment=622 > 2023-09-13T16:30:50.737192114Z 2023/09/14 00:30:50 info subscribe events > 2023-09-13T16:30:50.738325386Z mongo: no documents in result > 2023-09-13T16:30:50.738336077Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:50.738340797Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() > 2023-09-13T16:30:50.738344898Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() > 2023-09-13T16:30:50.738348268Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:50.738351908Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() > 2023-09-13T16:30:50.738355958Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:50.738380070Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() > 2023-09-13T16:30:50.738384400Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() > 2023-09-13T16:30:50.738388481Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() > 2023-09-13T16:30:50.738392661Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() > 2023-09-13T16:30:50.738396152Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1616 google.golang.org/grpc.(*Server).handleStream() > 2023-09-13T16:30:50.738399532Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() > 2023-09-13T16:30:50.738403222Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:50.738624378Z rpc error: code = Unknown desc = mongo: no documents in result > 2023-09-13T16:30:50.738632829Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:50.738636329Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() > 2023-09-13T16:30:50.738640019Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() > 2023-09-13T16:30:50.738643529Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() > 2023-09-13T16:30:50.738659060Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:28 github.com/cenkalti/backoff/v4.Retry() > 2023-09-13T16:30:50.738662311Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() > 2023-09-13T16:30:50.738679422Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() > 2023-09-13T16:30:50.738682772Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:50.788570685Z ts=2023-09-13T16:30:50.788Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=620 maxSegment=622 > 2023-09-13T16:30:50.938028827Z ts=2023-09-13T16:30:50.937Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=621 maxSegment=622 > 2023-09-13T16:30:50.938108033Z ts=2023-09-13T16:30:50.938Z caller=head.go:613 level=info component=tsdb msg="WAL segment loaded" segment=622 maxSegment=622 > 2023-09-13T16:30:50.938167167Z ts=2023-09-13T16:30:50.938Z caller=head.go:619 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=2.06331245s wal_replay_duration=294.550479ms total_replay_duration=2.358784145s > 2023-09-13T16:30:51.560316738Z ts=2023-09-13T16:30:51.560Z caller=main.go:993 level=info fs_type=EXT4_SUPER_MAGIC > 2023-09-13T16:30:51.560362692Z ts=2023-09-13T16:30:51.560Z caller=main.go:996 level=info msg="TSDB started" > 2023-09-13T16:30:51.560367342Z ts=2023-09-13T16:30:51.560Z caller=main.go:1177 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml > 2023-09-13T16:30:51.560761441Z ts=2023-09-13T16:30:51.560Z caller=main.go:1214 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=453.832µs db_storage=1.431µs remote_storage=1.59µs web_handler=490ns query_engine=1µs scrape=141.42µs scrape_sd=27.352µs notify=1.07µs notify_sd=4.001µs rules=1.75µs tracing=7.3µs > 2023-09-13T16:30:51.560796573Z ts=2023-09-13T16:30:51.560Z caller=main.go:957 level=info msg="Server is ready to receive web requests." > 2023-09-13T16:30:51.560802744Z ts=2023-09-13T16:30:51.560Z caller=manager.go:941 level=info component="rule manager" msg="Starting rule manager..." > 2023-09-13T16:30:51.846940566Z 2023/09/14 00:30:51 info subscribe events > 2023-09-13T16:30:51.848286533Z mongo: no documents in result > 2023-09-13T16:30:51.848322036Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:51.848326926Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() > 2023-09-13T16:30:51.848330616Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() > 2023-09-13T16:30:51.848334257Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:51.848352808Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() > 2023-09-13T16:30:51.848356888Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:51.848360889Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() > 2023-09-13T16:30:51.848364599Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() > 2023-09-13T16:30:51.848368219Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() > 2023-09-13T16:30:51.848372230Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() > 2023-09-13T16:30:51.848376090Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1616 google.golang.org/grpc.(*Server).handleStream() > 2023-09-13T16:30:51.848406982Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() > 2023-09-13T16:30:51.848415913Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:51.848417863Z rpc error: code = Unknown desc = mongo: no documents in result > 2023-09-13T16:30:51.848419623Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:51.848421423Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() > 2023-09-13T16:30:51.848423323Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() > 2023-09-13T16:30:51.848425253Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() > 2023-09-13T16:30:51.848427043Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:28 github.com/cenkalti/backoff/v4.Retry() > 2023-09-13T16:30:51.848428843Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() > 2023-09-13T16:30:51.848430754Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() > 2023-09-13T16:30:51.848432674Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:53.360018842Z 2023/09/14 00:30:53 info subscribe events > 2023-09-13T16:30:53.360941059Z mongo: no documents in result > 2023-09-13T16:30:53.360968881Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:53.360973191Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() > 2023-09-13T16:30:53.360976742Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() > 2023-09-13T16:30:53.360980152Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:53.360984012Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() > 2023-09-13T16:30:53.361005914Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:53.361010804Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() > 2023-09-13T16:30:53.361014174Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() > 2023-09-13T16:30:53.361017435Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() > 2023-09-13T16:30:53.361020665Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() > 2023-09-13T16:30:53.361024165Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1616 google.golang.org/grpc.(*Server).handleStream() > 2023-09-13T16:30:53.361027895Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() > 2023-09-13T16:30:53.361032496Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:53.361057978Z rpc error: code = Unknown desc = mongo: no documents in result > 2023-09-13T16:30:53.361063258Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:53.361066838Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() > 2023-09-13T16:30:53.361070429Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() > 2023-09-13T16:30:53.361090340Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() > 2023-09-13T16:30:53.361094580Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:28 github.com/cenkalti/backoff/v4.Retry() > 2023-09-13T16:30:53.361103681Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() > 2023-09-13T16:30:53.361105731Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() > 2023-09-13T16:30:53.361107601Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:56.402736142Z 2023/09/14 00:30:56 info subscribe events > 2023-09-13T16:30:56.403781127Z mongo: no documents in result > 2023-09-13T16:30:56.403815970Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:56.403820720Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/grpc/server/plugin_server.go:50 github.com/crawlab-team/crawlab-core/grpc/server.PluginServer.Register() > 2023-09-13T16:30:56.403824681Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:159 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler.func1() > 2023-09-13T16:30:56.403828341Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:31 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:56.403831711Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/auth/auth.go:46 github.com/grpc-ecosystem/go-grpc-middleware/auth.UnaryServerInterceptor.func1() > 2023-09-13T16:30:56.403834881Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:34 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1() > 2023-09-13T16:30:56.403838131Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/recovery/interceptors.go:29 github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1() > 2023-09-13T16:30:56.403841402Z /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:39 github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1() > 2023-09-13T16:30:56.403844502Z /go/pkg/mod/github.com/crawlab-team/crawlab-grpc@v0.6.0-beta.20211219.1930.0.20221020032435-afa1c691f73c/plugin_service_grpc.pb.go:161 github.com/crawlab-team/crawlab-grpc._PluginService_Register_Handler() > 2023-09-13T16:30:56.403848352Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1282 google.golang.org/grpc.(*Server).processUnaryRPC() > 2023-09-13T16:30:56.403851932Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:1616 google.golang.org/grpc.(*Server).handleStream() > 2023-09-13T16:30:56.403854863Z /go/pkg/mod/google.golang.org/grpc@v1.42.0/server.go:921 google.golang.org/grpc.(*Server).serveStreams.func1.2() > 2023-09-13T16:30:56.403859013Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit() > 2023-09-13T16:30:56.403920337Z rpc error: code = Unknown desc = mongo: no documents in result > 2023-09-13T16:30:56.403927018Z /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:13 github.com/crawlab-team/go-trace.TraceError() > 2023-09-13T16:30:56.403948689Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:132 github.com/crawlab-team/crawlab-core/notification.(*Service)._subscribe() > 2023-09-13T16:30:56.403953040Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() > 2023-09-13T16:30:56.403957000Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() > 2023-09-13T16:30:56.403960960Z /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.3/retry.go:28 github.com/cenkalti/backoff/v4.Retry() > 2023-09-13T16:30:56.403964531Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:105 github.com/crawlab-team/crawlab-core/notification.(*Service).subscribe() > 2023-09-13T16:30:56.403968441Z /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.3-0.20230804030437-664646cf1665/notification/service.go:64 github.com/crawlab-team/crawlab-core/notification.(*Service).Start() > 2023-09-13T16:30:56.403972001Z /usr/local/go/src/runtime/asm_amd64.s:1571 runtime.goexit()```
mansoorelahi commented 1 year ago

@tikazyq It is the same error on the pro version, and it happens frequently

Voyewodayer commented 10 months ago

I probably managed to pinpoint that it tends to happen while API call to /metrics endpoint is being made. Shortly after it tends to crash the container and rebuild it again with errors akin to the logs provided in my previous comment.

oleksa-dev commented 10 months ago

@tikazyq same issue on pro version

oleksa-dev commented 7 months ago

@tikazyq hi, any progress on this task?

tikazyq commented 7 months ago

You can switch to the latest version of docker image to resolve this issue.

oleksa-dev commented 7 months ago

@tikazyq hey, we've updated images to crawlab-pro:latest but still we have this abnormal issue. I've noticed that the last modifications on crawlab-pro:latest was made 5 month ago and on crawlab-pro:develop 4 month ago. Can you please verify that fix of abnormal issue is on crawlab-pro:latest and not on crawlab-pro:develop?

Screen Shot 2024-04-11 at 16 31 07
KrystianJanas commented 6 months ago

@tikazyq hi again, do you have any update to the above message?

anaghaKruko commented 5 months ago

@tikazyq Hi , could you tell how to disable the metrics flag

pratipal-dev commented 5 months ago

@tikazyq Hi , could you tell how to disable the metrics flag

We have observed that there is some kind of co-relation between the metrics and the abnormal status. As long as the metrics are off there is no issue, but some how it gets turned on automatically and all of a sudden the abnormal status starts popping up.

It would be really helpful to know of a way to disable it.

KrystianJanas commented 2 months ago

@tikazyq please answer us. The same problem with the same thing.