crawlab-team / crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
https://www.crawlab.cn
BSD 3-Clause "New" or "Revised" License
11.38k stars 1.8k forks source link

调度不成功,更新docker images后调度任务,一直是待定状态。 #1253

Closed liziqiang0523 closed 1 year ago

liziqiang0523 commented 1 year ago

Bug 描述 每次更新docker镜像后,原来的爬虫就不能正常调度了。一直是待定状态。 复现步骤 该 Bug 复现步骤如下

  1. 更新docker 镜像。重启crawlab服务。
  2. 重启crawlab服务。
  3. 调度以前的爬虫项目。

期望结果 任务调度恢复正常。不用再删除爬虫重新添加。

截屏 截屏1

日志 crawlab_master | /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.0/retry.go:28 github.com/cenkalti/backoff/v4.Retry() crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20221218100256-a28d12756f73/seaweedfs_manager.go:308 github.com/crawlab-team/crawlab-fs.(SeaweedFsManager).start.func1() crawlab_master | /usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit() crawlab_master | [GIN] 2023/01/12 - 10:15:15 | 200 | 10.277706ms | 127.0.0.1 | GET "/tasks?page=1&size=10&conditions=[]&sort=[]&stats=true" crawlab_master | [GIN] 2023/01/12 - 10:15:16 | 500 | 2.378937ms | 127.0.0.1 | GET "/filer/fs/63b525ec2428a9c16aeab08e/.gitignore" crawlab_master | unexpected EOF crawlab_master | /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.1/trace.go:6 github.com/crawlab-team/go-trace.PrintError() crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20221218100256-a28d12756f73/seaweedfs_manager.go:335 github.com/crawlab-team/crawlab-fs.(SeaweedFsManager).error() crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20221218100256-a28d12756f73/seaweedfs_manager.go:449 github.com/crawlab-team/crawlab-fs.(SeaweedFsManager).downloadFile() crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20221218100256-a28d12756f73/seaweedfs_manager.go:309 github.com/crawlab-team/crawlab-fs.(SeaweedFsManager).start.func1.1() crawlab_master | /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.0/retry.go:55 github.com/cenkalti/backoff/v4.RetryNotifyWithTimer() crawlab_master | /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.0/retry.go:34 github.com/cenkalti/backoff/v4.RetryNotify() crawlab_master | /go/pkg/mod/github.com/cenkalti/backoff/v4@v4.1.0/retry.go:28 github.com/cenkalti/backoff/v4.Retry() crawlab_master | /go/pkg/mod/github.com/crawlab-team/crawlab-fs@v0.6.0-beta.20211101.1940.0.20221218100256-a28d12756f73/seaweedfs_manager.go:308 github.com/crawlab-team/crawlab-fs.(*SeaweedFsManager).start.func1() crawlab_master | /usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit() crawlab_master | [GIN] 2023/01/12 - 10:15:16 | 500 | 2.059957ms | 127.0.0.1 | GET "/filer/fs/63b525ec2428a9c16aeab08e/.gitignore" crawlab_master | unexpected EOF

tikazyq commented 1 year ago

现在还存在这个问题吗?

bestRunner commented 1 year ago

现在还存在这个问题吗?

存在

tikazyq commented 1 year ago

看起来像是 seaweedfs 出问题了,请重启主节点再试试