crawlab-team / crawlab

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
https://www.crawlab.cn
BSD 3-Clause "New" or "Revised" License
11.38k stars 1.8k forks source link

定时任务执行失败,附容器报错日志 #1084

Closed briangt closed 1 year ago

briangt commented 2 years ago

Bug 描述 定时任务触发时报错,该爬虫任务手动执行时无问题,只配置了主节点,节点未满载,尝试过重启容器无效

截屏 爬虫配置 image

定时任务配置 image

下图第二个任务是手动执行的,无问题,第一个任务由定时任务触发,一直为待定状态,且未展示节点等数据 image

容器日志 2022/03/23 19:28:00 info cron: wake now 2022-03-23 19:28:00.001160995 +0800 CST 2022/03/23 19:28:00 info cron: run now 2022-03-23 19:28:00.001160995 +0800 CST entry 1 next 2022-03-23 19:30:00 +0800 CST 2022/03/23 19:28:00 error cron: panic stack ... goroutine 3458 [running]: github.com/robfig/cron/v3.Recover.func1.1.1(0x1437190, 0x1b2bc10) /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/chain.go:45 +0xbd panic(0x1099b20, 0x1adbd20) /usr/local/go/src/runtime/panic.go:965 +0x1b9 github.com/crawlab-team/crawlab-core/models/models.(User).GetId(0x0, 0x0, 0xc000000000) /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/models/models/user.go:17 +0x17 github.com/crawlab-team/crawlab-core/models/delegate.(ModelDelegate).upsertArtifact(0xc00084db60, 0x11b2fe0, 0xc000d9ed80) /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/models/delegate/model.go:300 +0x442 github.com/crawlab-team/crawlab-core/models/delegate.(ModelDelegate).add(0xc00084db60, 0x5, 0x0) /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/models/delegate/model.go:182 +0x125 github.com/crawlab-team/crawlab-core/models/delegate.(ModelDelegate).do(0xc00084db60, 0x120fafa, 0x3, 0x144db68, 0xc00084db60) /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/models/delegate/model.go:148 +0x94 github.com/crawlab-team/crawlab-core/models/delegate.(ModelDelegate).Add(0xc00084db60, 0xc000d9ed80, 0xc000467d70) /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/models/delegate/model.go:106 +0x45 github.com/crawlab-team/crawlab-core/task/scheduler.(Service).Enqueue(0xc000119200, 0x14574d8, 0xc000d9ed80, 0x0, 0x0) /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/task/scheduler/service.go:61 +0x182 github.com/crawlab-team/crawlab-core/spider/admin.(Service).scheduleTasks(0xc000251780, 0xc0002b89d8, 0xc00032c1b0, 0xc0002b89d8, 0x0) /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/spider/admin/service.go:111 +0x3e2 github.com/crawlab-team/crawlab-core/spider/admin.(Service).Schedule(0xc000251780, 0xabc24e9c2ef43a62, 0x5b8b065b, 0xc00032c1b0, 0x0, 0x0) /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/spider/admin/service.go:45 +0x85 github.com/crawlab-team/crawlab-core/schedule.(Service).schedule.func1() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/schedule/service.go:235 +0x489 github.com/robfig/cron/v3.FuncJob.Run(0xc000372840) /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/cron.go:131 +0x25 github.com/robfig/cron/v3.Recover.func1.1() /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/chain.go:53 +0x68 github.com/robfig/cron/v3.FuncJob.Run(0xc000b36cf0) /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/cron.go:131 +0x25 github.com/robfig/cron/v3.(Cron).startJob.func1(0xc0002ece60, 0x142e720, 0xc000b36cf0) /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/cron.go:307 +0x65 created by github.com/robfig/cron/v3.(*Cron).startJob /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/cron.go:305 +0x73

runtime error: invalid memory address or nil pointer dereference /go/pkg/mod/github.com/crawlab-team/go-trace@v0.1.0/trace.go:6 github.com/crawlab-team/go-trace.PrintError() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/schedule/logger.go:22 github.com/crawlab-team/crawlab-core/schedule.(Logger).Error() /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/chain.go:50 github.com/robfig/cron/v3.Recover.func1.1.1() /usr/local/go/src/runtime/panic.go:965 runtime.gopanic() /usr/local/go/src/runtime/panic.go:212 runtime.panicmem() /usr/local/go/src/runtime/signal_unix.go:734 runtime.sigpanic() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/models/models/user.go:17 github.com/crawlab-team/crawlab-core/models/models.(User).GetId() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/models/delegate/model.go:300 github.com/crawlab-team/crawlab-core/models/delegate.(ModelDelegate).upsertArtifact() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/models/delegate/model.go:182 github.com/crawlab-team/crawlab-core/models/delegate.(ModelDelegate).add() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/models/delegate/model.go:148 github.com/crawlab-team/crawlab-core/models/delegate.(ModelDelegate).do() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/models/delegate/model.go:106 github.com/crawlab-team/crawlab-core/models/delegate.(ModelDelegate).Add() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/task/scheduler/service.go:61 github.com/crawlab-team/crawlab-core/task/scheduler.(Service).Enqueue() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/spider/admin/service.go:111 github.com/crawlab-team/crawlab-core/spider/admin.(Service).scheduleTasks() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/spider/admin/service.go:45 github.com/crawlab-team/crawlab-core/spider/admin.(Service).Schedule() /go/pkg/mod/github.com/crawlab-team/crawlab-core@v0.6.0-beta.20211213.2006/schedule/service.go:235 github.com/crawlab-team/crawlab-core/schedule.(Service).schedule.func1() /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/cron.go:131 github.com/robfig/cron/v3.FuncJob.Run() /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/chain.go:53 github.com/robfig/cron/v3.Recover.func1.1() /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/cron.go:131 github.com/robfig/cron/v3.FuncJob.Run() /go/pkg/mod/github.com/robfig/cron/v3@v3.0.0/cron.go:307 github.com/robfig/cron/v3.(*Cron).startJob.func1() /usr/local/go/src/runtime/asm_amd64.s:1371 runtime.goexit()

tikazyq commented 2 years ago

Can you try again with the latest version?

fightingfish008 commented 2 years ago

今天更新到了最新新版本(0.6.0)。设置定时任务,也是遇到定时器不起作用的问题呢,手动指定节点启动正常。

image

日志节点并无任何反应,然而手动指定节点并启动时,就有日志打印。 image

且发现每次进来后之前选定的节点都显示为未选状态。

fightingfish008 commented 2 years ago

这边回复下之前遇到的定时任务问题,这边换了台性能稍高的机器即可(之前4G内存,新机器8G内存)

thachnv92 commented 1 year ago

I just got this bug with v0.6.0-1.