goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
24.01k stars 4.75k forks source link

Harbor job service memory used continued growth #20863

Open prometheus-tao opened 2 months ago

prometheus-tao commented 2 months ago

I found more InProgress status tasks in harbor ui , I try to stoped REPLICATION , InProcess task and docker restart harbor_jobservi , The follow is jobservice file error show.

Harbor_Version: 2.8.2 harbor_jobservi Memory Used : 35% (continued growth) machine: 8C16G

event error: {"errors":[{"code":"NOT_FOUND","message":"task with job ID xxx not found"}]} Aug 20 05:31:44 xx.xxx.xx.xx jobservice[1844]: , evt=status change: job=xxx, status=Error, revision=1723898689->http://core:8080/service/notifications/tasks/1004795, duration=1m2.510162039s Aug 20 05:31:46 xx.xxx.xx.xx jobservice[1844]: 2024-08-20T00:01:46Z [ERROR] [/jobservice/hook/hook_agent.go:159]: Retry: still failed after all retries: {"errors":[{"code":"NOT_FOUND","message":"task with job ID xxx not found"}]} Aug 20 05:31:46 xx.xxx.xx.xx jobservice[1844]: , evt=status change: job=xxx , status=Error, revision=1723894062->http://core:8080/service/notifications/tasks/1002949 Aug 20 05:31:46 xx.xxx.xx.xx jobservice[1844]: 2024-08-20T00:01:46Z [ERROR] [/jobservice/hook/hook_agent.go:155]: Retry: sending hook event error: {"errors":[{"code":"NOT_FOUND","message":"task with job ID xxx not found"}]} Aug 20 05:31:46 xx.xxx.xx.xx jobservice[1844]: , evt=status change: job=xxx , status=Error, revision=1724011394->http://core:8080/service/notifications/tasks/1047878, duration=1.839486175s

chlins commented 1 month ago

Please visit the job service dashboard to check if there are any pending or currently running jobs.

prometheus-tao commented 1 month ago

I manual stoped all pending status jobs, This is prombles occur when replications setted cron task ' /5 *'

prometheus-tao commented 1 month ago

I think the cron setted ' /5 *' is bug for harbor