search

ideal / mirror

Task scheduler for open source mirror site (initially for https://mirror.bjtu.edu.cn)
Other
207 stars 51 forks source link

无规律Crash #17

Open chestnutprog opened 3 years ago

chestnutprog commented 3 years ago

升级后有时会在运行中异常退出，暂时没有发现规律。

附两次crash的日志

Sep 24 14:00:00 mirror mirrord[25259]: 09-24 14:00:00 [ERROR   ][mirror.main     ] [Errno 3] No such process
Sep 24 14:00:00 mirror mirrord[25259]: Traceback (most recent call last):
Sep 24 14:00:00 mirror mirrord[25259]:   File "/usr/local/lib/python3.6/dist-packages/mirror/main.py", line 249, in start_daemon
Sep 24 14:00:00 mirror mirrord[25259]:     daemon.start()
Sep 24 14:00:00 mirror mirrord[25259]:   File "/usr/local/lib/python3.6/dist-packages/mirror/daemon.py", line 43, in start
Sep 24 14:00:00 mirror mirrord[25259]:     self.scheduler.start()
Sep 24 14:00:00 mirror mirrord[25259]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 84, in start
Sep 24 14:00:00 mirror mirrord[25259]:     self.schedule()
Sep 24 14:00:00 mirror mirrord[25259]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 169, in schedule
Sep 24 14:00:00 mirror mirrord[25259]:     self.stop_task(taskinfo)
Sep 24 14:00:00 mirror mirrord[25259]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 458, in stop_task
Sep 24 14:00:00 mirror mirrord[25259]:     task.stop()
Sep 24 14:00:00 mirror mirrord[25259]:   File "/usr/local/lib/python3.6/dist-packages/mirror/task.py", line 160, in stop
Sep 24 14:00:00 mirror mirrord[25259]:     os.kill(self.pid, signo)
Sep 24 14:00:00 mirror mirrord[25259]: ProcessLookupError: [Errno 3] No such process
Sep 24 14:00:00 mirror mirrord[25259]: 09-24 14:00:00 [INFO    ][taskstatus      ] Task Start: archlinuxcn
Sep 24 14:00:00 mirror mirrord[25259]: 09-24 14:00:00 [INFO    ][taskstatus      ] Task Enqueue: archlinuxcn
Sep 24 14:00:00 mirror mirrord[25259]: 09-24 14:00:00 [INFO    ][taskstatus      ] Task Start: archlinuxarm
Sep 24 14:00:00 mirror mirrord[25259]: 09-24 14:00:00 [INFO    ][taskstatus      ] Task Enqueue: archlinuxarm
Sep 24 14:00:00 mirror mirrord[25259]: 09-24 14:00:00 [INFO    ][taskstatus      ] Task Start: linuxmint
Sep 24 14:00:00 mirror mirrord[25259]: 09-24 14:00:00 [INFO    ][taskstatus      ] Task Enqueue: linuxmint
Sep 24 14:00:00 mirror mirrord[25259]: 09-24 14:00:00 [INFO    ][taskstatus      ] Task Start: grafana-panel
Sep 24 14:00:00 mirror mirrord[25259]: 09-24 14:00:00 [INFO    ][taskstatus      ] Task Enqueue: grafana-panel
Sep 24 14:01:00 mirror mirrord[25259]: --- Logging error ---
Sep 24 14:01:00 mirror mirrord[25259]: Traceback (most recent call last):
Sep 24 14:01:00 mirror mirrord[25259]:   File "/usr/lib/python3.6/logging/handlers.py", line 71, in emit
Sep 24 14:01:00 mirror mirrord[25259]:     if self.shouldRollover(record):
Sep 24 14:01:00 mirror mirrord[25259]:   File "/usr/lib/python3.6/logging/handlers.py", line 185, in shouldRollover
Sep 24 14:01:00 mirror mirrord[25259]:     self.stream = self._open()
Sep 24 14:01:00 mirror mirrord[25259]:   File "/usr/lib/python3.6/logging/__init__.py", line 1061, in _open
Sep 24 14:01:00 mirror mirrord[25259]:     return open(self.baseFilename, self.mode, encoding=self.encoding)
Sep 24 14:01:00 mirror mirrord[25259]: NameError: name 'open' is not defined
Sep 24 14:01:00 mirror mirrord[25259]: Call stack:
Sep 24 14:01:00 mirror mirrord[25259]:   File "/usr/local/lib/python3.6/dist-packages/mirror/component.py", line 35, in __del__
Sep 24 14:01:00 mirror mirrord[25259]:     _component_registry.deregister(self)
Sep 24 14:01:00 mirror mirrord[25259]:   File "/usr/local/lib/python3.6/dist-packages/mirror/component.py", line 90, in deregister
Sep 24 14:01:00 mirror mirrord[25259]:     self.stop([obj._name])
Sep 24 14:01:00 mirror mirrord[25259]:   File "/usr/local/lib/python3.6/dist-packages/mirror/component.py", line 141, in stop
Sep 24 14:01:00 mirror mirrord[25259]:     self.components[name].stop()
Sep 24 14:01:00 mirror mirrord[25259]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 320, in stop
Sep 24 14:01:00 mirror mirrord[25259]:     log.info("Stopping mirror scheduler")
Sep 24 14:01:00 mirror mirrord[25259]: Message: 'Stopping mirror scheduler'
Sep 24 14:01:00 mirror mirrord[25259]: Arguments: ()
Sep 24 14:01:00 mirror mirrord[25259]: 09-24 14:01:00 [INFO    ][mirror.scheduler] Stopping mirror scheduler
Sep 24 14:01:00 mirror systemd[1]: mirrord@bob.service: Main process exited, code=exited, status=1/FAILURE
Sep 24 14:01:00 mirror systemd[1]: mirrord@bob.service: Failed with result 'exit-code'.
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][systemtask      ] Finished task: logcleaner
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][taskstatus      ] Task Enqueue: logcleaner
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][taskstatus      ] Task Start: statistics-dayly
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][taskstatus      ] Task Enqueue: statistics-dayly
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [ERROR   ][pluginthread    ] Event handler TaskEnqueueEvent failed in <bound method Plugin.__on_task_enqueue of <mirror.plugins.taskstatus.plugin.Plugin object at 0x7fa3e148ddd8>> with exception: 'NoneType' object has no attribute 'time'
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][taskstatus      ] Task Start: grafana-panel
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][taskstatus      ] Task Enqueue: grafana-panel
Sep 30 01:00:24 mirror mirrord[10157]: 09-30 01:00:24 [INFO    ][mirror.scheduler] Task: grafana-panel ended with return code 0, pid 24723
Sep 30 01:00:24 mirror mirrord[10157]: Exception ignored in: <module 'threading' from '/usr/lib/python3.6/threading.py'>
Sep 30 01:00:24 mirror mirrord[10157]: Traceback (most recent call last):
Sep 30 01:00:24 mirror mirrord[10157]:   File "/usr/lib/python3.6/threading.py", line 1294, in _shutdown
Sep 30 01:00:24 mirror mirrord[10157]:     t.join()
Sep 30 01:00:24 mirror mirrord[10157]:   File "/usr/lib/python3.6/threading.py", line 1056, in join
Sep 30 01:00:24 mirror mirrord[10157]:     self._wait_for_tstate_lock()
Sep 30 01:00:24 mirror mirrord[10157]:   File "/usr/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
Sep 30 01:00:24 mirror mirrord[10157]:     elif lock.acquire(block, timeout):
Sep 30 01:00:24 mirror mirrord[10157]:   File "/usr/local/lib/python3.6/dist-packages/mirror/handler.py", line 92, in sigchld_handler
Sep 30 01:00:24 mirror mirrord[10157]:     raise mirror.error.MirrordTaskFinishedFakeError("Task finished, please stop sleep")
Sep 30 01:00:24 mirror mirrord[10157]: mirror.error.MirrordTaskFinishedFakeError: Task finished, please stop sleep
Sep 30 01:00:24 mirror systemd[1]: mirrord@bob.service: Main process exited, code=exited, status=1/FAILURE
Sep 30 01:00:25 mirror systemd[1]: mirrord@bob.service: Failed with result 'exit-code'.
ideal commented 3 years ago

我看一下。

ideal commented 3 years ago

第二部分这个，上面还有相关的日志吗？

chestnutprog commented 3 years ago

@ideal 

Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][mirror.scheduler] I am waking up...
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][mirror.scheduler] Starting task: statistics-dayly ...
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][mirror.scheduler] Task: statistics-dayly begin to run with pid 24722
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][mirror.scheduler] Starting task: grafana-panel ...
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][mirror.scheduler] Task: grafana-panel begin to run with pid 24723
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][mirror.scheduler] Task: statistics-dayly ended with return code 1, pid 24722
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [ERROR   ][mirror.main     ] Task finished, please stop sleep
Sep 30 01:00:00 mirror mirrord[10157]: Traceback (most recent call last):
Sep 30 01:00:00 mirror mirrord[10157]:   File "/usr/local/lib/python3.6/dist-packages/mirror/main.py", line 249, in start_daemon
Sep 30 01:00:00 mirror mirrord[10157]:     daemon.start()
Sep 30 01:00:00 mirror mirrord[10157]:   File "/usr/local/lib/python3.6/dist-packages/mirror/daemon.py", line 43, in start
Sep 30 01:00:00 mirror mirrord[10157]:     self.scheduler.start()
Sep 30 01:00:00 mirror mirrord[10157]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 81, in start
Sep 30 01:00:00 mirror mirrord[10157]:     self.sleep()
Sep 30 01:00:00 mirror mirrord[10157]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 92, in sleep
Sep 30 01:00:00 mirror mirrord[10157]:     self.append_tasks()
Sep 30 01:00:00 mirror mirrord[10157]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 259, in append_tasks
Sep 30 01:00:00 mirror mirrord[10157]:     self.append_task(taskname, task, since = now)
Sep 30 01:00:00 mirror mirrord[10157]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 277, in append_task
Sep 30 01:00:00 mirror mirrord[10157]:     if taskinfo in self.queue:
Sep 30 01:00:00 mirror mirrord[10157]:   File "/usr/local/lib/python3.6/dist-packages/mirror/queue.py", line 103, in __contains__
Sep 30 01:00:00 mirror mirrord[10157]:     return item in self.queue
Sep 30 01:00:00 mirror mirrord[10157]:   File "/usr/local/lib/python3.6/dist-packages/mirror/queue.py", line 33, in __eq__
Sep 30 01:00:00 mirror mirrord[10157]:     return self.name == other.name
Sep 30 01:00:00 mirror mirrord[10157]:   File "/usr/local/lib/python3.6/dist-packages/mirror/handler.py", line 92, in sigchld_handler
Sep 30 01:00:00 mirror mirrord[10157]:     raise mirror.error.MirrordTaskFinishedFakeError("Task finished, please stop sleep")
Sep 30 01:00:00 mirror mirrord[10157]: mirror.error.MirrordTaskFinishedFakeError: Task finished, please stop sleep
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][systemtask      ] Running task: logcleaner
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][logcleaner      ] Deleted log file: /var/log/rsync/ubuntu-releases.log.2020-09-19
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][logcleaner      ] Deleted log file: /var/log/rsync/opensuse.log.2020-09-19

......

Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][logcleaner      ] Deleted log file: /var/log/rsync/centos.log.2020-09-19
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][systemtask      ] Finished task: logcleaner
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][taskstatus      ] Task Enqueue: logcleaner
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][taskstatus      ] Task Start: statistics-dayly
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][taskstatus      ] Task Enqueue: statistics-dayly
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [ERROR   ][pluginthread    ] Event handler TaskEnqueueEvent failed in <bound method Plugin.__on_task_enqueue of <mirror.plugins.taskstatus.plugin.Plugin object at 0x7fa3e148ddd8>> with exception: 'NoneType' object has no attribute 'time'
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][taskstatus      ] Task Start: grafana-panel
Sep 30 01:00:00 mirror mirrord[10157]: 09-30 01:00:00 [INFO    ][taskstatus      ] Task Enqueue: grafana-panel
ideal commented 3 years ago

如还有问题请重新打开该issue。

chestnutprog commented 3 years ago 
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Enqueue: archlinuxcn
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Start: archlinuxarm
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Enqueue: archlinuxarm
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Start: archlinux
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Enqueue: archlinux
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Start: ubuntu
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Enqueue: ubuntu
Nov 02 20:00:04 mirror mirrord[1754]: 11-02 20:00:04 [INFO    ][mirror.scheduler] Task: archlinuxarm ended with return code 0, pid 45084
Nov 02 20:00:04 mirror mirrord[1754]: 11-02 20:00:04 [INFO    ][mirror.scheduler] I am going to sleep, next waking up: Mon Nov  2 20:02:00 2020
Nov 02 20:00:04 mirror mirrord[1754]: 11-02 20:00:04 [INFO    ][taskstatus      ] Task Enqueue: archlinuxarm
Nov 02 20:00:08 mirror mirrord[1754]: 11-02 20:00:08 [INFO    ][mirror.scheduler] Task: archlinux ended with return code 0, pid 45085
Nov 02 20:00:08 mirror mirrord[1754]: 11-02 20:00:08 [INFO    ][mirror.scheduler] Task: archlinuxcn ended with return code 0, pid 45082
Nov 02 20:00:08 mirror mirrord[1754]: 11-02 20:00:08 [ERROR   ][mirror.main     ] Task finished, please stop sleep
Nov 02 20:00:08 mirror mirrord[1754]: Traceback (most recent call last):
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/main.py", line 249, in start_daemon
Nov 02 20:00:08 mirror mirrord[1754]:     daemon.start()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/daemon.py", line 43, in start
Nov 02 20:00:08 mirror mirrord[1754]:     self.scheduler.start()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 84, in start
Nov 02 20:00:08 mirror mirrord[1754]:     self.schedule()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 129, in schedule
Nov 02 20:00:08 mirror mirrord[1754]:     self.init_sysinfo()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 217, in init_sysinfo
Nov 02 20:00:08 mirror mirrord[1754]:     self.current_conn = tcpconn()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/sysinfo.py", line 76, in tcpconn
Nov 02 20:00:08 mirror mirrord[1754]:     conn = tcp.search(line).groupdict()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/handler.py", line 92, in sigchld_handler
Nov 02 20:00:08 mirror mirrord[1754]:     raise mirror.error.MirrordTaskFinishedFakeError("Task finished, please stop sleep")
Nov 02 20:00:08 mirror mirrord[1754]: mirror.error.MirrordTaskFinishedFakeError: Task finished, please stop sleep
Nov 02 20:00:23 mirror mirrord[1754]: 11-02 20:00:23 [INFO    ][mirror.scheduler] Task: grafana-panel ended with return code 0, pid 45080
Nov 02 20:00:23 mirror mirrord[1754]: Exception ignored in: <module 'threading' from '/usr/lib/python3.6/threading.py'>
Nov 02 20:00:23 mirror mirrord[1754]: Traceback (most recent call last):
Nov 02 20:00:23 mirror mirrord[1754]:   File "/usr/lib/python3.6/threading.py", line 1294, in _shutdown
Nov 02 20:00:23 mirror mirrord[1754]:     t.join()
Nov 02 20:00:23 mirror mirrord[1754]:   File "/usr/lib/python3.6/threading.py", line 1056, in join
Nov 02 20:00:23 mirror mirrord[1754]:     self._wait_for_tstate_lock()
Nov 02 20:00:23 mirror mirrord[1754]:   File "/usr/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
Nov 02 20:00:23 mirror mirrord[1754]:     elif lock.acquire(block, timeout):
Nov 02 20:00:23 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/handler.py", line 92, in sigchld_handler
Nov 02 20:00:23 mirror mirrord[1754]:     raise mirror.error.MirrordTaskFinishedFakeError("Task finished, please stop sleep")
Nov 02 20:00:23 mirror mirrord[1754]: mirror.error.MirrordTaskFinishedFakeError: Task finished, please stop sleep
Nov 02 20:00:23 mirror systemd[1]: mirrord@bob.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 20:00:23 mirror systemd[1]: mirrord@bob.service: Failed with result 'exit-code'.
chestnutprog commented 3 years ago

reopen的权限似乎被关掉了？

ideal commented 3 years ago
屏幕快照 2020-11-05 上午12 07 54

在底下评论框这里，应该有的。

ideal commented 3 years ago 
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Enqueue: archlinuxcn
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Start: archlinuxarm
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Enqueue: archlinuxarm
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Start: archlinux
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Enqueue: archlinux
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Start: ubuntu
Nov 02 20:00:00 mirror mirrord[1754]: 11-02 20:00:00 [INFO    ][taskstatus      ] Task Enqueue: ubuntu
Nov 02 20:00:04 mirror mirrord[1754]: 11-02 20:00:04 [INFO    ][mirror.scheduler] Task: archlinuxarm ended with return code 0, pid 45084
Nov 02 20:00:04 mirror mirrord[1754]: 11-02 20:00:04 [INFO    ][mirror.scheduler] I am going to sleep, next waking up: Mon Nov  2 20:02:00 2020
Nov 02 20:00:04 mirror mirrord[1754]: 11-02 20:00:04 [INFO    ][taskstatus      ] Task Enqueue: archlinuxarm
Nov 02 20:00:08 mirror mirrord[1754]: 11-02 20:00:08 [INFO    ][mirror.scheduler] Task: archlinux ended with return code 0, pid 45085
Nov 02 20:00:08 mirror mirrord[1754]: 11-02 20:00:08 [INFO    ][mirror.scheduler] Task: archlinuxcn ended with return code 0, pid 45082
Nov 02 20:00:08 mirror mirrord[1754]: 11-02 20:00:08 [ERROR   ][mirror.main     ] Task finished, please stop sleep
Nov 02 20:00:08 mirror mirrord[1754]: Traceback (most recent call last):
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/main.py", line 249, in start_daemon
Nov 02 20:00:08 mirror mirrord[1754]:     daemon.start()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/daemon.py", line 43, in start
Nov 02 20:00:08 mirror mirrord[1754]:     self.scheduler.start()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 84, in start
Nov 02 20:00:08 mirror mirrord[1754]:     self.schedule()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 129, in schedule
Nov 02 20:00:08 mirror mirrord[1754]:     self.init_sysinfo()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/scheduler.py", line 217, in init_sysinfo
Nov 02 20:00:08 mirror mirrord[1754]:     self.current_conn = tcpconn()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/sysinfo.py", line 76, in tcpconn
Nov 02 20:00:08 mirror mirrord[1754]:     conn = tcp.search(line).groupdict()
Nov 02 20:00:08 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/handler.py", line 92, in sigchld_handler
Nov 02 20:00:08 mirror mirrord[1754]:     raise mirror.error.MirrordTaskFinishedFakeError("Task finished, please stop sleep")
Nov 02 20:00:08 mirror mirrord[1754]: mirror.error.MirrordTaskFinishedFakeError: Task finished, please stop sleep
Nov 02 20:00:23 mirror mirrord[1754]: 11-02 20:00:23 [INFO    ][mirror.scheduler] Task: grafana-panel ended with return code 0, pid 45080
Nov 02 20:00:23 mirror mirrord[1754]: Exception ignored in: <module 'threading' from '/usr/lib/python3.6/threading.py'>
Nov 02 20:00:23 mirror mirrord[1754]: Traceback (most recent call last):
Nov 02 20:00:23 mirror mirrord[1754]:   File "/usr/lib/python3.6/threading.py", line 1294, in _shutdown
Nov 02 20:00:23 mirror mirrord[1754]:     t.join()
Nov 02 20:00:23 mirror mirrord[1754]:   File "/usr/lib/python3.6/threading.py", line 1056, in join
Nov 02 20:00:23 mirror mirrord[1754]:     self._wait_for_tstate_lock()
Nov 02 20:00:23 mirror mirrord[1754]:   File "/usr/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
Nov 02 20:00:23 mirror mirrord[1754]:     elif lock.acquire(block, timeout):
Nov 02 20:00:23 mirror mirrord[1754]:   File "/usr/local/lib/python3.6/dist-packages/mirror/handler.py", line 92, in sigchld_handler
Nov 02 20:00:23 mirror mirrord[1754]:     raise mirror.error.MirrordTaskFinishedFakeError("Task finished, please stop sleep")
Nov 02 20:00:23 mirror mirrord[1754]: mirror.error.MirrordTaskFinishedFakeError: Task finished, please stop sleep
Nov 02 20:00:23 mirror systemd[1]: mirrord@bob.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 20:00:23 mirror systemd[1]: mirrord@bob.service: Failed with result 'exit-code'.

这个的版本是？

ideal commented 3 years ago

你先参考下 https://github.com/ideal/mirror/blob/master/util/systemd/system/mirrord%40.service#L11 修改下service，让systemd可以在任务失败退出后自动重启，可能需要sudo systemctl daemon-reload。我再看下这个问题。