search

galaxyproject / pulsar

Distributed job execution application built for Galaxy
https://pulsar.readthedocs.io
Apache License 2.0
37 stars 50 forks source link

Deleted jobs do not get cleaned up from `${manager}-preprocessing-jobs` #355

Open natefoo opened 6 months ago

natefoo commented 6 months ago 
[ndc@galaxy-db-02]~% psql -c "SELECT state, count(state) AS ct FROM job WHERE id in ($(ssh rocky@jetstream2.galaxyproject.org ls -rt /srv/pulsar/main/var/persisted_data/jetstream2-preprocessing-jobs | xargs echo | tr ' ' ,)) GROUP BY state"
  state  | ct
---------+-----
 deleted | 513
 error   |  15
 queued  |   2
(3 rows)
natefoo commented 6 months ago

Although the oldest was a week old, so somehow at some point they do, possibly on restart?

EDIT: yeah

[rocky@jetstream2 jetstream2-preprocessing-jobs]$ systemctl status pulsar-main
● pulsar-main.service - Pulsar
     Loaded: loaded (/etc/systemd/system/pulsar-main.service; enabled; preset: disabled)
     Active: active (running) since Thu 2024-03-14 13:42:30 UTC; 1 week 0 days ago