search

seqeralabs / nf-tower

Nextflow Tower system
https://tower.nf
Mozilla Public License 2.0
143 stars 51 forks source link

Zombie runs, redux: Jobs still shown as running #354

Closed lina-kim closed 2 years ago

lina-kim commented 2 years ago

Hi all,

First, thanks for creating and maintaining Nextflow Tower! I've been using it since it was in beta, and am a big fan.

I recently had an issue identical to #313: A number of previously killed jobs (15 in all) are still marked active on my dashboard, some apparently active for three weeks. All were launched and killed from the command line. Active tasks are labeled either aborted or running, with no exit codes.

Screenshot 2022-05-18 at 10 36 00

This isn't something I've seen in previous releases, so figured I'd bring this up. Thanks!

telatin commented 2 years ago

I have the same issue of "zombie" runs. Sometimes runs that didn't even start (e.g. failed to locate input reads) but all appear and cannot be deleted due to being ongoing.

mag g

pditommaso commented 2 years ago

Can you please copy & paste the workflow Ids here?

lina-kim commented 2 years ago

Thanks! Is there a good way of avoiding these in the future, other than not killing local jobs from the command line?

Workflow IDs for my zombie runs:

telatin commented 2 years ago

Recent zombie jobs:

Older zombie job still "running":

Hope I got them all right... Thanks for checking, @pditommaso !

pditommaso commented 2 years ago

Thanks for reporting this problem. We have indeed found an issue related to this. Some of the above workflows should be reported as terminated now.

lina-kim commented 2 years ago

Most are now terminated - thanks @pditommaso!