itamarst
commented
4 years ago Alaric says:
Basically, deleting dots on the production cluster was hanging. Looking in the code, it was putting things into etcd to mark the filesystem for deletion, but then the etcd watcher wasn't picking up that change to trigger the rest of the process. I suspected this was just down to etcd being overloaded on production... its logs were full of complaining about operations taking too long.
I think I'm coming down with amnesia! Deletion of what - projects, runners, files? Can you add more details to this @alaric-dotmesh as we may not remember when we eventually pick this issue up again.