Terminating instances cause some status queries to hang

[priteau]

When terminating an instance, any global status query and any status query on the instance hangs.

This is caused by the WorkspaceHomeImpl:destroy method taking a per-instance lock for the duration of the whole termination. This includes a lengthy call to the workspace control agent. The WorkspaceHomeImpl:find method also tries to take this lock and hangs while the termination is in progress.

[timf]

WSRF destroy semantics originally prohibited that destroy from being asynchronous. Converting it to asynchronous (WSRF semantics are probably not important at this point) would probably be the best overall solution but most work, consider the client half of that, too. Perhaps make it still appear synchronous to the caller whilst releasing the lock? Or allow read-only lock free access for status queries?

[priteau]

Tim, thank you very much for taking the time to comment on this issue!

I've taken a simple approach where I release the lock during the call to workspace control. Would you mind reviewing it? It is commit dd41e55173542e654e8dfa92763e1975794d36b6.

[timf]

I think if you remove the lock, other actions could proceed during the destroy. I can't dig into the code but to protect against this there would at least need to be a per-instance lock around a state change to "destroying" (if there is such an intermediate state, I can't remember) and then launch the long running task. That way, other attempts to alter the instance would see it as destroying and nothing could happen. Something like that, sorry I can't research it.

[priteau]

I think that is what I am doing. I keep the lock for almost everything during the destroy, except what I think is calling workspace-control.

[timf]

Oh.. then that sounds OK.

[priteau]

The aforementioned approach had some issues, so I reworked it by introducing an extra lock dedicated to destroy.