Closed Qi77Qi closed 2 months ago
Attention: Patch coverage is 75.00000%
with 1 line
in your changes missing coverage. Please review.
Project coverage is 73.92%. Comparing base (
a6893f4
) to head (e87240d
).
@lucymcnatt @cindy-broadinstitute I updated the PR a bit..the previous location where I added the diskId isn't correct given current implementation of the handleKubernetesError
and cleanUpAfterCreateAppError
. I added a comment to clarify.
I added the diskId when disk fails to detach instead because this is what I see in BEE after I launch an RStudio with bad helm chart
{"@timestamp":"2024-07-02T15:48:52.819519274Z","@version":"1","logger_name":"org.broadinstitute.dsde.workbench.leonardo.http.Boot","traceId":"90134d7c-631e-4dcc-ac8b-9d6044b5b2a6","version":"aee05698f","thread_name":"io-compute-blocker-100","stack_trace":"org.broadinstitute.dsde.workbench.leonardo.monitor.PubsubHandleMessageError$PubsubKubernetesError: An error occurred with a kubernetes operation from source disk during action deleteApp. \nOriginal message: The data disk failed to detach within the time limit, cannot proceed with delete disk\n","severity":"ERROR","serviceContext":{"service":"leonardo","version":"aee05698f"},"message":"An error occurred during resource clean up for app AppName(qi-rstudio-7-2-0) in project terra-quality-d7a5f18a"}
Basically, the disk wasn't cleaned up becuz it failed to detach. This PR will make is such that when disk fail to detach, the disk status will be marked as FAILED
in handleKubernetesError
instead of remaining in Ready
status. There might be some issue in how we determine disk is detached today worth looking into as well.
Last week, we had some bug in SAS chart that was preventing app creation to succeed. This revealed a different bug that if an app creation fails, the newly created PD may not be cleaned up, and this causes all future app creation in the same workspace to fail because the following error
What this PR will does is pass the diskId to the
handleKubernetesError
function, where it'll transition the disk status toFailed
. This way, UI won't attempt to reuse this disk and run into the above exception. We might want to consider adding a step to delete PD in GCP in the error handler for future improvements.Jira ticket: https://precisionmedicineinitiative.atlassian.net/browse/RW-12956
Summary of changes
What
-
Why
-
Testing these changes
What to test
Who tested and where
jenkins retest
orjenkins multi-test
.