Add a Wait switch to wait for the Databricks Job to finish

[blueelvis]

Hi,

I think it would be a good use case to add a -Wait switch for the Start-DatabricksJob (Or rename it to Execute-DatabricksJob) which keeps on waiting and outputs the status say every 30 seconds about the job.

-Pranav

[RichieBzzzt]

Hello,

there is a similar thing around Add-DatabricksNotebookJob . It might be better to create a separate function that does something like this -

   $runId = Add-DatabricksNotebookJob @DatabricksNotebookJob
    Write-Host "Run ID - "$runId
    Write-Host "Running tests. This is the info for the job running the tests..."
    Get-DatabricksRun -RunId $runId
    $DatabricksJobRunState = $null
    $wait = 1
    $currentStatus = "INITIATED"
    Write-Host "There is a 20 minute timeout for test jobs."
    do {
        Start-Sleep -Seconds 10
        $DatabricksJobRunState = Get-DatabricksRun -RunId $runId     
        $DatabricksJobRunLifeCycleState = $DatabricksJobRunState.state.life_cycle_state
        if ($currentStatus -ne $DatabricksJobRunLifeCycleState) {
            Write-Host "Status has altered from $currentStatus to $DatabricksJobRunLifeCycleState."
            $currentStatus = $DatabricksJobRunLifeCycleState
        }
        if (($wait % 5) -eq 0) {
            Write-Host "Job has not completed. Status is $DatabricksJobRunLifeCycleState."
        }
        $wait ++
    } until (($DatabricksJobRunLifeCycleState -notin "PENDING", "RUNNING", "TERMINATING", "SKIPPED", "INTERNAL_ERROR") -or ($wait -eq 120))
    Write-Host "Test Completed. Check here for the status of the job."
    $databricksRun = Get-DatabricksRun -RunId $runId     
    if ($databricksRun.state.result_state -ne "SUCCESS") {
        $databricksRun
        Write-Error "Job failed. Use url to open job and investigate"
        throw
    }

@simondmorias I can add this if you agree?

[simondmorias]

Sounds good. I think there are a number of places this would be useful. If you are happy to take on then that would be great. Thanks

[blueelvis]

Yeah, the -Wait would be beneficial in a lot of places. Right now, I am using this a similar logic as @RichieBzzzt mentioned in our scripts which is tedious. For example, cluster creation, library addition etc. It would be helpful in a lot of places :) Thanks!