get-databrickscluster not returning anything

[chlsmith]

Running Get-DatabricksCluster and I'm getting no results.

Running a Debug shows that the LIST endpoint is being hit as expected.

Invoke-RestMethod `
        -Uri 'https://adb-8989366248458805.5.azuredatabricks.net/api/2.0/clusters/list' `
        -Method GET `
        -Headers @{
  "Authorization" = "Bearer eyJ0eXAiOiJKV1blahblahblahblah

Running Verbose shows all of my clusters in json format, so I'm receiving the payload.

VERBOSE: HTTP/1.1 GET with 0-byte payload
VERBOSE: received -byte response of content type application/json
VERBOSE: Content encoding: utf-8
VERBOSE: Response: "{\"clusters\":[{\"cluster_id\":\"1108-16181(everything else removed)

All of the clusters are in TERMINATED state.

I have validated that all clusters are returned as expected using the Databricks CLI, so my permissions are as they should be.

I'm happy to help troubleshoot in any way I can, but other than checking everything above, I'm not sure what more to do.

DatabricksPS v1.11.0.8

[chlsmith]

I'm no expert, but some more things I've found.

I took the response from the -Verbose output and put it into a json text file. Until I removed all the backslashes () and the "'s around the whole string, I couldn't seem to do anything with the text. Once I did, though, I could read the json perfectly well in VS Code and the Prettify Json extension worked on it, too.

Again, this might not mean squat, but I'm trying to help as much as I can. :)

[gbrueckl]

well, thats weird Get-DatabricksCluster basically returns a list/array of cluster objects so we basically take the JSON response, convert it to a PowerShell object and return the clusters-array

[chlsmith]

i'm looking more using Postman. I'm getting the right results.

When I run the same through "invoke-restapi", I get data that looks right, but then when ConvertTo-Json happens, it doesn't come back in a usable format. so $result. Clusters doesn't work

[gbrueckl]

can you try to run Get-DatabricksCluster | ConvertTo-Json

[gbrueckl]

if you invoke it via Invoke-RestAPI you probably have to run ConvertFrom-Json to get a powershell object and use .clusters

[chlsmith]

i'm looking more using Postman. I'm getting the right results. However, when I run the same through "invoke-restapi", I get data, but then when ConvertTo-Json happens, it doesn't come back in a usable format.

"get-databrickscluster | convertto-json" comes back null.

[chlsmith]

I had to use "ConvertFrom-Json -AsHashtable" directly on the response to get a workable object. Once I did, I could get .clusters and everything underneath it.

[gbrueckl]

so you are saying that this code is not working for you:

Set-DatabricksEnvironment ...
$x = Get-DatabricksCluster 
$x.Count
$x | ConvertTo-Json

[chlsmith]

That is correct. No error message, though.

 run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.

❯ $clusters = Get-DatabricksCluster
❯ $clusters
❯ $clusters.count
0
❯ $clusters | ConvertTo-Json
null
~\local\repos % 

However, running Get-DatabricksCluster -Verbose shows the data in there:

Name                           Value
----                           -----
Authorization                  Bearer blahblahbla…Content-Type                   application/json

VERBOSE: Body: 

VERBOSE: HTTP/1.1 GET with 0-byte payload
VERBOSE: received -byte response of content type application/json
VERBOSE: Content encoding: utf-8
VERBOSE: Response: "{\"clusters\":[{\"cluster_id\":\"0902-184836-6lg1<redacted>

[gbrueckl]

Do the other Get-DatabricksXXX cmdlets work? Whats your powershell version? Where is Databricks hosted? Azure, AWS, GCP?

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Chris Smith @.> Sent: Tuesday, February 28, 2023 4:57:39 PM To: gbrueckl/Databricks.API.PowerShell @.> Cc: Gerhard Brueckl @.>; Comment @.> Subject: Re: [gbrueckl/Databricks.API.PowerShell] get-databrickscluster not returning anything (Issue #79)

That is correct. No error message, though.

run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.

❯ $clusters = Get-DatabricksCluster

❯ $clusters

❯ $clusters.count

0

❯ $clusters | ConvertTo-Json

null

~\local\repos %

However, running Get-DatabricksCluster -Verbose shows the data in there:

Name Value

---

Authorization Bearer blahblahbla…Content-Type application/json

VERBOSE: Body:

VERBOSE: HTTP/1.1 GET with 0-byte payload

VERBOSE: received -byte response of content type application/json

VERBOSE: Content encoding: utf-8

VERBOSE: Response: "{\"clusters\":[{\"cluster_id\":\"0902-184836-6lg1

— Reply to this email directly, view it on GitHubhttps://github.com/gbrueckl/Databricks.API.PowerShell/issues/79#issuecomment-1448430735, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC2VO4TG3Z6CXFZLMRD7QPDWZYN7HANCNFSM6AAAAAAVKUVR7U. You are receiving this because you commented.Message ID: @.***>

[chlsmith]

Interestingly, get-databricksjobs, get-databricksnodetype, etc, all appear to work.

Databricks hosted in Azure.

PSVersion                      7.3.2
PSEdition                      Core

[gbrueckl]

How many clusters do you have?

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Chris Smith @.> Sent: Tuesday, February 28, 2023 6:12:14 PM To: gbrueckl/Databricks.API.PowerShell @.> Cc: Gerhard Brueckl @.>; Comment @.> Subject: Re: [gbrueckl/Databricks.API.PowerShell] get-databrickscluster not returning anything (Issue #79)

Interestingly, get-databricksjobs, get-databricksnodetype, etc, all appear to work.

Databricks hosted in Azure.

PSVersion 7.3.2 PSEdition Core

— Reply to this email directly, view it on GitHubhttps://github.com/gbrueckl/Databricks.API.PowerShell/issues/79#issuecomment-1448552793, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC2VO4QL46QRHJEMK7SUU6DWZYWW5ANCNFSM6AAAAAAVKUVR7U. You are receiving this because you commented.Message ID: @.***>

[chlsmith]

Passing a cluster id also works, so the GET endpoint is okay. It's only the LIST one that seems to be bothering me.

7 "all-purpose compute" clusters. approx 30 job clusters

[chlsmith]

Is it possible that the info is coming back in multiple pages or something? Even when I run a curl and get the info, if I just copy/paste the output into a new json file, there's a return in the middle that makes it look like the json is badly formatted.

[chlsmith]

there IS an unusual return in the output! it's right in the middle of a message that says "Inactive cluster terminated (inactive for 70 minutes)" I get the same return from curl and it includes that return

[gbrueckl]

Can ypu share that part of the output? Maybe some invlid character?

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Chris Smith @.> Sent: Tuesday, February 28, 2023 6:42:14 PM To: gbrueckl/Databricks.API.PowerShell @.> Cc: Gerhard Brueckl @.>; Comment @.> Subject: Re: [gbrueckl/Databricks.API.PowerShell] get-databrickscluster not returning anything (Issue #79)

Is it possible that the info is coming back in multiple pages or something? Even when I run a curl and get the info, if I just copy/paste the output into a new json file, there's a return in the middle that makes it look like the json is badly formatted.

— Reply to this email directly, view it on GitHubhttps://github.com/gbrueckl/Databricks.API.PowerShell/issues/79#issuecomment-1448599725, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC2VO4SWIKHIWOKEOJD427LWZY2HNANCNFSM6AAAAAAVKUVR7U. You are receiving this because you commented.Message ID: @.***>

[chlsmith]

Hmm. I'm worried that there may be info within the output that the company won't want to be made public, so I'm shy about that.

It appears to just be

"Inactive cluster terminated (inactive for 
70 minutes).

but there are other instances of the same message within the remainder of the data that are not this way.

[chlsmith]

I just did it with invoke-webrequest instead and am getting the same. I think it's this list api that's causing the problem.

[chlsmith]

The only thing I see that works is to add "-AsHashTable" to the json conversion. That seems to work every time.

[gbrueckl]

if you run Get-DatabricksCluster -Debug, it will output the native PS command that is run in the background - it will look like this you can copy & paste that code and run it manually - if this is not working, please post the error message that you get

[chlsmith]

That is working as expected. The cluster data is there.

[gbrueckl]

is the call particularly slow? could it run into a timeout? I use a timeout of 60 seconds, but it should usually only take a couple of seconds

[chlsmith]

Hmm. I don't see it as particularly slow at all. Neither using just "invoke-restmethod" or using Postman. Response is complete.

[gbrueckl]

was just a guess as I am running out of ideas :D

[chlsmith]

haha! me too!

Locally, I did change the convertfrom-json to be:

        $result = $response | ConvertFrom-Json -AsHashtable -Depth 50

That converted it as I expected it to and I did a depth of 50 just to make sure everything was split out right. I still suspect it's something specific to the Clusters API. Who supports that? Microsoft or Databricks?

[gbrueckl]

Databricks The APIs are the same across all cloud vendors But they will usually tell you to open a support ticket at Microsoft first Do you know the root cause yet? Is it related to this malformed json?

Btw, did you also try $respone.clusters directly as thats what i use. Where $response is the result of Invoke-RestMethod

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Chris Smith @.> Sent: Wednesday, March 1, 2023 4:30:32 PM To: gbrueckl/Databricks.API.PowerShell @.> Cc: Gerhard Brueckl @.>; Comment @.> Subject: Re: [gbrueckl/Databricks.API.PowerShell] get-databrickscluster not returning anything (Issue #79)

haha! me too!

Locally, I did change the convertfrom-json to be:

    $result = $response | ConvertFrom-Json -AsHashtable -Depth 50

That converted it as I expected it to and I did a depth of 50 just to make sure everything was split out right. I still suspect it's something specific to the Clusters API. Who supports that? Microsoft or Databricks?

— Reply to this email directly, view it on GitHubhttps://github.com/gbrueckl/Databricks.API.PowerShell/issues/79#issuecomment-1450344199, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC2VO4QZJJFY6FORSJTTYUDWZ5TRRANCNFSM6AAAAAAVKUVR7U. You are receiving this because you commented.Message ID: @.***>

[chlsmith]

The response is null from Get-DatabricksClusters. It's strange as can be.

If I run "$clusters=Get-DatabricksCluster -verbose

The verbose output shows all the response data I would expect.

If I then just try to output "$clusters", that object is null.

So the response is coming back from the API, but something is happening after that within the code that is not generating a proper PS Object.

[chlsmith]

OMG. I'm pulling my hair out for I think something specific to this individual workspace. I have 5 workspaces, and get-databrickscluster works for the others.

Feel free to consider this issue resolved, but remember it down the road if others run across similar problems.

[chlsmith]

Hmm. The cluster with the problem has periods in the cluster_names. Could that be throwing the conversion off?

[gbrueckl]

I will try on my end tomorrow. Though, this should not break a JSON (de)serializer

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Chris Smith @.> Sent: Wednesday, March 1, 2023 8:49:35 PM To: gbrueckl/Databricks.API.PowerShell @.> Cc: Gerhard Brueckl @.>; Comment @.> Subject: Re: [gbrueckl/Databricks.API.PowerShell] get-databrickscluster not returning anything (Issue #79)

Hmm. The cluster with the problem has periods in the cluster_names. Could that be throwing the conversion off?

— Reply to this email directly, view it on GitHubhttps://github.com/gbrueckl/Databricks.API.PowerShell/issues/79#issuecomment-1450757168, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC2VO4Q6TQACZIUMAAVZK6DWZ6R47ANCNFSM6AAAAAAVKUVR7U. You are receiving this because you commented.Message ID: @.***>

[chlsmith]

that's the only thing I can think of that's different with this workspace. I have completely worked around it at this point, but it's something you definitely need to be aware of.

[gbrueckl]

just tried with a new cluster called a.b in my workspace but the cmdlet is still working is there anything else specific about your clusters? but its super weird

from a code point of view, this is what is creating the verbose output from the generic API call Invoke-DatabricksApiRequest:

$result = Invoke-RestMethod -Uri $apiUrl -Method $Method -Headers $headers -Body $Body -TimeoutSec $TimeoutSec
Write-Verbose "Response: $($result | ConvertTo-Json -Depth 10)"

return $result

this part then runs in the cluster cmdlet:

$result = Invoke-DatabricksApiRequest -Method $requestMethod -EndPoint $apiEndpoint -Body $parameters

    if ($ClusterID) {
      # if a ClusterID was specified, we return the result as it is
      return $result
    }
    else {
      # if no ClusterID was specified, we return the clusters as an array
      $clusters = $result.clusters

      if (-not $IncludeJobClusters) {
        $clusters = $clusters | Where-Object { $_.cluster_source -ne "JOB" }
      }

      return $clusters
    }

could you try the -IncludeJobClusters flag?

[chlsmith]

-IncludeJobClusters still didn't get it to return anything.

I think this is something very specific to this workspace. I have multiple others that have worked just fine. So strange.

[chlsmith]

I know this isn't "fixed", but it's a single case where I could see the issue, and I made a workaround.

[gbrueckl]

I would still like to figure out what the root cause of this problem was. Did you run any further tests?

Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Chris Smith @.> Sent: Tuesday, March 7, 2023 7:52:12 PM To: gbrueckl/Databricks.API.PowerShell @.> Cc: Gerhard Brueckl @.>; Comment @.> Subject: Re: [gbrueckl/Databricks.API.PowerShell] get-databrickscluster not returning anything (Issue #79)

I know this isn't "fixed", but it's a single case where I could see the issue, and I made a workaround.

— Reply to this email directly, view it on GitHubhttps://github.com/gbrueckl/Databricks.API.PowerShell/issues/79#issuecomment-1458664177, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AC2VO4TD4GWUAL7OTL3GG3LW257VZANCNFSM6AAAAAAVKUVR7U. You are receiving this because you commented.Message ID: @.***>