search

databricks / databricks-sdk-go

Databricks SDK for Go
https://docs.databricks.com/dev-tools/sdk-go.html
Apache License 2.0
51 stars 42 forks source link

[ISSUE] mws network connectivity config issue : invalid character '<' looking for beginning of value #1048

Closed trivedirishabh closed 2 months ago

trivedirishabh commented 2 months ago

Description We are trying to use terraform databricks_mws_network_connectivity_config resource in Azure to create ncc. It worked fine while creating it but the plan fails now with invalid character error. I have also tried to run python API /api/2.0/accounts/{account_id}/network-connectivity-configs to test it and it fails as well.

Reproduction

Terraform databricks provider version : 1.45.0, 1.46.0, 1.50.0, 1.52.0

terraform { required_providers { databricks = { source = "databricks/databricks" } } } resource "databricks_mws_network_connectivity_config" "ncc" { name = var.ncc_name region = var.ncc_region } resource "databricks_mws_ncc_binding" "ncc_binding" { for_each = toset(var.databricks_workspace_id) network_connectivity_config_id = databricks_mws_network_connectivity_config.ncc.network_connectivity_config_id workspace_id = each.key }

provider.tf provider "databricks" { alias = "account-console" azure_workspace_resource_id = var.admin_workspace_id azure_use_msi = true host = var.account_console_url account_id = var.account_id rate_limit = 5 }

Python import requests endpoint="https://accounts.azuredatabricks.net/api/2.0/accounts/{account_id}/network-connectivity-configs" token="" response=requests.get(endpoint, headers={"Authorization":f"Bearer {token}","Content-Type":"application/json"}) if response.status_code==200: print(response.text) else: print(response.text)

Expected behavior It should work seamlessly.

Is it a regression? Did this work in a previous version of the SDK? If so, which versions did you try?

Debug Logs The SDK logs helpful debugging information when debug logging is enabled. Set the log level to Trace by configuring the default logger to log at trace (for example: add logger.DefaultLogger = &logger.SimpleLogger{Level: logger.LevelTrace} to your program), and include the logs here.

Other Information

Additional context Python API ERROR <!doctype html>

Databricks - Sign in

TERRAFORM ERROR Error: cannot read mws network connectivity config: unexpected error handling request: invalid character '<' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues.

tirendra-reddy commented 2 months ago

Hi, I am also seeing this issue when using the Databricks CLI v0.224.0. Command : databricks account networks list

xjomu commented 2 months ago

I also get the same error:

│ Error: cannot read mws network connectivity config: unexpected error handling request: invalid character '<' looking for beginning of value. This is likely a bug in the Databricks SDK for Go or the underlying REST API. Please report this issue with the following debugging information to the SDK issue tracker at https://github.com/databricks/databricks-sdk-go/issues. Request log:

│ > * Referer: https://accounts.azuredatabricks.net/api/2.0/accounts/485aad67-3fb0-4276-a383-4b9524de5549/network-connectivity-configs/844aa2a7-0e4a-4093-a7a5-924385afd4d7?
│ > * Traceparent: 00-376907ec361a9aa7a0de6eb09a498224-992ea0d5a004a815-01
│ > * User-Agent: databricks-tf-provider/1.47.0 databricks-sdk-go/0.41.0 go/1.21.10 os/linux terraform/1.5.5 resource/mws_network_connectivity_config auth/azure-client-secret cicd/azure-devops
│ > * X-Databricks-Azure-Sp-Management-Token: ***
│ < HTTP/2.0 200 OK
│ < * Cache-Control: no-cache, no-store, must-revalidate
│ < * Content-Type: text/html; charset=utf-8
│ < * Date: Thu, 26 Sep 2024 12:28:33 GMT
│ < * Server: databricks
│ < * Set-Cookie: enable-armeria-server-for-ui-flags=false; Max-Age=1800; Expires=Thu, 26 Sep 2024 12:58:33 GMT; Secure; HTTPOnly; SameSite=Strictenable-armeria-workspace-server-for-ui-flags=false; Max-Age=1800; Expires=Thu, 26 Sep 2024 12:58:33 GMT; Secure; HTTPOnly; SameSite=Strictx-serve-assets-from-prpreview=; Max-Age=0; Expires=Thu, 26 Sep 2024 12:28:33 GMT; Path=/; Secure; HTTPOnly; SameSite=Strictx-user-has-acknowledged-prpreview=; Max-Age=0; Expires=Thu, 26 Sep 2024 12:28:33 GMT; Path=/; Secure; HTTPOnly; SameSite=Strict
│ < * Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
│ < * X-Content-Type-Options: nosniff
│ < * X-Request-Id: 50954312-7a26-47ec-a7e6-378c82069fba
│ < * X-Ui-Svc: true
│ < <!doctype html>
│ < <html lang="en">
│ <  <head>
│ <   <meta charset="utf-8">
│ <   <meta name="viewport" content="width=device-width,initial-scale=1">
│ <   <meta name="theme-color" content="#000000">
│ <   <meta name="description" content="Databricks Sign in">
│ <   <title>Databricks - Sign in</title>
│ <   <script>window.__DATABRICKS_CONFIG__={"isCuttingEdge":false,"publicPath":{"accounts-console":"[https://databricks-ui-assets.azureedge.net/"}}</script>](https://databricks-ui-assets.azureedge.net/%22%7D%7D%3C/script%3E)
│ <   <link rel="icon" href="[https://databricks-ui-assets.azureedge.net/favicon.ico">](https://databricks-ui-assets.azureedge.net/favicon.ico%22%3E)
│ <   <script defer src="[https://databricks-ui-assets.azureedge.net/static/js/7406.f587a9be.js"></script>](https://databricks-ui-assets.azureedge.net/static/js/7406.f587a9be.js%22%3E%3C/script%3E)
│ <   <script defer src="[https://databricks-ui-assets.azureedge.net/static/js/6864.01f9e23a.js"></script>](https://databricks-ui-assets.azureedge.net/static/js/6864.01f9e23a.js%22%3E%3C/script%3E)
│ <   <script defer src="[https://databricks-ui-assets.azureedge.net/static/js/8240.f4023493.js"></script>](https://databricks-ui-assets.azureedge.net/static/js/8240.f4023493.js%22%3E%3C/script%3E)
│ <   <script>
│ <   function setNoCdnAndReload() {
│ <       document.cookie = `x-databricks-cdn-inaccessible=true; path=/; max-age=86400`;
│ <       const metric = 'cdnFallbackOccurred';
│ <       const browserUserAgent = navigator.userAgent;
│ <       const browserTabId = window.browserTabId;
│ <       const performanceEntry = performance.getEntriesByType('resource').filter(e => e.initiatorType === 'script').slice(-1)[0]
│ <       sessionStorage.setItem('databricks-cdn-fallback-telemetry-key', JSON.stringify({ tags: { browserUserAgent, browserTabId }, performanceEntry}));
│ <       window.location.reload();
│ <   }
│ < </script>
│ <   <script>
│ <   // Set a manual timeout for dropped packets to CDN
│ <   function loadScriptWithTimeout(src, timeout) {
│ <      return new Promise((resolve, reject) => {
│ <         const script = document.createElement('script');
│ <           script.defer = true;
│ <           script.src = src;
│ <           script.onload = resolve;
│ <           script.onerror = reject;
│ <           document.head.appendChild(script);
│ <           setTimeout(() => {
│ <               reject(new Error('Script load timeout'));
│ <           }, timeout);
│ <       });
│ <   }
│ <   loadScriptWithTimeout('https://databricks-ui-assets.azureedge.net/static/js/login.d62c1721.js', 10000).catch(setNoCdnAndReload);
│ < </script>
│ <   <link href="[https://databricks-ui-assets.azureedge.net/static/css/7406.31dcf9da.css"](https://databricks-ui-assets.azureedge.net/static/css/7406.31dcf9da.css%22) rel="stylesheet">
│ <   <link href="[https://databricks-ui-assets.azureedge.net/static/css/8240.dfea0ccd.css"](https://databricks-ui-assets.azureedge.net/static/css/8240.dfea0ccd.css%22) rel="stylesheet">
│ <   <link href="[https://databricks-ui-assets.azureedge.net/static/css/login.cf7f0b55.css"](https://databricks-ui-assets.azureedge.net/static/css/login.cf7f0b55.css%22) rel="stylesheet">
│ <  </head>
│ <  <body>
│ <   <noscript>
│ <    You need to enable JavaScript to run this app.
│ <   </noscript>
│ <   <div id="login"></div>
│ <   <script>const telemetryEndpoint="/telemetry-unauth?t=",uiModuleName="accountsConsoleLogin";function shouldIgnoreError(e){return!1}function generateUuidV4(){const e=window.crypto?.randomUUID?.();return e||"xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g,(e=>{const n=16*Math.random()|0;return("x"===e?n:3&n|8).toString(16)}))}function networkConnectivityTags(){const e=window.navigator.onLine,n=window.navigator.connection?.rtt??-1,t=window.navigator.connection?.downlink??-1;return{browserNavigatorOnline:e,browserConnectionEstimatedRtt:n,browserConnectionEstimatedDownlink:t,browserConnected:e&&n>0&&t>0}}function getParams(e){let n;const t=/\+/g,o=/([^&=]+)=?([^&]*)/g,r=function(e){return decodeURIComponent(e.replace(t," "))},i={};for(n=o.exec(e);n;)i[r(n[1])]=r(n[2]),n=o.exec(e);return i}function getWorkspaceParam(){const e=getParams(window.location.search.substr(1)),n=/^\d+/.exec(e?.o??"");return null===n?void 0:n[0]}function createTelemetryRequestBody(e,n={},t=null){const o=Math.round(Date.now()/1e3),r={eventId:generateUuidV4(),metric:e,tags:{...n,...networkConnectivityTags(),browserTabId:window.browserTabId,browserUserAgent:navigator.userAgent},ts:o};return t&&(r.blob=t),JSON.stringify({uploadTime:o,items:[JSON.stringify(r)]})}function recordTelemetry(e,n={},t=""){const o=getWorkspaceParam(),r=o?{"Content-Type":"application/json","X-Databricks-Org-Id":o.toString()}:{"Content-Type":"application/json"},i={method:"POST",credentials:"include",body:createTelemetryRequestBody(e,n,t),headers:r};fetch(telemetryEndpoint+Date.now(),i)}window.__databricks_networkConnectivityTags=networkConnectivityTags,Object.defineProperty(window,"browserTabId",{value:generateUuidV4()}),window.recordTelemetry=recordTelemetry,recordTelemetry("uiInit",{uiModule:uiModuleName,eventId:"init",eventClientSource:uiModuleName,eventType:"init"});let logCount=0;function error_handler(e,n,t,o,r){logCount++>4||shouldIgnoreError(e)||recordTelemetry("uncaughtJsException",{eventType:"jsExceptionV3",jsExceptionMessage:e,jsExceptionSource:n,jsExceptionLineno:t,jsExceptionColno:o,jsExceptionBeforeInit:!0},r&&r.stack&&r.stack.toString())}function sendBeaconOnPageExit(e){if(navigator.sendBeacon){const n=e&&e.type||"unknown",t=(Math.round(Date.now()/1e3),createTelemetryRequestBody("uiInit",{eventType:"pageExitBeforeAppInitComplete",eventName:n,eventClientSource:uiModuleName}));navigator.sendBeacon(telemetryEndpoint+Date.now(),t)}}window.onerror=error_handler,window.onunhandledrejection=function(e){error_handler(String(e.reason),null,null,null,e.reason)},window.addEventListener("beforeunload",sendBeaconOnPageExit),window.addEventListener("unload",sendBeaconOnPageExit),window.addEventListener("pagehide",sendBeaconOnPageExit),window.cleanupAfterAppInit=()=>{window.removeEventListener("beforeunload",sendBeaconOnPageExit),window.removeEventListener("unload",sendBeaconOnPageExit),window.removeEventListener("pagehide",sendBeaconOnPageExit)}</script>
│ <  </body>
│ < </html>
slamerinline8 commented 2 months ago

We are also facing the same issue, with databricks provider version 1.52.0 It worked fine till yesterday (25/09/2024 around 18:00 CET)

I went back to 1.50.0 version but nothing changed.

jonathandbyrd commented 2 months ago

also getting the same error

databricks terraform provider version = 1.49

when we switch to 1.52 it gives us azure ad login errors instead of the mws ncc config issue

MyroslavaTarcha commented 2 months ago

Same issue, version = "1.49.1", yesterday was working fine

sdziadkowiec commented 2 months ago

Same here. Terraform and Python SDK using service principal app id & client secret, curl with AAD token, all failing. Other Account APIs are working just fine. Seems to be an authentication issue in the API - the error is caused by the initial call being redirected to the login screen with http code 303. I'm on West Europe if that matters.

Powershell code to reproduce:

az login

$databricksResource = "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"
$databricksToken = az account get-access-token --resource $databricksResource --query "accessToken" --output tsv

$apiUrl = "https://accounts.azuredatabricks.net/api/2.0/accounts/<DATABRICKS_ACCOUNT_ID>/network-connectivity-configs"
#$apiUrl = "https://accounts.azuredatabricks.net/api/2.0/accounts/<DATABRICKS_ACCOUNT_ID>/scim/v2/Groups" # this one works just fine

$response = Invoke-RestMethod -Uri $apiUrl -Headers @{ 
    Authorization = "Bearer $databricksToken"; 
    Accept = "application/json" ;
} -Method Get

$response

Also for reference, debug logs from Python SDK: ncc_error_python_sdk3

sdziadkowiec commented 2 months ago

To make things more interesting, the same API gets called in Account Console and responds fine... image

MyroslavaTarcha commented 2 months ago

Seems to be fixed, works now for me

xjomu commented 2 months ago

Yes, it works for me as well. Thank you and well done!

jonathandbyrd commented 2 months ago

yep, working for me as of 330 pm EST yesterday, thanks

trivedirishabh commented 2 months ago

I am closing this issue as I got an update from databricks that it is fixed. There was a recent code change which has caused this and they rolled back changes.

Thank you everyone for your inputs.