search

paiqo / Databricks-VSCode

VSCode extension to work with Databricks
GNU General Public License v3.0
123 stars 27 forks source link

Setting HTTP_PROXY, HTTPS_PROXY and SSLverification in the extentions #76

Closed surfii3z closed 2 years ago

surfii3z commented 2 years ago

Hi,

Thanks for the awesome project.

Do you have any recommended ways to input HTTP_PROXY, HTTPS_PROXY and SSLVerification (similar to insecure=True in databricks-cli source codes) in the extention?

I am constrained under a corporate's VPN and the only way for my VScode to have the internet connection is to going through some proxy address.

Thanks

gbrueckl commented 2 years ago

please check this old issue: https://github.com/paiqo/Databricks-VSCode/issues/29 - something similar had been discussed there already

you can try to set the VSCode setting http.proxySupport, maybe also http.proxyStrictSSL

surfii3z commented 2 years ago

@gbrueckl thanks for your help.

Actually both situations are totally different.

If I'm not wrong, that issue was solved by disabling the proxy.

However, in my case, I would need to go through the company proxy to connect to Databricks REST API.

gbrueckl commented 2 years ago

if you enable http.proxySupport, the system proxy will also be used by this extension - did you try that?

surfii3z commented 2 years ago

@gbrueckl Yeah, I have tried the following setting but it did not work.

"http.proxy": "http://PROXY_ADDRESS:PROXY_PORT",
"http.proxySupport": "on",
"http.proxyStrictSSL": false,

By the way, my databricks clusters list command in the terminal would not work too.

databricks clusters list
>>> Error: ConnectionError: HTTPSConnectionPool(host='<Databricks Workspace URL>', port=443): Max retries exceeded with url: /api/2.0/clusters/list (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f2ef6305820>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

However, if I do the following, it works.

export http_proxy=http://PROXY_ADDRESS:PROXY_PORT
export http_proxy=http://PROXY_ADDRESS:PROXY_PORT

# append insecure = True to databricks CLI config
# REF: https://stackoverflow.com/a/65285795
echo "insecure = True" >> ~/.databrickscfg 

databricks clusters list
>>> <List of clusters>
gbrueckl commented 2 years ago

I made a temporary version for you which you can download here: https://gbpublic.blob.core.windows.net/files/databricks-vscode-0.9.4.vsix to make it work for your scenario you then need to set http.proxySupport = "on" and http.proxyStrictSSL= false in your VS Code settings

can you try this please and report back here

surfii3z commented 2 years ago

@gbrueckl Thanks for the solution.

I tried with the following setting


    "http.proxy": "http://PROXY_ADDRESS:PROXY_PORT",
    "http.proxyStrictSSL": false,
    "http.proxySupport": "on",
    "http.proxyAuthorization": null,

But it could not solve the problem.

The behavior is the same with the previous version. I clicked refreshing workspace but it hanged there.

gbrueckl commented 2 years ago

can you check and post the Output of the extension generated on your machine? image

surfii3z commented 2 years ago

I didn't know I could do that haha.

Logger initialized!
Trying to get config from Workspace settings ...
No Workspace settings found! Trying to get User-Settings instead ...
Initializing ConnectionManager CLI ...
Configuration DEFAULT: Property "localSyncSubfolders" was not provided - using the default value!
Configuration DEFAULT: Property "exportFormats" was not provided - using the default value!
Configuration DEFAULT: Property "useCodeCells" was not provided - using the default value "true"!
Setting 'databricks.lastActiveConnection' to 'DEFAULT' ...
Storing Personal Access Token in System KeyChain 'DEFAULT-API-Token' ...
Activating Databricks Connection 'DEFAULT' ...
Initializing Databricks API Service ...
Strict Proxy SSL verification disabled due to setting "http.proxyStrictSSL": false !
Proxy support is enabled due to setting "http.proxySupport": "on"!
Testing new Databricks API (<Databricks Workspace URL>) settings ()...
GET 2.0/workspace/list
Params:{"params":{"path":"/"}}
ERROR: TypeError: Cannot read property 'data' of undefined
gbrueckl commented 2 years ago

are you sure it is actually related to the proxy? can you try it on a network that does not use/need a proxy?

surfii3z commented 2 years ago

If I leave the vscode proxy related settings as default values, and I disable my VPN (such that I won't be constrained by my company's proxy). It would work.

Logger initialized!
Trying to get config from Workspace settings ...
No Workspace settings found! Trying to get User-Settings instead ...
Initializing ConnectionManager CLI ...
Configuration DEFAULT: Property "localSyncSubfolders" was not provided - using the default value!
Configuration DEFAULT: Property "exportFormats" was not provided - using the default value!
Configuration DEFAULT: Property "useCodeCells" was not provided - using the default value "true"!
Setting 'databricks.lastActiveConnection' to 'DEFAULT' ...
Storing Personal Access Token in System KeyChain 'DEFAULT-API-Token' ...
Activating Databricks Connection 'DEFAULT' ...
Initializing Databricks API Service ...
Testing new Databricks API (<Databricks Workspace URL>) settings ()...
GET 2.0/workspace/list
Params:{"params":{"path":"/"}}
Response: 
{"objects":[{"object_type":"DIRECTORY","path":"/<Directory-Name>","object_id":<Object-id>},]}
Databricks API Service initialized!
Using cluster with id 'undefined' for SQL Browser!
Initializing ConnectionManager CLI ...
Configuration DEFAULT: Property "localSyncSubfolders" was not provided - using the default value!
Configuration DEFAULT: Property "exportFormats" was not provided - using the default value!
Configuration DEFAULT: Property "useCodeCells" was not provided - using the default value "true"!
Setting 'databricks.lastActiveConnection' to 'DEFAULT' ...
GET 2.0/workspace/list
Params:{"params":{"path":"/"}}
GET 2.0/clusters/list
Params:null
GET 2.1/jobs/list
Params:{"params":{"expand_tasks":true}}
GET 2.0/dbfs/list
Params:{"params":{"path":"/"}}
GET 2.0/secrets/scopes/list
Params:null
GET 2.0/repos
Params:{"params":{}}
Storing Personal Access Token in System KeyChain 'DEFAULT-API-Token' ...
Response:

I could download/ upload or view files in Databricks as the extension intended.

gbrueckl commented 2 years ago

ok then it is indeed very likely related to the proxy, just wanted to rule out other stuff first

it is a bit weird though that your output does show nothing related to the Proxy - are you sure you are using the VSIX I linked above?

surfii3z commented 2 years ago

@gbrueckl Yeah, I think so. What I did was to uninstall the previous version and then install the extension using your .visx file. The extension version installed is 0.9.4, so I think I am using what you provided. temp

gbrueckl commented 2 years ago

maybe you can try setting some environment variables as described here: https://levelup.gitconnected.com/how-to-resolve-certificate-errors-in-nodejs-app-involving-ssl-calls-781ce48daded

otherwise I am out of ideas on how I could solve this within the extension :(

surfii3z commented 2 years ago

Thanks for guide. However, I don't quite understand how I could set nodejs environment variable to the extension. Do I have to modify your source code? or do you have any suggestions?

gbrueckl commented 2 years ago

these are regular system environment variables, not specific to NodeJS you would set them in your environment (and/or persist them) and then start VSCode afterwards

gbrueckl commented 2 years ago

could you try the environment variable settings yet?

surfii3z commented 2 years ago

@gbrueckl Sorry for the slow response due to my travel, I will try and let you know by today.

surfii3z commented 2 years ago

So what I did was

export http_proxy=http://PROXY_ADDRESS:PROXY_PORT
export http_proxy=http://PROXY_ADDRESS:PROXY_PORT
export NODE_TLS_REJECT_UNAUTHORIZED=0

code # start VSCode

But it showed the same error...

Logger initialized!
Trying to get config from Workspace settings ...
No Workspace settings found! Trying to get User-Settings instead ...
Initializing ConnectionManager CLI ...
Configuration DEFAULT: Property "localSyncSubfolders" was not provided - using the default value!
Configuration DEFAULT: Property "exportFormats" was not provided - using the default value!
Configuration DEFAULT: Property "useCodeCells" was not provided - using the default value "true"!
Setting 'databricks.lastActiveConnection' to 'DEFAULT' ...
Storing Personal Access Token in System KeyChain 'DEFAULT-API-Token' ...
Activating Databricks Connection 'DEFAULT' ...
Initializing Databricks API Service ...
Strict Proxy SSL verification disabled due to setting "http.proxyStrictSSL": false !
Proxy support is enabled due to setting "http.proxySupport": "on"!
Testing new Databricks API (<Databricks Workspace URL>) settings ()...
GET 2.0/workspace/list
Params:{"params":{"path":"/"}}
ERROR: TypeError: Cannot read property 'data' of undefined

Anyway, I appreciate your help :)

thaiphv commented 2 years ago

Hi @surfii3z looks like you were able to resolve the issue yourself. Would you be able to show how you resolve it?

surfii3z commented 2 years ago

Hi @surfii3z looks like you were able to resolve the issue yourself. Would you be able to show how you resolve it?

I would say I gave up rather than solving the problem XD

ashitabh commented 2 years ago

running VS code behind company proxy needs the following command parameters code --proxy-server="http://user:password@myorgproxy.com:800/" It's working for me

yuanhao-li commented 2 years ago

I have the same problem here. Sad can't use this extension.

I saw databricks-vscode is using the axios as http client. There's an issue about https over http. Maybe it is related to this issue. Check it out: https://github.com/axios/axios/issues/3384

btw, azure plugins work well to me.

@gbrueckl

gbrueckl commented 2 years ago

the next version will change from Axios to Fetch API but I cannot say if this will fix the issue for you. This is especially hard for me to debug as I cannot easily reproduce the issue on my side as I do not have a proxy and even if I would, its configuration could be fundamentally different from yours.

in general, do other extensions that access resources on the web work in your case?

yuanhao-li commented 2 years ago

Hi thanks for the feedback. I know it is hard to you to reproduce the test environment. I appreciate your work here very much. Let me know if you need someone to test it.

And yes, the other extensions work well with the default proxy settings of VS Code.

gbrueckl commented 2 years ago

what other VSCode settings with regards to the proxy are you using?

yuanhao-li commented 2 years ago

what other VSCode settings with regards to the proxy are you using?

I don't have extra proxy settings in VS Code. It's been done in Network Connection Setting (Mac) and also as Environment Variables in the Terminal (http_proxy, https_proxy). VS Code just reads the settings from there.

VS Code proxy settings: http.proxySupport: "Override" http.proxyStrictSSL: "true" http.proxy: "" // empty

gbrueckl commented 2 years ago

@yuanhao-li can you please try this release and see if it works for you https://github.com/paiqo/Databricks-VSCode/releases/tag/v1.2.0

Note: it is not yet published to the VSCode gallery

yuanhao-li commented 2 years ago

@gbrueckl thanks for the update. I can confirm that the new version works out of box without custom settings in VS Code. Congrats.

PRkudupu commented 6 months ago

I am facing the same issue. Note : This happens while I am trying to connect to databricks via VPN in databricks cli. I tried setting HTTP_PROXY. I am getting the following error "'Unable to connect to proxy', NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x7f4c84932ad0>: Failed to resolve ' " I tried setting up "set HTTPS_PROXY=http://username:password@{proxy host}:{port}" Did not help

gbrueckl commented 6 months ago

the error message suggest that it cannot find the proxy or could not resolve its name - can you please check again if its configured properly?

are you using the environment variables or the VSCode settings to set the proxy?