Databricks assets bundle failing to deploy from WSL

tarique-msci commented 2 months ago

Issue

The databricks bundle assets is failing to deploy from WSL whereas deployment from native windows is successful. The machine is connected to a firm wide VPN which allows connection to the Databricks workspace. As far as I have tested other databricks cli commands like fs, clusters etc and even the databricks bundle validation is also working. Is there something with the WSL networking that might be causing this?

Configuration

The Databricks CLI version is v0.226.0

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

Run `databricks bundle deploy .' in WSL when windows is connected to a VPN

Expected Behavior

The bundle should be deployable through wsl as well.

Actual Behavior

Getting stuck on the deploy command and then erroring out.

OS and CLI version

OS: Ubuntu 22.04 on WSL 2

Debug Logs

The error from debug logging:

<   "error_code": "RESOURCE_EXHAUSTED",
<   "message": "Too many requests. Please wait a moment and try again. If the issue persists, consider adjusting... (66 more bytes)"
< } pid=278699 mutator=seq mutator=deploy mutator=seq mutator=seq mutator=deferred mutator=seq mutator=files.Upload sdk=true
10:52:20 DEBUG POST /api/2.0/workspace/delete

michaelschuett-tomtom commented 2 months ago

It just looks like you are hitting a rate limit and should have nothing to do with WSL.

tarique-msci commented 2 months ago

It just looks like you are hitting a rate limit and should have nothing to do with WSL.

If I deploy from Windows during the same time it does not give me any rate limiting error. Also is there a way to control requests rate from the CLI?

otrofimov commented 2 months ago

@tarique-msci I'm deploying from WSL when my host windows system is connected to VPN. No issues. Deployed using all the CLI versions starting v0.221

benedikt-mayer commented 2 months ago

We got this issue when the databricks CLI was trying to remove a lot of existing files which were not supposed to be synced anymore. The CLI then calls the API once per file to be removed, which leads to the "too many requests". What helped was deleting the no-longer-synced files manually from the databricks UI once. All within WSL.

andrewnester commented 2 months ago

It's indeed not related to DABs itself but most likely related to your project or system setup

tarique-msci commented 2 months ago

It's indeed not related to DABs itself but most likely related to your project or system setup

Clearing the .databricks local directory and bundle uploads on the Databricks workspace fixed this issue. As @benedikt-mayer said the CLI was trying to remove a lot of the existing files which were not supposed to be synced anymore. So I think this is something that should be handled by the databricks cli and is not an issue of the local system setup or project and should be an issue that has to be fixed in databricks cli.

pietern commented 2 months ago

The underlying reason you describe, where the deploy would try and delete many files that were previously synchronized into the workspace, sounds like the CLI was working as intended.

What's not working as intended is that the rate limit was hit and that it was unable to successfully retry. This could have been caused by different rate limits being applied at the user level, workspace level, and IP address level. If you observed this issue while connected to a VPN, it is possible that you inadvertently were sharing the outbound IP with others, and hitting the rate limits sooner than when not connected to a VPN.

databricks / cli