paiqo / Databricks-VSCode

VSCode extension to work with Databricks
GNU General Public License v3.0
124 stars 27 forks source link

Proper repo workflow #191

Open twsl opened 5 months ago

twsl commented 5 months ago

Let's assume i have python project git repo locally where i developed a library and have some notebooks to execute said code. how can i properly sync my code between my local folder and the specific repo user folder on databricks? The reason i need full sync support is because i want to install the library from within the repo so i can import it into the notebooks.

The original databricks extension lets me sync my code to a folder in the user directory, but a feature to properly sync to the repo folder seems to be missing from both folders.

gbrueckl commented 4 months ago

please check Execution modes

if you are in offline-mode you can download/sync a whole Repo-folder

when referencing a library from your Repo, it will always use the deployed version of the library however, you can also change your library locally and upload it

twsl commented 4 months ago

I looked through the execution modes, but they don't seem to be suited to my needs. The live execution mode doesn't work, because of databricks shitty git limitations (e.g. git repos don't support git submodules), the offline execution doesnt offer support for full repo sync and doesnt allow me to set the sync location for the whole vscode workspace.

gbrueckl commented 2 months ago

what exactly do you mean by "full repo sync"? you can sync everything that you have local to the Databricksworkspace