search

databricks / cli

Databricks CLI
Other
133 stars 51 forks source link

development mode and compute id #1488

Open dinjazelena opened 3 months ago

dinjazelena commented 3 months ago

Hey, when we set development mode and compute_id to override all of the job's compute to given all purpose cluster which is great for fast testing. But problem is that whenever we deploy our whl artifact which tasks depends on, and every time we deploy we install whl file on global cluster level library, so each deployment requires Uninstall and Restart of cluster in order to install whl for new deployment. Why it cant install it in separate context? It happens even if u provide in library pypi for example pandas 2.0.0, installs it on global level, so in next deployment if u wanna try 2.0.1, u cant, its gona say 2.0.0 already exists. It kills purpose of development mode and compute_id..

andrewnester commented 3 months ago

This is indeed a behaviour right now which is caused by limitation on cluster side itself. We're looking at ways to improve this behaviour for all purpose clusters. I'll keep this thread updated

tritrang2 commented 1 week ago

+1 for this enhancement.