search

databricks / databricks-cli

(Legacy) Command Line Interface for Databricks
Other
386 stars 234 forks source link

databricks workspace import(_dir) should not automatically convert non-notebook .py files to notebooks #618

Open cloudyday opened 1 year ago

cloudyday commented 1 year ago

On the Databricks platform, it is possible to use regular python modules in Repos and Workspaces. It is even suggested by Databricks to use typical software engineering best practises. Here and here.

However, when using the Databricks CLI for importing files, especially a directory, into a Workspace by using databricks workspace import_dir, all .py files get automatically transformed to notebooks. This makes them incompatible because they cannot be imported as modules anymore.

The import and import_dir commands should probably only convert .py files starting with # Databricks notebook source to notebooks.

fferegrino commented 1 year ago

Glad to know I am not the only one, I opened this issue before realising you had already raised this concern.

https://github.com/databricks/databricks-cli/issues/630

osuleiman commented 1 year ago

I've just created a PR to help tackle this problem.

643