Add isort for import cells

[javiruiz]

I've been using blackbricks extensively with my remote Databricks notebooks, and I would like to add isort so it can also reformat cells with imports. Any hints about how to proceed?

[bsamseth]

I've been thinking about this myself, as it would be nice to add. Currently, I make sure to run isort on the notebooks when they are stored locally, and that works fine because the notebooks are just plain Python files in isort's mind. However, this doesn't address running isort on remote notebooks.

The main reason why it isn't already part of blackbricks is that I'm thinking about all other types of linters as well, and adding them one by one doesn't seem like a good idea. Instead I'm thinking about an extension system, so that you can plug in any linter/formatter you want and just let blackbricks handle the orchestration and remote notebook connection. Not 100% sure how to best do that at this time though. Happy to take suggestions.

A temporary solution is to sync the notebook locally, either with git or dbx, running isort, and then syncing it back to Databricks.

[javiruiz]

Thanks for your thorough response! It would be nice to have that extension system. Let's see if we come up with an idea for this. In the meantime, I'll use your suggestion of syncing the notebook locally.

[NodeJSmith]

@bsamseth While writing a different comment I started looking at the flake8 plugin architecture. Is this the kind of system you had in mind? https://flake8.pycqa.org/en/latest/plugin-development/registering-plugins.html

[bsamseth]

@NodeJSmith Yes, it is. I was looking at pytest, but it's using the same entry_points system. It would just define the entry point in pyproject.toml instead of setup.py, but otherwise similar.

The idea would then be that blackbricks would be refactored to only

• Read file, local or remote (as today)
• Call all registered plugins on the contents.
• Normalize notebook whitespace (e.g. whitespace around cells, whitespace in MAGIC blocks) (as today)
• Compute diff / check diff / write back file, depending on what mode is used (as today)
• Handle output back to the console (as today)

A few outstanding questions before starting any implementation though:

• Should it call the plugins on the entire file or cell by cell? I.e. black could do all in one, while sqlparse would need to only look at the SQL cells. Could provide two hooks, process_file and process_cell and let the plugin decide. That's my initial thought, anyway.
• How to determine the order of the plugins? I.e. if a flake8 plugin was made, it should run after black, as it would otherwise complain about formatting that black would later fix. Perhaps a config option, with a default set by a sensible ordering of all known plugins, and any unknown plugins ordered called last.
• Should something be included by default? Probably yes, at least black. It could still be written as a "plugin", but would be included in the main repo and installed always.
• What about config options? Should avoid copying config options for all tools into blackbricks, and rather depend on each tool's supported config formats. But not everything has a config file supported (e.g. sqlparse), in which case maybe it should inject command line flags.

Happy to hear some thoughts :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.