Add --black option to nbconvert?

[cdeil]

I also think it would be great if there was built-in support to the black code formatter on Jupyter notebooks, i.e. reformat the input code cells.

In the project we work on, @bultako wrote some code (~ 50 lines) to do this using the black Python API, including some special processing to make the parsing of Jupyter / iPython specifics like % work, and to offer the task to format the notebook as a command line option here:

https://github.com/gammapy/gammapy/blob/089d552885256c560c3febdb4610b98b4e708bf0/gammapy/scripts/jupyter.py#L86

So Jupyter devs and @ambv as black creator - would either of you be willing to have built-in support for applying black on code in Jupyter notebooks? I.e. add this feature to jupyter or black directly, instead of a tiny third package for this task? I think that would be much nicer for users (see my similar comment in https://github.com/jupyter/nbconvert/issues/637#issuecomment-436191485).

[bollwyvl]

Really good idea! Black only works with python, though, so it's unlikely to be a good fit as a jupyter-level tool. It's also still advertised as beta, and is unapologetically py3 only, while nbconvert had some time left in 2/3 land.

What could be very useful in nbconvert is a general code formatter preprocessor, and flag alias, which showed an example that used, for python...

nbconvert --format-code=black

Or c++/js/Java/obj-c...

nbconvert --format-code=clang-format

Or anything else that accepted code on stdin and spat back out code on stdout. Heck, 2to3 would be keen.

Formatting code inside markdown is a mine field, so probably best left alone.

Finally, there could be cells that you didn't want to format for some reason. This could be handled by a formalized cell metadata tag... e.g. skip-format. This would likely be the quickest way to deal with non-standard syntax like magics.

Kernels that introduce non-standard syntax should probably bear the burden of providing the appropriate modifications to formatting tools, or should strive to have those included in upstreams. So yeah, either black --ipython, ipython format or iblack would have to exist for a full solution.

Getting a formatter into IPython is probably the most robust solution, as one could easily imagine an interactive %%black (or %%format black|yapf) being very useful, and new fancy syntax would land there first, so could be robustly tested. Further, only IPython would know how to deal with %%html and %%markdown, etc. Which in turn might require additional dependencies.

Also, ipython has already transitioned to py3 only, which is good for black. Even then, IPython probably wouldn't adopt a formatter as a runtime dependency, and would still require a user to include it in an environment.

Thanks for starting the discussion!

[SylvainCorlay]

This will be resolved when #1005 gets in, because we will be able to use JupyterLab themes.

[MSeal]

I don't think #1005 addresses the higher level topic @SylvainCorlay . Where you thinking of a different issue?

[MSeal]

I agree with @bollwyvl 's points as well. Though black is now officially in the python github, which means it's more formally the supported solution than before: https://github.com/python/black ! So I think getting first class support for this would be great. Likely this would require a new message pattern across zeromq if we wanted code linting / cleanup to be supported generally. In the ipython case this could be pushing code into black and returning the results for clients to consume.

I'll look at opening a discussion topic on https://discourse.jupyter.org/ about such a change, since it spans more than any one project to support properly.

[SylvainCorlay]

I don't think #1005 addresses the higher level topic @SylvainCorlay . Where you thinking of a different issue?

Hum i was looking for an older issue with respect to a dark theme. Not the black formatter.

[bollwyvl]

Exciting stuff about black!

I hacked up a custom comm and labextension for this purpose, based on the LSP data model:

https://github.com/deathbeds/lintotype

No release, but the binder works! The high road would be a full kernel-level LS, but it's easier to show than tell for some of these features.

I didn't do an nbconvert p(re|ost)processor, but presumably it could be done... Spin up the kernel in question, and interrogate it, I guess? Especially with something as malleable as ipython's syntax (see deathbeds/pidgin) it's really hard to reason about, but again, it's easy to opt out.

By the by, pretty much anything on deathbeds is up for adoption, if it looks interesting!

On Fri, May 3, 2019, 11:43 Matthew Seal notifications@github.com wrote:

I agree with @bollwyvl https://github.com/bollwyvl 's points as well. Though black is now officially in the python github: https://github.com/python/black ! So I think getting first class support for this would be great. Likely this would require a new message pattern across zeromq if we wanted code linting / cleanup to be supported generally. In the ipython case this could be pushing code into black and returning the results for clients to consume.

I'll look at opening a discussion topic on https://discourse.jupyter.org/ about such a change, since it spans more than any one project to support properly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyter/nbconvert/issues/908#issuecomment-489140950, or mute the thread https://github.com/notifications/unsubscribe-auth/AAALCRAPYPX5CIY4EXZAO6DPTRMRRANCNFSM4GCAMYVQ .

[t-makaro]

I don't think this belongs in nbconvert because it is python specific. However, I do believe that tools to lint and auto-format notebooks would be nice. I've been wanting to start writing a CLI program that lints notebooks with whatever one's favourite linter is. I know there are a couple tools out there but none really satisfy me.

[mwouts]

Nice question... Actually I think I have an answer for that, it's

jupytext --pipe black notebook.ipynb

Jupytext is a tool that converts notebooks to many formats, including Python scripts (and I am the author).

With the above command, the notebook is piped into black as a Python script with the percent format (cells are separated with # %% markers). The updated input cells are then reinjected into the original notebook. Both the metadata and the outputs are preserved in the operation.

You could also run jupytext --pipe black on *.ipynb, or even use it as a Git pre-commit hook - cf the documentation.

[t-makaro]

We could implement similar functionality in nbconvert by piping each individual cell though as code to another program with a preprecessor.