How to use with Jupytext Markdown notebooks

[basnijholt]

We keep our notebooks in Markdown format using Jupytext. Is there an easy way to get nbQA to work with these files?

I can probably write a pre-commit hook that

• converts md -> ipynb
• runs nbQA
• converts back to md

But I wonder whether there is a simpler solution.

[MarcoGorelli]

Hey @basnijholt

I think you can sync Python files with notebooks in Jupytext? If so, you could run your Python tools on those files directly

If you want to keep them in markdown, I don't think there's a simpler solution, but thanks anyway for your issue!

[rgommers]

We just ran into this for SciPy. Wouldn't it be useful for nbqa to be able to do this? Either by natively understanding jupytext files, or by doing the "convert to ipynb, run nbqa, convert back" internally?

Jupytext is the best way to keep Jupyter notebooks in version control without making a mess, so it would be nice if Jupytext had broad tooling support.

[MarcoGorelli]

Hey @rgommers

Looks like there's a few people interested in this. Given this line in the shed readme, I think this might include @zac-hd

I'll take a look, just want to make sure this doesn't accidentally open up a rabbit hole of features to support

Reopening for now then - in the meantime, some local script which does "convert to ipynb, run nbqa, convert back" sounds like it should be fine

As a side-note, you don't need nbqa to run black on notebooks, I'd suggest using nbqa for running other linters

[MarcoGorelli]

@basnijholt if you have some .md notebooks handy, would really appreciate it if you could try this out and help unearth bugs before a release comes out later this week:

$ pip install git+https://github.com/MarcoGorelli/nbQA.git@jupytext jupytext black
$ nbqa black . --nbqa-files '\.md$'

[basnijholt]

Thanks a lot @MarcoGorelli!

I have tested this on the Adaptive documentation: https://github.com/python-adaptive/adaptive/tree/master/docs/source/tutorial

However, it's not doing anything and simply reporting:

All done! ✨ 🍰 ✨
48 files left unchanged.

When I call it directly:

(base) basnijholt-macbook  ➜  tutorial git:(master) ✗  nbqa black . --nbqa-files tutorial.AverageLearner.md
No valid Python notebooks found in given path(s)

[MarcoGorelli]

Thanks @basnijholt for trying this out!

I've had a look at that notebook, and it looks like it's missing

  language: python

in kernelspec, which nbqa was expecting (to be safe, I was thinking of not formatting notebooks not explicitly marked as Python)

I'll see if there's a way around this - perhaps just checking kernelspec->name is enough

[MarcoGorelli]

Hey @basnijholt - I've pushed another commit now, to check the kernelspec name as well. Now, from python-adaptive, I get:

$ nbqa black . --nbqa-files '\.md$'
reformatted docs/source/tutorial/tutorial.Learner2D.md
reformatted docs/source/tutorial/tutorial.LearnerND.md
reformatted docs/source/algorithms_and_examples.md

All done! ✨ 🍰 ✨
3 files reformatted, 44 files left unchanged.

Note that, to keep execution: in the metadata, you'd probably need to set that in a jupytext config file, e.g. like https://github.com/pymc-devs/pymc-examples/blob/b05301c39be2fdeebfc1c0ed9549c72e53bba61f/.jupytext.toml (but with execution instead of substitutions)

[MarcoGorelli]

Should 🤞 be available, as of version ~1.5.0~ 1.5.1

[MarcoGorelli]

@all-contributors please add @basnijholt and @rgommers for ideas

[allcontributors[bot]]

@MarcoGorelli

I've put up a pull request to add @basnijholt! :tada:

[MarcoGorelli]

looks like these need doing separately

@all-contributors please add @rgommers for ideas

[allcontributors[bot]]

@MarcoGorelli

I've put up a pull request to add @rgommers! :tada: