Skip piping non-code cells

rijobro commented 3 years ago

To autofix notebooks, I run a command like this:

jupytext "$filename" \
        --pipe "autoflake --in-place --remove-unused-variables --imports numpy,monai,matplotlib,torch,ignite {}" \
        --pipe "isort -" \
        --pipe "black -l 79 -" \
        --pipe autopep8 \
        --pipe "sed 's/ = list()/ = []/'"

This is removing spaces from the end of markdown lines, where a double space is needed to add a paragraph break.

Is there any way to prevent these autofixes from being applied to anything other than python cells?

rijobro commented 3 years ago

I'm not sure if this can be solved with --pipe-fmt. I can't see a list of arguments for this option, I've only seen auto:percent and py:percent. I'm not sure what the latter does, but it doesn't solve my problem.

mwouts commented 3 years ago

Hi @rijobro , thanks for sharing your use case!

This is removing spaces from the end of markdown lines, where a double space is needed to add a paragraph break.

Very interesting! Yes sure the Python-oriented tools don't know that the comment is written in Markdown, and thus they don't respect this specificity.

Is there any way to prevent these autofixes from being applied to anything other than python cells?

Not at the moment. --pipe runs the selected tool on the text representation of the full notebook (in the format --pipe-fmt, which as you saw defaults to the percent format, i.e. the one with explicit # %% cell markers - probably the most robust format for this use case).

But I agree, most of the tools above will only act on the code, so it would make sense to allow running them only on the code cells. Would a --pipe-code option be what you're after? If you are open to making a PR, here a a few pointers. I think we could:

(in the --pipe-code mode), make a copy of the original notebook that contain only the code cells before exporting it to the text buffer: https://github.com/mwouts/jupytext/blob/41b7e9c93b50da1d6feb486aceb2c4d534374090/jupytext/cli.py#L922
and also, when we override the inputs of the original notebook with the reformatted ones at https://github.com/mwouts/jupytext/blob/41b7e9c93b50da1d6feb486aceb2c4d534374090/jupytext/cli.py#L972 we should tell to the combine_inputs_with_outputs function that we want to preserve the non-code cells.

rijobro commented 3 years ago

Thanks for the quick reply!

Your suggested solution certainly looks like it would solve the problem. Unfortunately, I don't think I'll have time to implement it. In the meantime I've opted for a quick-and-easy fix: --pipe "autopep8 - --ignore W291".

This won't remove any EOL spaces for comments, which will include markdown and non-markdown, but it's a liveable solution for the time being.

mwouts / jupytext

Skip piping non-code cells #723