Configurations in `pyproject.toml` seem to be ignored

[bkarpati]

Jupytext version: 1.13.1 Python version: 3.7.5

I have a project structure similar to the following:

project/
   |-pyproject.toml
   |-module/
      |-submodule_1/
      |-submodule_2/
         |-tests/
            |-test_notebook.ipynb

I added the following to my pyproject.toml file:

[tool.jupytext]
formats = "ipynb,py:light"

I am running jupyter from the project directory where the pyproject.toml file is also located. However, it seems that the configurations are ignored. When I save test_notebook.ipynb no .py file is generated or if there already is one, it doesn't get updated.

If instead of modifying the pyproject.toml file, I create a jupytext.toml file at the same location and with the content

formats = "ipynb,py:light"

everything works as expected, test_notebook.ipynb is paired properly.

Am I doing something wrong when trying to configure jupytext through the pyproject.toml file? Or have I perhaps stumbled on a bug here?

And last but not least, thank you for the fantastic library.

[mwouts]

Hi @bkarpati , thank you for reporting the issue.

I have turned the report above in a test, and at least locally the test seems to pass.

To understand what the problem is I would advice to do the following:

1. Is project indeed the root for your Jupyter notebook server? Your report above states that clearly, but maybe we should double check, e.g. what is the full URL for the test_notebook.ipynb file?
2. Which version of Jupytext is being used by Jupyter? The option to use pyproject.toml was introduced rather recently (version 1.12.0)
3. Can you provide an extract of the logs of the Jupyter server when you save the notebook in both cases (pyproject.toml vs jupytext.toml)?

Thank you for you help!

[bkarpati]

I'm not that familiar with the workings of Jupyter notebooks so I apologise in advance if I fail to answer the questions properly.

I recreated locally the specific project structure I specified in my original issue. Fortunately(?), I see the same issue. So everything was done in this dummy environment.

Note, I omitted the timestamps from the log messages below.

1. When I start the notebook from my terminal I get the following log: [I NotebookApp] Serving notebooks from local directory: C:\dev\project which is indeed the root directory. When I navigate to and open the test_notebook.ipynb file I get this log: [W NotebookApp] Notebook module/submodule_2/tests/test_notebook.ipynb is not trusted and when I save it I get: [I NotebookApp] Saving file at /module/submodule_2/tests/test_notebook.ipynb which to me suggests that the notebook is picked up relative to the root project directory.

2. I ran !pip list inside the notebook which gave me jupytext 1.13.1

3. With log levels set to the default, I see the following when saving the notebook with jupytext.toml: [I NotebookApp] Loading Jupytext configuration file at /jupytext.toml [I NotebookApp] Saving file at /module/submodule_2/tests/test_notebook.ipynb [I NotebookApp] Loading Jupytext configuration file at /jupytext.toml [I NotebookApp] Saving test_notebook.ipynb [W NotebookApp] Notebook module/submodule_2/tests/test_notebook.ipynb is not trusted [I NotebookApp] Saving test_notebook.py in format py:light Perhaps worth mentioning that I also see a lot of the following log as I move around the notebook interface in this case: [I NotebookApp] Loading Jupytext configuration file at /jupytext.toml When I have the config in the pyproject.toml file and save the notebook I see the following: [I NotebookApp] Saving file at /module/submodule_2/tests/test_notebook.ipynb [I NotebookApp] Saving test_notebook.ipynb [W NotebookApp] Notebook module/submodule_2/tests/test_notebook.ipynb is not trusted And opposed to the other case I don't see any of the Loading jupytext configuration file logs

[mwouts]

Hi @bkarpati , sorry for the delay in answering. Yes the logs above do confirm what you reported.

But there is still one thing that I'd like to investigate... I am not so sure of which environment gets described when you do !pip list in the notebook, so may I ask you to stop your Jupyter server, and in the same terminal where it was running, could you run pip list or jupytext --version? Because what matters here is the version of Jupytext used by Jupyter (and not the one that you might have installed in the kernel used by the notebook)

[bkarpati]

Now it's my turn to say sorry for the belated reply.

Running pip list or jupytext --version in the terminal itself also returns 1.13.1.

Is it possible that the behaviour is affected by the version of Jupyter itself? I have the following related packages installed:

ipykernel                         6.4.1
jupyter                           1.0.0
jupyter-client                    7.0.3
jupyter-console                   6.4.0
jupyter-core                      4.8.1
jupyterlab-pygments               0.1.2
jupyterlab-widgets                1.0.2
nbclient                          0.5.8
nbconvert                         6.3.0
nbformat                          5.1.3

[mwouts]

Hello @bkarpati , well that is interesting !

I'd like to see if we can reproduce that programmatically.

Could you please clone this branch: test_pyproject_toml_876, and execute

pytest tests/test_cm_config.py -k test_pyproject_toml_file_issue_876

Does the test pass for you as well? Or not?

Then, what happens if you replace tmpdir in the test with the root dir for your Jupyter install, and just execute that part? https://github.com/mwouts/jupytext/blob/79ceedf4549a11ae778ddc4b08b8d043d4590c69/tests/test_cm_config.py#L210-L217

[bkarpati]

Hey @mwouts,

I have cloned the test_pyproject_toml_876 branch and ran the specified command. It reports that both test cases have passed. The plot thickens.

Unfortunately, I don't quite follow your second point. What exactly do you mean by

the root dir for your Jupyter install

Do you actually mean one of the Jupyter related directories in the site-packages of my virtual environment? Or my local example project dir where I reproduced the issue? Or perhaps a third option?

I did try modifying the test to perform according to the second option. So basically running in my project dir instead of the tmpdir and the pyproject.toml test case did pass.

[mwouts]

I have cloned the test_pyproject_toml_876 branch and ran the specified command. It reports that both test cases have passed.

Great! Thank you for checking out

I did try modifying the test to perform according to the second option.

Yes the root dir is by default the directory where you launch Jupyter

So basically running in my project dir instead of the tmpdir and the pyproject.toml test case did pass.

Interesting! So in the context of the test the pyproject.toml is active, but not in Jupyter.

Well at the moment I only see one additional thing that you can check - can you see and open (read) the pyproject.toml in Jupyter?

Or maybe also - does it help if you compare the logs of the test with those of Jupyter? (you might have to append --log-file-level INFO to the command above)

[bkarpati]

Sorry for letting this issue sit stale for such a long time. I will try to get back to it in the coming days and give your latest suggestions a go.

[richtong]

OK, I think I have a similar issue which is that the pyproject.toml is actually picked up in the --notebook-dir that is set for jupyter when you start it. So for instance for instance, my data is in ./data and the pyproject.toml needs to be there as well otherwise it doesn't know how to sync (in my case) md and ipynb files.

This was puzzling to me for a long because sometimes I start jupyter lab at the home directory . and sometimes I set the --notebook-dir to ./data and the sync and reading of pyproject.toml happens in the former but not the later case.