Clicking on .py file downloads it instead of opening it.

gjover commented 1 year ago

Scenario:

Create an example notebook in myfolder/test.ipynb
Pair notebook with percent script

Issue

Go to myfolder folder
when clicking on test.py the browser downloads the file

Expected behaviour

test.py opens as any other notebook

Environment I have test it in Chrome and Firefox versions: rise 5.7.1 jupyter 1.0.0 jupyter_contrib_nbextensions 0.7.0 jupytext 1.14.4

mwouts commented 1 year ago

Hey @gjover , thank you for reaching out. What version of Jupyter Lab are you using?

By default you need to use a right click to open .py files as notebooks.

However, if you follow the instructions With a click on the text file in JupyterLab in the README.md, you should be able to do what you expect i.e. open .py files as notebooks by default.

Please let us know if that works for you.

gjover commented 1 year ago

Hi, I am using jupyter notebooks.

one can see this behavior in this binder: https://mybinder.org/v2/gh/gjover/modelitzacio.git/main?filepath=notebooks/Index.py

mwouts commented 1 year ago

Hey @gjover , thank you for the link.

I have been able to reproduce the documented behavior on Binder (with both your binder and the binder for this repository).

Locally I have not yet been able to reproduce. Do you have the same issue locally? If so, could you please share the output of jupyter --version? My versions are:

$ jupyter --version
Selected Jupyter core packages...
IPython          : 8.6.0
ipykernel        : 6.17.1
ipywidgets       : 8.0.2
jupyter_client   : 7.4.7
jupyter_core     : 5.2.0
jupyter_server   : 1.23.4
jupyterlab       : 3.5.3
nbclient         : 0.5.13
nbconvert        : 7.2.5
nbformat         : 5.7.0
notebook         : 6.5.2
qtconsole        : 5.4.0
traitlets        : 5.7.1

LoicGrobol commented 1 year ago

This issue also happens on binderhub with md files like this. I haven't been able to reproduce it in local, but it was working fine in binder only a few weeks ago.

It happens in Binder both with a direct link file and with first opening the nb interface and navigating to the file
In a (still binder) jupyterlab session, it does opens the md file but as raw text instead of a notebook

mwouts commented 1 year ago

When I execute jupyter --version on the binder for this project I get this:

Selected Jupyter core packages...
IPython          : 7.33.0
ipykernel        : 6.16.2
ipywidgets       : 8.0.2
jupyter_client   : 7.4.9
jupyter_core     : 4.11.1
jupyter_server   : 2.0.0rc3
jupyterlab       : 4.0.0a32
nbclient         : 0.7.0
nbconvert        : 7.2.9
nbformat         : 5.7.3
notebook         : 6.4.12
qtconsole        : not installed
traitlets        : 5.9.0

while when I run it on @gjover 's binder I get these versions:

jovyan@jupyter-gjover-2dmodelitzacio-2dej6yzjnk:~$ jupyter --version
Selected Jupyter core packages...
IPython          : 7.33.0
ipykernel        : 6.21.3
ipywidgets       : 8.0.4
jupyter_client   : 8.0.3
jupyter_core     : 4.12.0
jupyter_server   : 1.23.5
jupyterlab       : 3.4.8
nbclient         : 0.7.2
nbconvert        : 6.5.3
nbformat         : 5.7.3
notebook         : 6.4.12
qtconsole        : 5.4.1
traitlets        : 5.9.0

I don't have time now for this, but it would be interesting to test whether we can reproduce the issue locally by installing these versions.

matthew-brett commented 1 year ago

Just to say - we have this problem with our 2i2c JupyterHub

https://github.com/2i2c-org/infrastructure/issues/2302

and it's making a mess of our textbook pages - so if there's anything I can do to help debug - please let me know.

mwouts commented 1 year ago

Hi @matthew-brett . Yes sure I can imagine the impact of this... I am not able to reproduce locally (more on this below). If someone else could confirm that the issue does not occur locally we could probably open an issue with https://github.com/jupyterhub/repo2docker. What do you think?

Locally I have changed my env to have

IPython          : 8.6.0
ipykernel        : 6.17.1
ipywidgets       : 8.0.2
jupyter_client   : 8.0.3
jupyter_core     : 4.12.0
jupyter_server   : 1.23.5
jupyterlab       : 3.4.8
nbclient         : 0.7.2
nbconvert        : 7.2.5
nbformat         : 5.7.0
notebook         : 6.4.12
qtconsole        : 5.4.0
traitlets        : 5.7.1

which mimics most of @gjover 's env on Binder, and still the issue is not present here.

matthew-brett commented 1 year ago

I too cannot reproduce it locally. However, @yuvipanda did reproduce it locally, and I don't think we ever worked out what might differ between our systems. @yuvipanda - can you comment? I reported my full environment before, that does not have the problem locally, but it was:

Homebrew Python 3.10.9, mac M2.
A fresh virtualenv
Moved ~/.jupyter and ~/.config directories out of the way
Upgraded pip to latest (this had no effect either way in previous tests)

$ pip install jupytext notebook
$ cat > some.md << EOF
# Heading

Some text.
EOF
$ jupyter notebook

After that, opening some.md gives me the classic notebook interface.

$ pip list

Package                  Version
------------------------ -------
anyio                    3.6.2
appnope                  0.1.3
argon2-cffi              21.3.0
argon2-cffi-bindings     21.2.0
arrow                    1.2.3
asttokens                2.2.1
attrs                    22.2.0
backcall                 0.2.0
beautifulsoup4           4.11.2
bleach                   6.0.0
cffi                     1.15.1
comm                     0.1.2
debugpy                  1.6.6
decorator                5.1.1
defusedxml               0.7.1
executing                1.2.0
fastjsonschema           2.16.2
fqdn                     1.5.1
idna                     3.4
ipykernel                6.21.2
ipython                  8.10.0
ipython-genutils         0.2.0
isoduration              20.11.0
jedi                     0.18.2
Jinja2                   3.1.2
jsonpointer              2.3
jsonschema               4.17.3
jupyter_client           8.0.3
jupyter_core             5.2.0
jupyter-events           0.6.3
jupyter_server           2.3.0
jupyter_server_terminals 0.4.4
jupyterlab-pygments      0.2.2
jupytext                 1.14.4
markdown-it-py           2.2.0
MarkupSafe               2.1.2
matplotlib-inline        0.1.6
mdit-py-plugins          0.3.4
mdurl                    0.1.2
mistune                  2.0.5
nbclassic                0.5.2
nbclient                 0.7.2
nbconvert                7.2.9
nbformat                 5.7.3
nest-asyncio             1.5.6
notebook                 6.5.2
notebook_shim            0.2.2
packaging                23.0
pandocfilters            1.5.0
parso                    0.8.3
pexpect                  4.8.0
pickleshare              0.7.5
pip                      23.0.1
platformdirs             3.0.0
prometheus-client        0.16.0
prompt-toolkit           3.0.37
psutil                   5.9.4
ptyprocess               0.7.0
pure-eval                0.2.2
pycparser                2.21
Pygments                 2.14.0
pyrsistent               0.19.3
python-dateutil          2.8.2
python-json-logger       2.0.7
PyYAML                   6.0
pyzmq                    25.0.0
rfc3339-validator        0.1.4
rfc3986-validator        0.1.1
Send2Trash               1.8.0
setuptools               66.1.1
six                      1.16.0
sniffio                  1.3.0
soupsieve                2.4
stack-data               0.6.2
terminado                0.17.1
tinycss2                 1.2.1
toml                     0.10.2
tornado                  6.2
traitlets                5.9.0
uri-template             1.2.0
wcwidth                  0.2.6
webcolors                1.12
webencodings             0.5.1
websocket-client         1.5.1
wheel                    0.38.4

I get the same (correct JupyterText behavior in classic) from the same procedure on an Intel Mac running Homebrew Python 3.9.16, in Firefox, Safari and Chrome.

Likewise for a Rasberry Pi running Raspbian Python 3.9.2.

matthew-brett commented 1 year ago

Sorry - also - I think someone else at 2i2c - ?Pris - also succeeded in reproducing locally "a conda/mamba environment", but I know no more details than that.

I can reproduce it, serving the notebooks from a recentish Jupyter-Notebook-based container:

docker run  -it  --rm  -p 8888:8888 lisacuk/lishub-base:2283cf4

https://github.com/lisds/lisds-images/tree/2283cf4/lishub-base

mwouts commented 1 year ago

@matthew-brett then I would recommend opening an issue at https://github.com/jupyterhub/repo2docker/issues

Also do you have the same issue with JupyterLab?

For another project I use JupyterLab rather than Notebook and that one seems to continue working as expected, cf. this link (a custom default_setting_overrides.json is required to open text documents as notebooks by default, see also binder/postBuild).

matthew-brett commented 1 year ago

For me - JupyterLab works as expected - but for my students, using the same hub, they have to right-clck open-as to open as a notebook. I've added the fix you referred to - will ask the students whether that resolves it for them.

However, I'd like to use Classic if I can - I'm still getting some bugs with Lab.

matthew-brett commented 1 year ago

Sorry to ask - but for a correct image setup, what configuration should be present for Classic? At the moment, I have:

RUN jupyter serverextension enable jupytext

RUN echo 'c.NotebookApp.contents_manager_class = "jupytext.TextFileContentsManager"' >> .jupyter/jupyter_notebook_config.py

Is there anything else I could try to configure the image correctly?

mwouts commented 1 year ago

Sorry to ask - but for a correct image setup, what configuration should be present for Classic?

Thanks for asking actually! Well it has been a long time I last setup a binder but my expectation is that you only have to install jupytext. Nothing else, as installing it enables the extensions by default. And I would recommend that you remove the explicit contents manager class, that instruction is really a last resort and might cause some other issues.

matthew-brett commented 1 year ago

The explicit contents manager step was a debug hack of last resort.

My speculation was that the binder / JupyterHub setup somehow disabled the Jupytext configuration. Is that possible? How would I detect it if so.

What would a bug-report to https://github.com/jupyterhub/repo2docker/issues look like? I'm worried that I don't understand the configuration process well enough to say anything more than "Jupytext configuration used to work and now it doesn't" - to which the obvious reply is - not our problem ...

yuvipanda commented 1 year ago

Here are the instructions I used locally to reproduce this.

Create a fresh venv
pip install jupyterlab jupytext nbgitpuller
jupyter notebook or jupyter lab to start the server (reproduces either way)
Go to http://localhost:8888/git-pull?repo=https%3A//github.com/lisds/textbook&urlpath=tree/textbook/data-frames/df_index.Rmd&branch=main, which is the local version of the link you are using. See that it is just downloading the .Rmd file.
Go to http://localhost:8888/git-pull?repo=https%3A//github.com/lisds/textbook&urlpath=lab/tree/textbook/data-frames/df_index.Rmd&branch=main, which is the jupyterlab variant of the same nbgitpuller URL (note the lab/ after urlpath=). This puts you in the plaintext editor with the file open, rather than as a notebook.

I can reproduce these as of now, on Mac OS, python 3.10. Here's the full list of installed packages:

aiofiles==22.1.0
aiosqlite==0.18.0
anyio==3.6.2
appnope==0.1.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
astroid==2.15.0
asttokens==2.2.1
attrs==22.2.0
autopep8==1.6.0
Babel==2.12.1
backcall==0.2.0
beautifulsoup4==4.12.0
black==23.1.0
bleach==6.0.0
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==3.1.0
click==8.1.3
comm==0.1.2
debugpy==1.6.6
decorator==5.1.1
defusedxml==0.7.1
dill==0.3.6
docstring-to-markdown==0.11
executing==1.2.0
fastjsonschema==2.16.3
flake8==6.0.0
fqdn==1.5.1
idna==3.4
ipykernel==6.22.0
ipython==8.11.0
ipython-genutils==0.2.0
isoduration==20.11.0
isort==5.12.0
jedi==0.18.2
Jinja2==3.1.2
json5==0.9.11
jsonpointer==2.3
jsonschema==4.17.3
jupyter-events==0.6.3
jupyter-ydoc==0.2.3
jupyter_client==8.1.0
jupyter_core==5.3.0
jupyter_server==2.5.0
jupyter_server_fileid==0.8.0
jupyter_server_terminals==0.4.4
jupyter_server_ydoc==0.8.0
jupyterlab==3.6.2
jupyterlab-pygments==0.2.2
jupyterlab_server==2.20.0
jupytext==1.14.5
lazy-object-proxy==1.9.0
markdown-it-py==2.2.0
MarkupSafe==2.1.2
matplotlib-inline==0.1.6
mccabe==0.7.0
mdit-py-plugins==0.3.5
mdurl==0.1.2
mistune==2.0.5
mypy==1.1.1
mypy-extensions==1.0.0
nbclassic==0.5.3
nbclient==0.7.2
nbconvert==7.2.10
nbformat==5.8.0
nbgitpuller==1.1.1
nest-asyncio==1.5.6
notebook==6.5.3
notebook_shim==0.2.2
packaging==23.0
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.11.1
pexpect==4.8.0
pickleshare==0.7.5
platformdirs==3.1.1
pluggy==1.0.0
prometheus-client==0.16.0
prompt-toolkit==3.0.38
psutil==5.9.4
ptyprocess==0.7.0
pure-eval==0.2.2
pycodestyle==2.10.0
pycparser==2.21
pydocstyle==6.2.3
pyflakes==3.0.1
Pygments==2.14.0
pylint==2.17.0
pyls-isort==0.2.2
pylsp-mypy==0.6.6
pylsp-rope==0.1.11
pyrsistent==0.19.3
python-dateutil==2.8.2
python-json-logger==2.0.7
python-lsp-black==1.2.1
python-lsp-jsonrpc==1.0.0
python-lsp-server==1.7.1
pytoolconfig==1.2.5
PyYAML==6.0
pyzmq==25.0.2
requests==2.28.2
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rope==1.7.0
Send2Trash==1.8.0
six==1.16.0
sniffio==1.3.0
snowballstemmer==2.2.0
soupsieve==2.4
stack-data==0.6.2
terminado==0.17.1
tinycss2==1.2.1
toml==0.10.2
tomli==2.0.1
tomlkit==0.11.6
tornado==6.2
traitlets==5.9.0
typing_extensions==4.5.0
ujson==5.7.0
uri-template==1.2.0
urllib3==1.26.15
wcwidth==0.2.6
webcolors==1.12
webencodings==0.5.1
websocket-client==1.5.1
whatthepatch==1.0.4
wrapt==1.15.0
y-py==0.5.9
yapf==0.32.0
ypy-websocket==0.8.2

Hope this is helpful.

yuvipanda commented 1 year ago

I can also validate that jupytext is installed, and shows up in jupyter server extension list and in jupyter serverextension list

mwouts commented 1 year ago

Thank you so much, @yuvipanda ! This is super helpful.

I will have a look at this tonight.

Naive question: Why are you accessing the url through nbgitpuller ? Is it the way it is done on Binder ?

yuvipanda commented 1 year ago

Not related to binder, it was just how it was originally reported to us. nbgitpuller just redirects you to the end URL eventually, so it should ideally be irrelevant, other than deciding if it goes through via classic notebook or lab.

matthew-brett commented 1 year ago

Just to confirm the previous clash of our experience - I followed Yuvi's recipe exactly, with the addition, that I moved my .jupyter and .config folders out of the way, before doing the replication.

I get the correct loading of the notebook with those exact steps.

Mac M2, Homebrew Python 3.10.10:

Package                  Version
------------------------ ---------
aiofiles                 22.1.0
aiosqlite                0.18.0
anyio                    3.6.2
appnope                  0.1.3
argon2-cffi              21.3.0
argon2-cffi-bindings     21.2.0
arrow                    1.2.3
asttokens                2.2.1
attrs                    22.2.0
Babel                    2.12.1
backcall                 0.2.0
beautifulsoup4           4.12.0
bleach                   6.0.0
certifi                  2022.12.7
cffi                     1.15.1
charset-normalizer       3.1.0
comm                     0.1.2
debugpy                  1.6.6
decorator                5.1.1
defusedxml               0.7.1
executing                1.2.0
fastjsonschema           2.16.3
fqdn                     1.5.1
idna                     3.4
ipykernel                6.22.0
ipython                  8.11.0
ipython-genutils         0.2.0
isoduration              20.11.0
jedi                     0.18.2
Jinja2                   3.1.2
json5                    0.9.11
jsonpointer              2.3
jsonschema               4.17.3
jupyter_client           8.1.0
jupyter_core             5.3.0
jupyter-events           0.6.3
jupyter_server           2.5.0
jupyter_server_fileid    0.8.0
jupyter_server_terminals 0.4.4
jupyter_server_ydoc      0.8.0
jupyter-ydoc             0.2.3
jupyterlab               3.6.2
jupyterlab-pygments      0.2.2
jupyterlab_server        2.20.0
jupytext                 1.14.5
markdown-it-py           2.2.0
MarkupSafe               2.1.2
matplotlib-inline        0.1.6
mdit-py-plugins          0.3.5
mdurl                    0.1.2
mistune                  2.0.5
nbclassic                0.5.3
nbclient                 0.7.2
nbconvert                7.2.10
nbformat                 5.8.0
nbgitpuller              1.1.1
nest-asyncio             1.5.6
notebook                 6.5.3
notebook_shim            0.2.2
packaging                23.0
pandocfilters            1.5.0
parso                    0.8.3
pexpect                  4.8.0
pickleshare              0.7.5
pip                      23.0.1
platformdirs             3.1.1
prometheus-client        0.16.0
prompt-toolkit           3.0.38
psutil                   5.9.4
ptyprocess               0.7.0
pure-eval                0.2.2
pycparser                2.21
Pygments                 2.14.0
pyrsistent               0.19.3
python-dateutil          2.8.2
python-json-logger       2.0.7
PyYAML                   6.0
pyzmq                    25.0.2
requests                 2.28.2
rfc3339-validator        0.1.4
rfc3986-validator        0.1.1
Send2Trash               1.8.0
setuptools               67.3.2
six                      1.16.0
sniffio                  1.3.0
soupsieve                2.4
stack-data               0.6.2
terminado                0.17.1
tinycss2                 1.2.1
toml                     0.10.2
tomli                    2.0.1
tornado                  6.2
traitlets                5.9.0
uri-template             1.2.0
urllib3                  1.26.15
wcwidth                  0.2.6
webcolors                1.12
webencodings             0.5.1
websocket-client         1.5.1
wheel                    0.38.4
y-py                     0.5.9
ypy-websocket            0.8.2

matthew-brett commented 1 year ago

@yuvipanda - I notice you've got quite a few refactoring packages like black and rope in your installed packages list - is the virtualenv really clean?

I have also done pip install -r yuvis_list.txt using your list above, and retested, but I still get the correct behavior.

yuvipanda commented 1 year ago

Ah, I forgot my venv setup installs some language server stuff. I made a fully clean venv, and got the same behavior as before.

Here's pip freeze:

aiofiles==22.1.0
aiosqlite==0.18.0
anyio==3.6.2
appnope==0.1.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
asttokens==2.2.1
attrs==22.2.0
Babel==2.12.1
backcall==0.2.0
beautifulsoup4==4.12.0
bleach==6.0.0
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==3.1.0
comm==0.1.2
debugpy==1.6.6
decorator==5.1.1
defusedxml==0.7.1
executing==1.2.0
fastjsonschema==2.16.3
fqdn==1.5.1
idna==3.4
ipykernel==6.22.0
ipython==8.11.0
ipython-genutils==0.2.0
isoduration==20.11.0
jedi==0.18.2
Jinja2==3.1.2
json5==0.9.11
jsonpointer==2.3
jsonschema==4.17.3
jupyter-events==0.6.3
jupyter-ydoc==0.2.3
jupyter_client==8.1.0
jupyter_core==5.3.0
jupyter_server==2.5.0
jupyter_server_fileid==0.8.0
jupyter_server_terminals==0.4.4
jupyter_server_ydoc==0.8.0
jupyterlab==3.6.2
jupyterlab-pygments==0.2.2
jupyterlab_server==2.20.0
jupytext==1.14.5
markdown-it-py==2.2.0
MarkupSafe==2.1.2
matplotlib-inline==0.1.6
mdit-py-plugins==0.3.5
mdurl==0.1.2
mistune==2.0.5
nbclassic==0.5.3
nbclient==0.7.2
nbconvert==7.2.10
nbformat==5.8.0
nbgitpuller==1.1.1
nest-asyncio==1.5.6
notebook==6.5.3
notebook_shim==0.2.2
packaging==23.0
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
platformdirs==3.1.1
prometheus-client==0.16.0
prompt-toolkit==3.0.38
psutil==5.9.4
ptyprocess==0.7.0
pure-eval==0.2.2
pycparser==2.21
Pygments==2.14.0
pyrsistent==0.19.3
python-dateutil==2.8.2
python-json-logger==2.0.7
PyYAML==6.0
pyzmq==25.0.2
requests==2.28.2
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
Send2Trash==1.8.0
six==1.16.0
sniffio==1.3.0
soupsieve==2.4
stack-data==0.6.2
terminado==0.17.1
tinycss2==1.2.1
toml==0.10.2
tornado==6.2
traitlets==5.9.0
uri-template==1.2.0
urllib3==1.26.15
wcwidth==0.2.6
webcolors==1.12
webencodings==0.5.1
websocket-client==1.5.1
y-py==0.5.9
ypy-websocket==0.8.2

yuvipanda commented 1 year ago

I primarily use Firefox, perhaps this is browser related?

matthew-brett commented 1 year ago

I said this in the previous thread, but I tried on Safari and Chrome as well, with the same result (correct Jupytext behavior). I usually use Firefox too.

Did you move your .jupyter and .config directories as well? (I'm sure you did, just clutching at straws).

mwouts commented 1 year ago

@yuvipanda I confirm that I can reproduce this locally. I have followed the following steps:

python -m venv jupytext_1051
conda deactivate  # python was from a conda env
source jupytext_1051/bin/activate
pip install notebook jupytext
jupyter notebook

Now what puzzles me is that not all the .py files are downloaded. I think I saw this:

Untitled.py at the root was always downloaded (and there does exist a paired Untitled.ipynb)
The same file copied to Untitled2.py opens as a notebook
.py files in a subfolder (of the jupyter server root) were always displayed correctly as notebooks

Below are the logs when I click on Untitled.py (downloaded) then on Untitled2.py (opens as a notebook). You will note that Untitled.py (the one which is downloaded instead of opened as a notebook) does not even appear in the logs.

(jupytext_1051) ~$ jupyter notebook

  _   _          _      _
 | | | |_ __  __| |__ _| |_ ___
 | |_| | '_ \/ _` / _` |  _/ -_)
  \___/| .__/\__,_\__,_|\__\___|
       |_|

Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.

https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html

Please note that updating to Notebook 7 might break some of your extensions.

[I 22:52:48.272 NotebookApp] [Jupytext Server Extension] Deriving a JupytextContentsManager from LargeFileManager
[I 22:52:48.274 NotebookApp] Serving notebooks from local directory: /home/marc
[I 22:52:48.274 NotebookApp] Jupyter Notebook 6.5.3 is running at:
[I 22:52:48.274 NotebookApp] http://localhost:8888/?token=4ec6514f4d9b3834ffcfbbf1f6fdc5b5fc92ffdbcbdd5e6e
[I 22:52:48.274 NotebookApp]  or http://127.0.0.1:8888/?token=4ec6514f4d9b3834ffcfbbf1f6fdc5b5fc92ffdbcbdd5e6e
[I 22:52:48.274 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 22:52:48.300 NotebookApp] 

    To access the notebook, open this file in a browser:
        file:///home/marc/.local/share/jupyter/runtime/nbserver-21702-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=4ec6514f4d9b3834ffcfbbf1f6fdc5b5fc92ffdbcbdd5e6e
     or http://127.0.0.1:8888/?token=4ec6514f4d9b3834ffcfbbf1f6fdc5b5fc92ffdbcbdd5e6e
[I 22:52:55.627 NotebookApp] Kernel started: a288ab77-df65-4cc5-8c09-cce6493cfab8, name: itables
[W 22:52:55.668 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20230321225248 (127.0.0.1) 9.040000ms referer=http://localhost:8888/notebooks/Untitled2.py

It has been a long time since I last looked at these kind of logs but obviously the issue is related to the 'GET' evens.

If I redo the same with --log-level 0 I do find a line for Untitled.py (the one that is downloaded):

[D 22:58:42.078 NotebookApp] 200 GET /api/contents?type=directory&_=1679439521951 (127.0.0.1) 5.390000ms
[D 22:58:44.976 NotebookApp] 200 GET /files/Untitled.py (127.0.0.1) 1.760000ms

followed later on by

[D 22:58:47.279 NotebookApp] 200 GET /notebooks/Untitled2.py (127.0.0.1) 244.260000ms

when the second file is opened as a notebook.

This seems to be consistent with the fact that Untitled.py is downloaded (GET with a files/ prefix) while Untitled2.py opens as a notebook.

My expectation is that Jupytext tells Jupyter that both files are notebooks, and at least from the icons this seems to be the case. I am not sure what part of Jupyter decides how to build the GET query. @yuvipanda any idea?

GeorgianaElena commented 1 year ago

My expectation is that Jupytext tells Jupyter that both files are notebooks, and at least from the icons this seems to be the case. I am not sure what part of Jupyter decides how to build the GET query. @yuvipanda any idea?

@mwouts, I noticed that before going to the URL with the files/ prefix, the initial request is for an URL with /notebooks in the path (which I believe it's set by Jupytext). But that one gets a 301 HTTP status code towards the one containing /files in the path.

My best guess is that this redirect is set by nbclassic in https://github.com/jupyter/nbclassic/blob/f237e24acf1f6a7f08c0593a75a41f5e618b2832/nbclassic/notebookapp.py#L256-L265.

mwouts commented 1 year ago

Interesting! Thank you @GeorgianaElena for the pointers.

the initial request is for an URL with /notebooks in the path (which I believe it's set by Jupytext)

Yes - I think this is consistent with the notebook icon on the .py file - Jupytext's content manager told Jupyter that the .py file was a notebook.

But that one gets a 301 HTTP status code towards the one containing /files in the path.

Do we know where the 301 might come from? I see it means moved permanently... Isn't that surprising? Any idea which part of the code generates that HTTP status code?

mwouts commented 1 year ago

To be more explicit, I am looking for a quick workaround for this in the form of 'downgrade package xxx to version yyy' - the question is which package :smile: ... and then, we will take the time to provide a proper fix.

GeorgianaElena commented 1 year ago

Do we know where the 301 might come from? I see it means moved permanently... Isn't that surprising? Any idea which part of the code generates that HTTP status code?

I believe it might be coming from the tornando's RedirectHandler set by nbclassic here.

To be more explicit, I am looking for a quick workaround for this in the form of 'downgrade package xxx to version yyy'

I'm not sure about this. Are there any Jupytext tests against a jupyter_server and nbclassic setup? I found an older issue about this not working https://github.com/mwouts/jupytext/issues/991, so this might be happening from some time. The trigger now might be the fact more and more transitions from jupyter_notebook to jupyter_server happen. For example, both mybinder and the 2i2c hubs were migrated to jupyter_server and this was what the setup that exposed this behaviour. Sorry if I'm not being very helpful with this one.

mwouts commented 1 year ago

Hi @GeorgianaElena , well yes this is super helpful!

Are there any Jupytext tests against a jupyter_server and nbclassic setup?

No you're right, I have never tested this

The trigger now might be the fact more and more transitions from jupyter_notebook to jupyter_server happen.

Agreed! By the way, in the empty virtual env above where we just do pip install notebook, this transition also seem to have taken place. Maybe we should document how to do the opposite temporary transition while we fix this issue? Locally I have tried uninstalling nbclassic then jupyter_server but the bug is still present.

mwouts commented 1 year ago

This is one of the strangest bug I have ever seen. In the virtual env described above, then

jupytext notebook --port 8888 has the bug (i.e. Untitled.py is downloaded)
but jupytext notebook --port 8889 works fine (i.e. Untitled.py opens as a notebook).

Also, I usually use Firefox, but if I open the notebook server in Chromium then it works fine.

I have the feeling that I know too little JS to address this issue.

On the Python side I have tried to suppress the tornado redirect mentioned by @GeorgianaElena, that had no effect. I have also search for 301 in my env, and could not find anything obvious. I have suspected that the async transition of jupyter_server could be involved in this, but reverting jupyter_server<2 does not fix this either.

@yuvipanda , @GeorgianaElena , @fcollonval , do you thing we could get help from the Jupyter Core team on this?

Again the description of the issue is the following:

Run pip install notebook jupytext in an empty virtual env
Start jupyter notebook
Try to open a .py file at the root - then the file is downloaded (using a GET files/ command) rather than opened with notebook (using GET notebooks/)

I find this issue strange because:

The .py file has a notebook icon, meaning that Jupytext's contents manager did return the information that Untitled.py is a notebook
This does not happen for every .py file (locally I have been able to open test.py and Untitled2.py as notebooks)
This does not happen on port 8889
This does not happen with all web navigators

mwouts commented 1 year ago

I see that I can clear the 301 redirect cache in Firefox with Ctrl+Shit+Delete. This is probably what I missed when doing experiments on the Python side.

mwouts commented 1 year ago

After clearing the cache I see that, to reproduce, I need to pip install jupyterlab jupytext (pip install notebook jupytext does not have the bug), and then I need to launch the server with either jupyter lab and go to the classic interface, or directly launch it with jupyter nbclassic.

The logs say:

(jupytext_1051) marc@tuf:~$ jupyter nbclassic
(...)
[I 2023-03-26 18:56:52.803 ServerApp] 301 GET /notebooks/Untitled.py (@127.0.0.1) 0.46ms

and despite the GET /notebooks command the .py file is downloaded.

mwouts commented 1 year ago

Indeed @GeorgianaElena you were right, Jupytext and NBclassic never worked together.

A temporary workaround for this is

pip install 'notebook<6'

(don't forget to clear the browser cache)

I am not sure how this workaround would translate to Binder.

A fix for nbclassic seems to remove the redirect at https://github.com/jupyter/nbclassic/blob/f237e24acf1f6a7f08c0593a75a41f5e618b2832/nbclassic/notebookapp.py#L256-L265

(the redirect does not seem to be required to open text files in edit mode neither).

I will now open a PR on nbclassic. Thank you everyone here, and thanks @GeorgianaElena for pointing us at the cause of this!

mwouts commented 1 year ago

A temporary workaround for Binder is to add this to the binder/postBuild file:

# This is a **TEMPORARY** workaround for https://github.com/mwouts/jupytext/issues/1051
# Please make sure to remove this when a fix is provided in nbclassic
pip install git+https://github.com/mwouts/nbclassic.git@do_not_redirect_notebook_to_files

matthew-brett commented 1 year ago

Thanks! That workaround works for me, setting up an image for a current JupyterHub (in fact on 2i2c).

mwouts commented 1 year ago

nbclassic==0.5.5 has been released.

The definitive workaround is to require nbclassic>=0.5.5 in binder/requirements.txt as done here: https://github.com/mwouts/jupytext/blob/main/binder/requirements.txt#L8-L9

I plan to delete my fork of nbclassic (and hence break the temporary workaround) in two weeks from now, so please switch to the definitive workaround. Thanks!

mwouts commented 1 year ago

FYI I have just deleted my fork of nbclassic. Hope nobody was using it anymore (if so, please switch to the definitive workaround - see above).

mwouts / jupytext

Clicking on .py file downloads it instead of opening it. #1051