`Extension error` when both `"myst_nb"` and `"myst_parser"` are in `extensions`

NickleDave commented 2 years ago

Describe the bug

Hi, thanks so much for MyST-NB, I am very happy I can include .md notebooks in my build, and switch over from nbsphinx fairly painlessly.

I am not even sure if this is a bug per se but it is behavior that was a bit surprising so I want to raise an issue about it, since I would guess that other people that switch to myst-parser and then add myst-nb might run into the same situation.

context I added MyST-NB to a project after switching from .rst to MyST parser, because I wanted to additionally execute notebooks during builds, which double as tutorial material in the docs.

This gave me something like the following in my conf.py

extensions = [
    "myst_nb",
    "myst_parser",
    ...
]

expectation I expected to be able to just build and execute notebooks after this.

bug But instead I get:

$ sphinx-build -nW --keep-going -b html docs/ docs/_build/html
Running Sphinx v4.5.0

Extension error:
source_suffix '.md' is already registered

(I get the same error when using sphinx-autobuild, if it matters.)

problem

I think this might be intended behavior, judging from this note in the docs?

Loading the myst_nb extension also activates the myst_parser extension, for enabling the MyST flavour of Markdown. It is not required to add this explicitly in the list of extensions.

But it's still surprising, at least to me.

I would guess one of the following behaviors might be a bit less confusing:

a very explicit error, like: there is no need to add myst_parser when you are using myst_nb, please remove myst_parser from extensions in conf.py
- by the same token, if this is the usage pattern, then maybe the docs should be revised to an admonition that very clearly states "you will need to remove myst_parser from extensions if you add myst_nb"
or make it so that MyST-NB and MyST parser "know" about each other and can gracefully co-exist as extensions -- I realize this might be easier said than done depending on the plumbing involved

Reproduce the bug

I am able to get this to happen with a minimal build so I'm pretty sure it's not something unique to my setup.

make a venv
pip install sphinx + myst-parser
make a dummy index.md like the one attached
additionally pip install myst-nb
modify conf.py as above so extensions includes myst_nb and myst_parser
???
profit
er no I mean get the same error

$ sphinx-build -nW --keep-going -b html docs/ docs/_build/html
Running Sphinx v4.5.0

Extension error:
source_suffix '.md' is already registered

index.md

List your environment

Not using jupyter-book
Python 3.10
Ubuntu-ish

 pip freeze
alabaster==0.7.12
asttokens==2.0.5
attrs==21.4.0
Babel==2.10.1
backcall==0.2.0
certifi==2022.5.18.1
charset-normalizer==2.0.12
click==8.1.3
debugpy==1.6.0
decorator==5.1.1
docutils==0.17.1
entrypoints==0.4
executing==0.8.3
fastjsonschema==2.15.3
greenlet==1.1.2
idna==3.3
imagesize==1.3.0
importlib-metadata==4.11.4
ipykernel==6.13.0
ipython==8.4.0
jedi==0.18.1
Jinja2==3.1.2
jsonschema==4.6.0
jupyter-cache==0.5.0
jupyter-client==7.3.1
jupyter-core==4.10.0
markdown-it-py==2.1.0
MarkupSafe==2.1.1
matplotlib-inline==0.1.3
mdit-py-plugins==0.3.0
mdurl==0.1.1
myst-nb==0.15.0
myst-parser==0.17.2
nbclient==0.5.13
nbformat==5.4.0
nest-asyncio==1.5.5
packaging==21.3
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
prompt-toolkit==3.0.29
psutil==5.9.1
ptyprocess==0.7.0
pure-eval==0.2.2
Pygments==2.12.0
pyparsing==3.0.9
pyrsistent==0.18.1
python-dateutil==2.8.2
pytz==2022.1
PyYAML==6.0
pyzmq==23.1.0
requests==2.27.1
six==1.16.0
snowballstemmer==2.2.0
Sphinx==4.5.0
sphinx-togglebutton==0.3.1
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
SQLAlchemy==1.4.37
stack-data==0.2.0
tabulate==0.8.9
tornado==6.1
traitlets==5.2.2.post1
typing_extensions==4.2.0
urllib3==1.26.9
wcwidth==0.2.5
zipp==3.8.0

welcome[bot] commented 2 years ago

Thanks for opening your first issue here! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.
If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).
Welcome to the EBP community! :tada:

Timost commented 2 years ago

Hi ! +1 on this issue.

I think warning about this behaviour should be made more visible, maybe in the https://myst-nb.readthedocs.io/en/latest/quickstart.html

NickleDave commented 2 years ago

I noticed when googling that there used to be a {warning} callout but it must have been removed at some point?

From this page: https://myst-nb.readthedocs.io/en/v0.9.0/_sources/use/myst.md.txt

:::{warning}
If you are using MyST-NB in your documentation, do not activate `myst-parser`. It will
be automatically activated by `myst-nb`.
:::

This is more direct (and possibly not the friendliest of language, maybe why it was removed) but still doesn't state directly that activating both will cause an error

CsatiZoltan commented 1 year ago

@NickleDave If I replace 'myst_parser' with 'myst_nb', in the extensions variable, while writing in Markdown syntax

source_suffix = {'.md': 'markdown'`}

I get the error

Sphinx error:
Source parser for markdown not registered
make: *** [html] Error 2

It seems to me that the myst_parser is required to write in Markdown.

ahasha commented 1 year ago

I have the same issue -- I have non-notebook documentation written in myst-markdown, and wanted to switch from nbsphinx to myst-nb for my notebook-driven documentation pages to avoid their pandoc dependency and get fully into the executablebooks ecosystem. However, it seems I cannot have both because of this issue.

FYI -- I have installed myst_nb from the current main branch, rev 3d6a5d1, in order to use it with sphinx 6.2.1

pwuertz commented 1 year ago

I'm new to this and quite confused. Maybe the general idea is to pipe everything via jupytext into myst_nb because every .md file is potentially a Notebook file?

https://docs.readthedocs.io/en/stable/guides/jupyter.html#using-notebooks-in-other-formats suggests:

# conf.py
nb_custom_formats = {
    ".md": ["jupytext.reads", {"fmt": "mystnb"}],
}

In which case myst-nb handles both plain .md files and Notebooks in .md format? Is there any downside to this (handling .md via myst-nb instead of using myst-parser)?

maxstrobel commented 1 year ago

+1 same issue here 😅 are already workarounds available?

NickleDave commented 1 year ago

Sounds like the question of whether to specify both myst-nb and myst-parser in extension is confusing in general.

But we're now at a point on this issue where people are chiming in with multiple things that might or might not be related.

@chrisjsewell @choldgraf apologies in advance for adding to your notifications but what do you think about clarifying here and/or adding info in the docs about when and how to use these extensions together?

michaelweinold commented 3 weeks ago

https://github.com/executablebooks/MyST-NB/pull/622

executablebooks / MyST-NB