executablebooks / MyST-NB

Parse and execute ipynb files in Sphinx
https://myst-nb.readthedocs.io
BSD 3-Clause "New" or "Revised" License
200 stars 80 forks source link

Does not work with Sphinx's internationalization workflow #587

Open tonywu6 opened 3 months ago

tonywu6 commented 3 months ago

Describe the bug

Sphinx uses gettext for internationalization: https://www.sphinx-doc.org/en/master/usage/advanced/intl.html

With this workflow, when translations are provided for notebooks, MyST-NB fails with a nbformat.reader.NotJSONError:

python -m sphinx -T -E -b html -D language=de docs docs/_build/html
Full traceback ``` Traceback (most recent call last): File ".venv/lib/python3.11/site-packages/nbformat/reader.py", line 20, in parse_json nb_dict = json.loads(s, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^ File "~/.rye/py/cpython@3.11.8/install/lib/python3.11/json/__init__.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "~/.rye/py/cpython@3.11.8/install/lib/python3.11/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "~/.rye/py/cpython@3.11.8/install/lib/python3.11/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) The above exception was the direct cause of the following exception: Traceback (most recent call last): File ".venv/lib/python3.11/site-packages/sphinx/cmd/build.py", line 298, in build_main app.build(args.force_all, args.filenames) File ".venv/lib/python3.11/site-packages/sphinx/application.py", line 355, in build self.builder.build_update() File ".venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 293, in build_update self.build(to_build, File ".venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 313, in build updated_docnames = set(self.read()) ^^^^^^^^^^^ File ".venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 420, in read self._read_serial(docnames) File ".venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 441, in _read_serial self.read_doc(docname) File ".venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 498, in read_doc publisher.publish() File ".venv/lib/python3.11/site-packages/docutils/core.py", line 236, in publish self.apply_transforms() File ".venv/lib/python3.11/site-packages/docutils/core.py", line 216, in apply_transforms self.document.transformer.apply_transforms() File ".venv/lib/python3.11/site-packages/sphinx/transforms/__init__.py", line 83, in apply_transforms super().apply_transforms() File ".venv/lib/python3.11/site-packages/docutils/transforms/__init__.py", line 182, in apply_transforms transform.apply(**kwargs) File ".venv/lib/python3.11/site-packages/sphinx/transforms/i18n.py", line 397, in apply patch = publish_msgstr(self.app, msgstr, source, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".venv/lib/python3.11/site-packages/sphinx/transforms/i18n.py", line 73, in publish_msgstr doc = reader.read( ^^^^^^^^^^^^ File ".venv/lib/python3.11/site-packages/docutils/readers/__init__.py", line 70, in read self.parse() File ".venv/lib/python3.11/site-packages/docutils/readers/__init__.py", line 76, in parse self.parser.parse(self.input, document) File ".venv/lib/python3.11/site-packages/myst_nb/sphinx_.py", line 89, in parse notebook = nb_reader.read(inputstring) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".venv/lib/python3.11/site-packages/myst_nb/core/read.py", line 38, in standard_nb_read return nbf.reads(text, as_version=NOTEBOOK_VERSION) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".venv/lib/python3.11/site-packages/nbformat/__init__.py", line 89, in reads nb = reader.reads(s, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File ".venv/lib/python3.11/site-packages/nbformat/reader.py", line 76, in reads nb_dict = parse_json(s, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^ File ".venv/lib/python3.11/site-packages/nbformat/reader.py", line 26, in parse_json raise NotJSONError(message) from e nbformat.reader.NotJSONError: Notebook does not appear to be JSON: 'Franz jagt im komplett verwahrlosten Ta... Exception occurred: File ".venv/lib/python3.11/site-packages/nbformat/reader.py", line 26, in parse_json raise NotJSONError(message) from e nbformat.reader.NotJSONError: Notebook does not appear to be JSON: 'Franz jagt im komplett verwahrlosten Ta... ```

Translations look like this

#: ../../index.ipynb:10002
msgid "The quick brown **fox** jumps over the lazy dog"
msgstr "**Franz** jagt im komplett verwahrlosten Taxi quer durch Bayern"

Sphinx will attempt to parse each msgstr using the parser configured for the corresponding source file (in this case the Notebook parser) because translations might contain markups:

https://github.com/sphinx-doc/sphinx/blob/ae51974e217cc7590e1b9c68f18af3e043027c6d/sphinx/transforms/i18n.py#L72-L78

Perhaps fall back to MyST-Parser in this case?

Reproduce the bug

https://github.com/tonywu6/issues-repro-myst-nb-intl

List your environment

python==3.11
sphinx==7.2.6
docutils==0.20.1
myst-nb==1.0.0
myst-parser==2.0.0
welcome[bot] commented 3 months ago

Thanks for opening your first issue here! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.
If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).
Welcome to the EBP community! :tada:

OriolAbril commented 1 month ago

@tonywu6 I have added a PR #600 which should fix this. It would be great if you could try it out and ensure it works for you too.

tonywu6 commented 1 month ago

@OriolAbril Can confirm that i18n is working as expected with this patch (tried on some of our repos)