executablebooks / MyST-Parser

An extended commonmark compliant parser, with bridges to docutils/sphinx
https://myst-parser.readthedocs.io
MIT License
753 stars 197 forks source link

1.0.0 + 80b892ff: pytest is failing in `tests/test_renderers/test_fixtures_docutils.py::test_syntax_extensions[104-replacements]` unit #780

Closed kloczek closed 5 months ago

kloczek commented 1 year ago

What version of myst-parser are you using?

1.0.0 + all commits up to 80b892ff

What version dependencies are you using?

Here is list of installed modules in build env

```console Package Version ----------------------------- ------- alabaster 0.7.13 asttokens 2.2.1 Babel 2.12.1 backcall 0.2.0 beautifulsoup4 4.12.2 build 0.10.0 charset-normalizer 3.1.0 decorator 5.1.1 distro 1.8.0 docutils 0.19 exceptiongroup 1.1.1 executing 1.2.0 gpg 1.20.0 idna 3.4 imagesize 1.4.1 importlib-metadata 6.6.0 iniconfig 2.0.0 installer 0.7.0 ipython 8.12.0 jedi 0.18.2 Jinja2 3.1.2 libcomps 0.1.19 linkify-it-py 2.0.2 markdown-it-py 2.2.0 MarkupSafe 2.1.2 matplotlib-inline 0.1.6 mdit-py-plugins 0.4.0 mdurl 0.1.2 packaging 23.1 parso 0.8.3 pexpect 4.8.0 pickleshare 0.7.5 pluggy 1.0.0 prompt-toolkit 3.0.38 ptyprocess 0.7.0 pure-eval 0.2.2 Pygments 2.15.1 pyproject_hooks 1.0.0 pytest 7.3.1 pytest-datadir 1.4.1 pytest_param_files 0.3.4 pytest-regressions 2.4.2 python-dateutil 2.8.2 pytz 2023.2 PyYAML 6.0 requests 2.30.0 setuptools 67.7.2 six 1.16.0 snowballstemmer 2.2.0 soupsieve 2.4.1 Sphinx 6.2.1 sphinx_pytest 0.1.1 sphinxcontrib-applehelp 1.0.4 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 stack-data 0.6.2 tomli 2.0.1 traitlets 5.9.0 typing_extensions 4.5.0 uc-micro-py 1.0.2 urllib3 1.26.15 wcwidth 0.2.6 wheel 0.40.0 zipp 3.15.0 ```

What operating system are you using?

Mac

Describe the Bug

Looks like MyST-Parser from current master fails in one unit.

Expected Behavior

pytest should not fail

To Reproduce

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-myst-parser-1.0.0~no_loopy_deps-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-myst-parser-1.0.0~no_loopy_deps-2.fc35.x86_64/usr/lib/python3.8/site-packages + /usr/bin/pytest -ra -m 'not network' ============================= test session starts ============================== platform linux -- Python 3.8.16, pytest-7.3.1, pluggy-1.0.0 rootdir: /home/tkloczko/rpmbuild/BUILD/MyST-Parser-1.0.0 configfile: pyproject.toml plugins: datadir-1.4.1, sphinx_pytest-0.1.1, regressions-2.4.2, pytest_param_files-0.3.4 collected 1074 items tests/test_anchors.py . [ 0%] tests/test_docutils.py ........... [ 1%] tests/test_inventory.py .............. [ 2%] tests/test_commonmark/test_commonmark.py .............s................. [ 5%] ..................................s.s................................... [ 12%] ........................................................................ [ 18%] ........................................................................ [ 25%] ........................................................................ [ 32%] ........................................................................ [ 38%] ........................................................................ [ 45%] ........................................................................ [ 52%] ........................................................................ [ 58%] .......................................... [ 62%] tests/test_html/test_html_to_nodes.py ................. [ 64%] tests/test_html/test_parse_html.py ..................... [ 66%] tests/test_renderers/test_error_reporting.py ................. [ 67%] tests/test_renderers/test_fixtures_docutils.py ......................... [ 70%] .............................................s.......................F.. [ 77%] [ 77%] tests/test_renderers/test_fixtures_sphinx.py ........................... [ 79%] ...........................................s...........s.s.........s..s. [ 86%] ...............ss..............................s........................ [ 92%] .... [ 93%] tests/test_renderers/test_include_directive.py ........ [ 94%] tests/test_renderers/test_myst_config.py ....................... [ 96%] tests/test_renderers/test_myst_refs.py ......... [ 97%] tests/test_renderers/test_parse_directives.py ................ [ 98%] tests/test_sphinx/test_sphinx_builds.py ................ [100%] =================================== FAILURES =================================== ___________________ test_syntax_extensions[104-replacements] ___________________ file_params = ParamTestData(line=104, title='replacements', description='--myst-enable-extensions=replacements', content='(c) (C) (r...agraph>\n © © ® ® ™ ™ (p) (P) ± …\n', index=5, fmt=) @pytest.mark.param_file(FIXTURE_PATH / "docutil_syntax_extensions.txt") def test_syntax_extensions(file_params): """The description is parsed as a docutils commandline""" settings = settings_from_cmdline(file_params.description) report_stream = StringIO() settings["warning_stream"] = report_stream doctree = publish_doctree( file_params.content, parser=Parser(), settings_overrides=settings, ) > file_params.assert_expected(doctree.pformat(), rstrip_lines=True) E AssertionError: Actual does not match expected E --- /home/tkloczko/rpmbuild/BUILD/MyST-Parser-1.0.0/tests/test_renderers/fixtures/docutil_syntax_extensions.txt:104 E +++ (actual) E @@ -1,3 +1,3 @@ E E E E E - © © ® ® ™ ™ (p) (P) ± … E + © © ® ® ™ ™ § § ± … tests/test_renderers/test_fixtures_docutils.py:123: AssertionError =============================== warnings summary =============================== tests/test_docutils.py: 11 warnings tests/test_renderers/test_error_reporting.py: 18 warnings tests/test_renderers/test_fixtures_docutils.py: 96 warnings tests/test_renderers/test_fixtures_sphinx.py: 169 warnings tests/test_renderers/test_include_directive.py: 8 warnings tests/test_renderers/test_myst_config.py: 23 warnings tests/test_renderers/test_myst_refs.py: 9 warnings tests/test_sphinx/test_sphinx_builds.py: 58 warnings /home/tkloczko/rpmbuild/BUILDROOT/python-myst-parser-1.0.0~no_loopy_deps-2.fc35.x86_64/usr/lib/python3.8/site-packages/myst_parser/mdit_to_docutils/base.py:66: DeprecationWarning: The frontend.OptionParser class will be replaced by a subclass of argparse.ArgumentParser in Docutils 0.21 or later. settings = OptionParser(components=(parser_cls,)).get_default_values() tests/test_docutils.py: 864 warnings tests/test_renderers/test_error_reporting.py: 1224 warnings tests/test_renderers/test_fixtures_docutils.py: 7448 warnings tests/test_renderers/test_fixtures_sphinx.py: 11492 warnings tests/test_renderers/test_include_directive.py: 544 warnings tests/test_renderers/test_myst_config.py: 3680 warnings tests/test_renderers/test_myst_refs.py: 612 warnings tests/test_sphinx/test_sphinx_builds.py: 3944 warnings /usr/lib64/python3.8/optparse.py:1000: DeprecationWarning: The frontend.Option class will be removed in Docutils 0.21 or later. option = self.option_class(*args, **kwargs) tests/test_docutils.py::test_help_text /home/tkloczko/rpmbuild/BUILD/MyST-Parser-1.0.0/tests/test_docutils.py:97: DeprecationWarning: The frontend.OptionParser class will be replaced by a subclass of argparse.ArgumentParser in Docutils 0.21 or later. OptionParser(components=(Parser,)).print_help(stream) tests/test_renderers/test_fixtures_docutils.py: 10 warnings /home/tkloczko/rpmbuild/BUILD/MyST-Parser-1.0.0/tests/test_renderers/test_fixtures_docutils.py:131: DeprecationWarning: Publisher.setup_option_parser is deprecated, and will be removed in Docutils 0.21. option_parser = pub.setup_option_parser() tests/test_renderers/test_fixtures_docutils.py: 10 warnings tests/test_renderers/test_myst_config.py: 23 warnings /usr/lib/python3.8/site-packages/docutils/core.py:118: DeprecationWarning: The frontend.OptionParser class will be replaced by a subclass of argparse.ArgumentParser in Docutils 0.21 or later. return OptionParser( tests/test_renderers/test_fixtures_sphinx.py::test_syntax_elements[1-Raw] tests/test_renderers/test_fixtures_sphinx.py::test_syntax_elements[1-Raw] /usr/lib/python3.8/site-packages/sphinxcontrib/htmlhelp/__init__.py:26: RemovedInSphinx80Warning: The alias 'sphinx.util.progress_message' is deprecated, use 'sphinx.http_date.epoch_to_rfc1123' instead. Check CHANGES for Sphinx API modifications. from sphinx.util import progress_message tests/test_renderers/test_include_directive.py::test_render[51-Include code:] /home/tkloczko/rpmbuild/BUILD/MyST-Parser-1.0.0/tests/test_renderers/test_include_directive.py:29: PendingDeprecationWarning: nodes.Node.traverse() is obsoleted by Node.findall(). for node in doctree.traverse(): tests/test_renderers/test_myst_config.py: 23 warnings /home/tkloczko/rpmbuild/BUILD/MyST-Parser-1.0.0/tests/test_renderers/test_myst_config.py:25: DeprecationWarning: Publisher.setup_option_parser is deprecated, and will be removed in Docutils 0.21. option_parser = pub.setup_option_parser() -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================== short test summary info ============================ SKIPPED [1] tests/test_commonmark/test_commonmark.py:23: Expects '+++' to be unconverted (not block break). SKIPPED [2] tests/test_commonmark/test_commonmark.py:28: Thematic breaks on the first line conflict with front matter syntax SKIPPED [1] tests/test_renderers/test_fixtures_docutils.py:96: (`docutils.parsers.rst.directives.body.LineBlock`) SKIP: MockingError: MockState has not yet implemented attribute 'nest_line_block_lines' SKIPPED [2] tests/test_renderers/test_fixtures_sphinx.py:74: SKIP: Tested in sphinx builds SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:74: SKIP: MockingError: MockState has not yet implemented attribute 'nested_list_parse' SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:74: SPHINX4-SKIP productionlist (`sphinx.domains.std.ProductionList`): SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:74: SPHINX4-SKIP rst:directive:option (`sphinx.domains.rst.ReSTDirectiveOption`): SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:96: SKIP cpp:expr (`sphinx.domains.cpp.CPPExprRole`): SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:96: SKIP cpp:texpr (`sphinx.domains.cpp.CPPExprRole`): SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:96: SKIP: Non-deterministic output FAILED tests/test_renderers/test_fixtures_docutils.py::test_syntax_extensions[104-replacements] ```
kloczek commented 1 year ago

Just tested 2.0.0. Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-myst-parser-2.0.0~no_loopy_deps-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-myst-parser-2.0.0~no_loopy_deps-2.fc35.x86_64/usr/lib/python3.8/site-packages + /usr/bin/pytest -ra -m 'not network' ==================================================================================== test session starts ==================================================================================== platform linux -- Python 3.8.17, pytest-7.3.1, pluggy-1.0.0 rootdir: /home/tkloczko/rpmbuild/BUILD/MyST-Parser-2.0.0 configfile: pyproject.toml plugins: httpserver-1.0.4, mock-3.10.0, flaky-3.7.0, datadir-1.4.1, regressions-2.4.2, sphinx_pytest-0.1.1, pytest_param_files-0.3.4 collected 1074 items tests/test_anchors.py . [ 0%] tests/test_docutils.py ........... [ 1%] tests/test_inventory.py .............. [ 2%] tests/test_commonmark/test_commonmark.py .............s...................................................s.s........................................................................ [ 15%] ..................................................................................................................................................................................... [ 32%] ..................................................................................................................................................................................... [ 49%] ................................................................................................................................................... [ 62%] tests/test_html/test_html_to_nodes.py ................. [ 64%] tests/test_html/test_parse_html.py ..................... [ 66%] tests/test_renderers/test_error_reporting.py ................. [ 67%] tests/test_renderers/test_fixtures_docutils.py ......................................................................s.......................... [ 77%] tests/test_renderers/test_fixtures_sphinx.py ......................................................................s...........s.s.........s..s................ss.................... [ 89%] ..........s............................ [ 93%] tests/test_renderers/test_include_directive.py ........ [ 94%] tests/test_renderers/test_myst_config.py ....................... [ 96%] tests/test_renderers/test_myst_refs.py ......... [ 97%] tests/test_renderers/test_parse_directives.py ................ [ 98%] tests/test_sphinx/test_sphinx_builds.py ...........F.... [100%] ========================================================================================= FAILURES ========================================================================================== _____________________________________________________________________________________ test_gettext_html _____________________________________________________________________________________ app = , status = <_io.StringIO object at 0x7fe06b523ca0>, warning = <_io.StringIO object at 0x7fe06b4643a0> get_sphinx_app_doctree = .read at 0x7fe06b4681f0>, get_sphinx_app_output = .read at 0x7fe06b468310> @pytest.mark.sphinx( buildername="html", srcdir=os.path.join(SOURCE_DIR, "gettext"), freshenv=True, confoverrides={"language": "fr", "gettext_compact": False, "locale_dirs": ["."]}, ) def test_gettext_html( app, status, warning, get_sphinx_app_doctree, get_sphinx_app_output, ): """Test gettext message extraction.""" app.build() assert "build succeeded" in status.getvalue() # Build succeeded warnings = warning.getvalue().strip() assert warnings == "" try: get_sphinx_app_doctree( app, docname="index", regress=True, ) finally: get_sphinx_app_doctree( app, docname="index", resolve=True, regress=True, ) > get_sphinx_app_output( app, filename="index.html", regress_html=True, regress_ext=".html", ) tests/test_sphinx/test_sphinx_builds.py:455: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ app = , buildername = 'html', filename = 'index.html', encoding = 'utf-8', regress_html = True, regress_ext = '.html', replace = None def read( app, buildername="html", filename="index.html", encoding="utf-8", regress_html=False, regress_ext=".html", replace=None, ): outpath = path(os.path.join(str(app.srcdir), "_build", buildername, filename)) if not outpath.exists(): raise OSError(f"no output file exists: {outpath}") try: # introduced in sphinx 3.0 content = outpath.read_text(encoding=encoding) except AttributeError: content = outpath.text(encoding=encoding) if regress_html: # only regress the inner body, since other sections are non-deterministic soup = BeautifulSoup(content, "html.parser") doc_div = soup.findAll("div", {"class": "documentwrapper"})[0] # pygments 2.11.0 introduces a whitespace tag for pygment_whitespace in doc_div.select("pre > span.w"): pygment_whitespace.replace_with(pygment_whitespace.text) text = doc_div.prettify() for find, rep in (replace or {}).items(): text = text.replace(find, rep) > file_regression.check(text, extension=regress_ext, encoding="utf8") E AssertionError: FILES DIFFER: E /tmp/pytest-of-tkloczko/pytest-7/test_gettext_html0/test_sphinx_builds/test_gettext_html.html E /tmp/pytest-of-tkloczko/pytest-7/test_gettext_html0/test_sphinx_builds/test_gettext_html.obtained.html E HTML DIFF: /tmp/pytest-of-tkloczko/pytest-7/test_gettext_html0/test_sphinx_builds/test_gettext_html.obtained.diff.html E --- E +++ E @@ -7,7 +7,7 @@ E E gras E E - E + E ¶ E E tests/test_sphinx/conftest.py:88: AssertionError --------------------------------------------------------------------------------- Captured stdout teardown ---------------------------------------------------------------------------------- # testroot: root # builder: html # srcdir: /home/tkloczko/rpmbuild/BUILD/MyST-Parser-2.0.0/tests/test_sphinx/sourcedirs/gettext # outdir: /home/tkloczko/rpmbuild/BUILD/MyST-Parser-2.0.0/tests/test_sphinx/sourcedirs/gettext/_build/html # status: Running Sphinx v6.2.1 loading translations [fr]... not available for built-in messages myst v2.0.0: MdParserConfig(commonmark_only=False, gfm_only=False, enable_extensions={'deflist'}, disable_syntax=[], all_links_external=False, url_schemes=('http', 'https', 'mailto', 'ftp'), ref_domains=None, fence_as_directive=set(), number_code_blocks=[], title_to_header=False, heading_anchors=0, heading_slug_func=None, html_meta={}, footnote_transition=True, words_per_minute=200, substitutions={}, linkify_fuzzy_links=True, dmath_allow_labels=True, dmath_allow_space=True, dmath_allow_digits=True, dmath_double_inline=False, update_mathjax=True, mathjax_classes='tex2jax_process|mathjax_process|math|output_area', enable_checkboxes=False, suppress_warnings=[], highlight_code_blocks=True) building [mo]: targets for 1 po files that are out of date writing output... [100%] fr/LC_MESSAGES/index.mo building [html]: targets for 1 source files that are out of date updating environment: [new config] 1 added, 0 changed, 0 removed reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index generating indices... genindex done writing additional pages... search done copying images... [100%] fun-fish.png copying static files... done copying extra files... done dumping search index in French (code: fr)... done dumping object inventory... done build succeeded. The HTML pages are in tests/test_sphinx/sourcedirs/gettext/_build/html. # warning: ===================================================================================== warnings summary ====================================================================================== tests/test_docutils.py: 11 warnings tests/test_renderers/test_error_reporting.py: 18 warnings tests/test_renderers/test_fixtures_docutils.py: 96 warnings tests/test_renderers/test_fixtures_sphinx.py: 169 warnings tests/test_renderers/test_include_directive.py: 8 warnings tests/test_renderers/test_myst_config.py: 23 warnings tests/test_renderers/test_myst_refs.py: 9 warnings tests/test_sphinx/test_sphinx_builds.py: 58 warnings /home/tkloczko/rpmbuild/BUILDROOT/python-myst-parser-2.0.0~no_loopy_deps-2.fc35.x86_64/usr/lib/python3.8/site-packages/myst_parser/mdit_to_docutils/base.py:66: DeprecationWarning: The frontend.OptionParser class will be replaced by a subclass of argparse.ArgumentParser in Docutils 0.21 or later. settings = OptionParser(components=(parser_cls,)).get_default_values() tests/test_docutils.py: 864 warnings tests/test_renderers/test_error_reporting.py: 1224 warnings tests/test_renderers/test_fixtures_docutils.py: 7448 warnings tests/test_renderers/test_fixtures_sphinx.py: 11492 warnings tests/test_renderers/test_include_directive.py: 544 warnings tests/test_renderers/test_myst_config.py: 3680 warnings tests/test_renderers/test_myst_refs.py: 612 warnings tests/test_sphinx/test_sphinx_builds.py: 3944 warnings /usr/lib64/python3.8/optparse.py:1000: DeprecationWarning: The frontend.Option class will be removed in Docutils 0.21 or later. option = self.option_class(*args, **kwargs) tests/test_docutils.py::test_help_text /home/tkloczko/rpmbuild/BUILD/MyST-Parser-2.0.0/tests/test_docutils.py:97: DeprecationWarning: The frontend.OptionParser class will be replaced by a subclass of argparse.ArgumentParser in Docutils 0.21 or later. OptionParser(components=(Parser,)).print_help(stream) tests/test_renderers/test_fixtures_docutils.py: 10 warnings /home/tkloczko/rpmbuild/BUILD/MyST-Parser-2.0.0/tests/test_renderers/test_fixtures_docutils.py:131: DeprecationWarning: Publisher.setup_option_parser is deprecated, and will be removed in Docutils 0.21. option_parser = pub.setup_option_parser() tests/test_renderers/test_fixtures_docutils.py: 10 warnings tests/test_renderers/test_myst_config.py: 23 warnings /usr/lib/python3.8/site-packages/docutils/core.py:118: DeprecationWarning: The frontend.OptionParser class will be replaced by a subclass of argparse.ArgumentParser in Docutils 0.21 or later. return OptionParser( tests/test_renderers/test_fixtures_sphinx.py::test_syntax_elements[1-Raw] tests/test_renderers/test_fixtures_sphinx.py::test_syntax_elements[1-Raw] /usr/lib/python3.8/site-packages/sphinxcontrib/htmlhelp/__init__.py:26: RemovedInSphinx80Warning: The alias 'sphinx.util.progress_message' is deprecated, use 'sphinx.http_date.epoch_to_rfc1123' instead. Check CHANGES for Sphinx API modifications. from sphinx.util import progress_message tests/test_renderers/test_include_directive.py::test_render[51-Include code:] /home/tkloczko/rpmbuild/BUILD/MyST-Parser-2.0.0/tests/test_renderers/test_include_directive.py:29: PendingDeprecationWarning: nodes.Node.traverse() is obsoleted by Node.findall(). for node in doctree.traverse(): tests/test_renderers/test_myst_config.py: 23 warnings /home/tkloczko/rpmbuild/BUILD/MyST-Parser-2.0.0/tests/test_renderers/test_myst_config.py:25: DeprecationWarning: Publisher.setup_option_parser is deprecated, and will be removed in Docutils 0.21. option_parser = pub.setup_option_parser() -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================================================== short test summary info ================================================================================== SKIPPED [1] tests/test_commonmark/test_commonmark.py:23: Expects '+++' to be unconverted (not block break). SKIPPED [2] tests/test_commonmark/test_commonmark.py:28: Thematic breaks on the first line conflict with front matter syntax SKIPPED [1] tests/test_renderers/test_fixtures_docutils.py:96: (`docutils.parsers.rst.directives.body.LineBlock`) SKIP: MockingError: MockState has not yet implemented attribute 'nest_line_block_lines' SKIPPED [2] tests/test_renderers/test_fixtures_sphinx.py:74: SKIP: Tested in sphinx builds SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:74: SKIP: MockingError: MockState has not yet implemented attribute 'nested_list_parse' SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:74: SPHINX4-SKIP productionlist (`sphinx.domains.std.ProductionList`): SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:74: SPHINX4-SKIP rst:directive:option (`sphinx.domains.rst.ReSTDirectiveOption`): SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:96: SKIP cpp:expr (`sphinx.domains.cpp.CPPExprRole`): SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:96: SKIP cpp:texpr (`sphinx.domains.cpp.CPPExprRole`): SKIPPED [1] tests/test_renderers/test_fixtures_sphinx.py:96: SKIP: Non-deterministic output FAILED tests/test_sphinx/test_sphinx_builds.py::test_gettext_html - AssertionError: FILES DIFFER: ================================================================ 1 failed, 1061 passed, 12 skipped, 30270 warnings in 13.43s ================================================================ ```