search

rstcheck / rstcheck-core

Core library behind rstcheck.
http://rstcheck-core.rtfd.io/
MIT License
4 stars 8 forks source link

[Bug]: Macos-specific tests failing on darwin in 1.0.2 #19

Closed viraptor closed 1 year ago

viraptor commented 2 years ago

To Dos

Example Code (python)

Failing tests:
- test_error_without_config_file_macos
- test_file_1_is_bad_without_config_macos
- test_error_without_config_file_macos
- test_file_1_is_bad_without_config_macos

Relevant log output

=================================== FAILURES ===================================
__________ TestWithoutConfigFile.test_error_without_config_file_macos __________

    @staticmethod
    @pytest.mark.skipif(sys.platform != "darwin", reason="MacOS specific error count")
    def test_error_without_config_file_macos() -> None:
        """Test bad example without set config file and implicit config file shows errors.

        On MacOS the cpp code block generates an additional error compared to linux:
        ``(ERROR/3) (cpp) warning: no newline at end of file [-Wnewline-eof]``
        """
        test_file = EXAMPLES_DIR / "without_configuration" / "bad.rst"
        init_config = config.RstcheckConfig()

        result = checker.check_file(test_file, init_config)

>       assert len(result) == 7
E       assert 6 == 7
E        +  where 6 = len([{'line_number': 15, 'message': "(cpp) error: use of undeclared identifier 'x'", 'source_origin': PosixPath('/private/...('/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/source/testing/examples/without_configuration/bad.rst')}])

init_config = RstcheckConfig(report_level=None, ignore_directives=None, ignore_roles=None, ignore_substitutions=None, ignore_languages=None, ignore_messages=None, config_path=None, recursive=None, warn_unknown_settings=None)
result     = [{'line_number': 15, 'message': "(cpp) error: use of undeclared identifier 'x'", 'source_origin': PosixPath('/private/...('/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/source/testing/examples/without_configuration/bad.rst')}]
test_file  = PosixPath('/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/source/testing/examples/without_configuration/bad.rst')

tests/integration_tests/test_file_checker.py:149: AssertionError
__________ TestWithConfigFile.test_file_1_is_bad_without_config_macos __________

    @staticmethod
    @pytest.mark.skipif(sys.platform != "darwin", reason="MacOS specific error count")
    def test_file_1_is_bad_without_config_macos() -> None:
        """Test bad file ``bad.rst`` without config file is not ok.

        On MacOS the cpp code block generates an additional error compared to linux:
        ``(ERROR/3) (cpp) warning: no newline at end of file [-Wnewline-eof]``
        """
        test_file = EXAMPLES_DIR / "with_configuration" / "bad.rst"
        init_config = config.RstcheckConfig(config_path=pathlib.Path("NONE"))

        result = checker.check_file(test_file, init_config)

>       assert len(result) == 7
E       assert 6 == 7
E        +  where 6 = len([{'line_number': 13, 'message': "(cpp) error: use of undeclared identifier 'x'", 'source_origin': PosixPath('/private/...ath('/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/source/testing/examples/with_configuration/bad.rst')}])

init_config = RstcheckConfig(report_level=None, ignore_directives=None, ignore_roles=None, ignore_substitutions=None, ignore_languages=None, ignore_messages=None, config_path=PosixPath('NONE'), recursive=None, warn_unknown_settings=None)
result     = [{'line_number': 13, 'message': "(cpp) error: use of undeclared identifier 'x'", 'source_origin': PosixPath('/private/...ath('/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/source/testing/examples/with_configuration/bad.rst')}]
test_file  = PosixPath('/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/source/testing/examples/with_configuration/bad.rst')

tests/integration_tests/test_file_checker.py:218: AssertionError
__________ TestWithoutConfigFile.test_error_without_config_file_macos __________

capsys = <_pytest.capture.CaptureFixture object at 0x1115bb640>

    @staticmethod
    @pytest.mark.skipif(sys.platform != "darwin", reason="MacOS specific error count")
    def test_error_without_config_file_macos(capsys: pytest.CaptureFixture[str]) -> None:
        """Test bad example without set config file and implicit config file shows errors.

        On MacOS the cpp code block generates an additional error compared to linux:
        ``(ERROR/3) (cpp) warning: no newline at end of file [-Wnewline-eof]``
        """
        test_file = EXAMPLES_DIR / "without_configuration" / "bad.rst"
        init_config = config.RstcheckConfig()
        _runner = runner.RstcheckMainRunner(check_paths=[test_file], rstcheck_config=init_config)

        result = _runner.run()

        assert result != 0
>       assert len(ERROR_CODE_REGEX.findall(capsys.readouterr().err)) == 7
E       assert 6 == 7
E        +  where 6 = len(['(ERROR/3)', '(WARNING/2)', '(INFO/1)', '(ERROR/3)', '(INFO/1)', '(ERROR/3)'])
E        +    where ['(ERROR/3)', '(WARNING/2)', '(INFO/1)', '(ERROR/3)', '(INFO/1)', '(ERROR/3)'] = <built-in method findall of re.Pattern object at 0x10f4dfb50>('/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/tmpnhix3cjs.cpp:15: (ERROR/3) (cpp) error: use of undecla...les/without_configuration/bad.rst:9: (ERROR/3) Unknown interpreted text role "custom-role".\nError! Issues detected.\n')
E        +      where <built-in method findall of re.Pattern object at 0x10f4dfb50> = re.compile('\\([A-Z]*?/\\d\\)').findall
E        +      and   '/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/tmpnhix3cjs.cpp:15: (ERROR/3) (cpp) error: use of undecla...les/without_configuration/bad.rst:9: (ERROR/3) Unknown interpreted text role "custom-role".\nError! Issues detected.\n' = CaptureResult(out='', err='/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/tmpnhix3cjs.cpp:15: (ERROR/3) (...es/without_configuration/bad.rst:9: (ERROR/3) Unknown interpreted text role "custom-role".\nError! Issues detected.\n').err
E        +        where CaptureResult(out='', err='/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/tmpnhix3cjs.cpp:15: (ERROR/3) (...es/without_configuration/bad.rst:9: (ERROR/3) Unknown interpreted text role "custom-role".\nError! Issues detected.\n') = <bound method CaptureFixture.readouterr of <_pytest.capture.CaptureFixture object at 0x1115bb640>>()
E        +          where <bound method CaptureFixture.readouterr of <_pytest.capture.CaptureFixture object at 0x1115bb640>> = <_pytest.capture.CaptureFixture object at 0x1115bb640>.readouterr

_runner    = <rstcheck_core.runner.RstcheckMainRunner object at 0x10ff5c130>
capsys     = <_pytest.capture.CaptureFixture object at 0x1115bb640>
init_config = RstcheckConfig(report_level=None, ignore_directives=None, ignore_roles=None, ignore_substitutions=None, ignore_languages=None, ignore_messages=None, config_path=None, recursive=None, warn_unknown_settings=None)
result     = 1
test_file  = PosixPath('/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/source/testing/examples/without_configuration/bad.rst')

tests/integration_tests/test_runner.py:243: AssertionError
__________ TestWithConfigFile.test_file_1_is_bad_without_config_macos __________

capsys = <_pytest.capture.CaptureFixture object at 0x111337340>

    @staticmethod
    @pytest.mark.skipif(sys.platform != "darwin", reason="MacOS specific error count")
    def test_file_1_is_bad_without_config_macos(capsys: pytest.CaptureFixture[str]) -> None:
        """Test bad file ``bad.rst`` without config file is not ok.

        On MacOS the cpp code block generates an additional error compared to linux:
        ``(ERROR/3) (cpp) warning: no newline at end of file [-Wnewline-eof]``
        """
        test_file = EXAMPLES_DIR / "with_configuration" / "bad.rst"
        config_file = pathlib.Path("NONE")
        init_config = config.RstcheckConfig(config_path=config_file)
        _runner = runner.RstcheckMainRunner(check_paths=[test_file], rstcheck_config=init_config)

        result = _runner.run()

        assert result != 0
>       assert len(ERROR_CODE_REGEX.findall(capsys.readouterr().err)) == 7
E       assert 6 == 7
E        +  where 6 = len(['(ERROR/3)', '(WARNING/2)', '(INFO/1)', '(ERROR/3)', '(INFO/1)', '(ERROR/3)'])
E        +    where ['(ERROR/3)', '(WARNING/2)', '(INFO/1)', '(ERROR/3)', '(INFO/1)', '(ERROR/3)'] = <built-in method findall of re.Pattern object at 0x10f4dfb50>('/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/tmpdkk4rrv9.cpp:13: (ERROR/3) (cpp) error: use of undecla...amples/with_configuration/bad.rst:7: (ERROR/3) Unknown interpreted text role "custom-role".\nError! Issues detected.\n')
E        +      where <built-in method findall of re.Pattern object at 0x10f4dfb50> = re.compile('\\([A-Z]*?/\\d\\)').findall
E        +      and   '/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/tmpdkk4rrv9.cpp:13: (ERROR/3) (cpp) error: use of undecla...amples/with_configuration/bad.rst:7: (ERROR/3) Unknown interpreted text role "custom-role".\nError! Issues detected.\n' = CaptureResult(out='', err='/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/tmpdkk4rrv9.cpp:13: (ERROR/3) (...mples/with_configuration/bad.rst:7: (ERROR/3) Unknown interpreted text role "custom-role".\nError! Issues detected.\n').err
E        +        where CaptureResult(out='', err='/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/tmpdkk4rrv9.cpp:13: (ERROR/3) (...mples/with_configuration/bad.rst:7: (ERROR/3) Unknown interpreted text role "custom-role".\nError! Issues detected.\n') = <bound method CaptureFixture.readouterr of <_pytest.capture.CaptureFixture object at 0x111337340>>()
E        +          where <bound method CaptureFixture.readouterr of <_pytest.capture.CaptureFixture object at 0x111337340>> = <_pytest.capture.CaptureFixture object at 0x111337340>.readouterr

_runner    = <rstcheck_core.runner.RstcheckMainRunner object at 0x10ff8f3a0>
capsys     = <_pytest.capture.CaptureFixture object at 0x111337340>
config_file = PosixPath('NONE')
init_config = RstcheckConfig(report_level=None, ignore_directives=None, ignore_roles=None, ignore_substitutions=None, ignore_languages=None, ignore_messages=None, config_path=PosixPath('NONE'), recursive=None, warn_unknown_settings=None)
result     = 1
test_file  = PosixPath('/private/tmp/nix-build-python3.10-rstcheck-core-1.0.2.drv-0/source/testing/examples/with_configuration/bad.rst')

tests/integration_tests/test_runner.py:319: AssertionError

Description

Found in the automatic tests while packaging for nixpkgs.

Operating System

macOS

Operating System Details

nixpkgs packaging CI

Python Version

3.10.8

rstcheck Version

1.0.2

Additional Context

https://hydra.nixos.org/build/197455312/nixlog/1

Cielquan commented 2 years ago

Because I do not have a mac it will be hard for me to help. I setup the tests via the GHA CI pipeline by trial and error as good as I could.

Which macOS version is used in your pipeline. I could not find any in the link.

Because the GHA pipeline uses macos-latest which is as of today 12.6.1 (see this GHA run) and used 11.6.8 before (see this GHA run). Both version/GHA runs are fully green.

EDIT: fixed typo

viraptor commented 2 years ago

Effectively 10 (not really, but the sdk comes from 10, because reasons :( ) If this helps, the error for testing/examples/without_configuration/bad.rst is:

>>> pprint.pp(res)
[{'source_origin': PosixPath('/tmp/tmpswma28tr.cpp'),
  'line_number': 15,
  'message': "(cpp) error: use of undeclared identifier 'x'"},
 {'source_origin': PosixPath('/tmp/rstcheck-core/testing/examples/without_configuration/bad.rst'),
  'line_number': 4,
  'message': '(WARNING/2) Title underline too short.'},
 {'source_origin': PosixPath('/tmp/rstcheck-core/testing/examples/without_configuration/bad.rst'),
  'line_number': 6,
  'message': '(INFO/1) No directive entry for "custom-directive" in module '
             '"docutils.parsers.rst.languages.en".'},
 {'source_origin': PosixPath('/tmp/rstcheck-core/testing/examples/without_configuration/bad.rst'),
  'line_number': 6,
  'message': '(ERROR/3) Unknown directive type "custom-directive".'},
 {'source_origin': PosixPath('/tmp/rstcheck-core/testing/examples/without_configuration/bad.rst'),
  'line_number': 9,
  'message': '(INFO/1) No role entry for "custom-role" in module '
             '"docutils.parsers.rst.languages.en".'},
 {'source_origin': PosixPath('/tmp/rstcheck-core/testing/examples/without_configuration/bad.rst'),
  'line_number': 9,
  'message': '(ERROR/3) Unknown interpreted text role "custom-role".'}]

which means that with the expected ['(ERROR/3)', '(WARNING/2)', '(INFO/1)', '(ERROR/3)', '(INFO/1)', '(ERROR/3)']), we're missing the first error. On a matching version that's:

E         + /var/folders/vk/lksjdszd34q5hwqfvhhvp7380000gp/T/tmp4m27urp5.cpp:16: (ERROR/3) (cpp) warning: no newline at end of file [-Wnewline-eof]

So if I understand correctly, the newline-eof error/warning is version specific and doesn't appear before 11.

Cielquan commented 1 year ago

I hope to find time on the weekend to have another look at this.

And when the one additional error only comes on macos 11 and later it should be easy to fix.

Cielquan commented 1 year ago

We are lucky macos 10 is supported by GHA until December this year: https://github.com/actions/runner-images/issues/5583

Currently testing stuff.

EDIT: Well the waiting time for a runner is very long .. 30 min and still not started: https://github.com/rstcheck/rstcheck-core/actions/runs/3498142891

Cielquan commented 1 year ago

Because the macos 10 GHA CI still did not start (>2h later) I made some changes for the failing tests on the test-me-macos-10 branch based on your provided information. I used platform.mac_ver() for version detection.

Then I ran it in CI for macos 11 and 12 (which run) and it failed for all python versions except 3.11, because python < 3.11 detects the macos version wrong?!

Here is a CI run to shows what I mean (look at the '##### DO STUFF' steps): https://github.com/rstcheck/rstcheck-core/actions/runs/3499447749

I ran the following commands in CI and below are the results.

python -c 'import sys;print(f"{sys.version=}")'
python -c 'import platform;print(f"{platform.system()=}")'
python -c 'import platform;print(f"{platform.release()=}")'
python -c 'import platform;print(f"{platform.version()=}")'
python -c 'import platform;print(f"{platform.platform()=}")'
python -c 'import platform;print(f"{platform.mac_ver()=}")'

macos 12 & py 3.8

sys.version='3.8.14 (default, Sep  7 2022, 14:26:37) \n[Clang 12.0.0 (clang-1200.0.32.29)]'
platform.system()='Darwin'
platform.release()='21.6.0'
platform.version()='Darwin Kernel Version 21.6.0: Thu Sep 29 20:12:57 PDT 2022; root:xnu-8020.240.7~1/RELEASE_X86_64'
platform.platform()='macOS-10.16-x86_64-i386-64bit'
platform.mac_ver()=('10.16', ('', '', ''), 'x86_64')

macos 12 & py 3.11

sys.version='3.11.0 (v3.11.0:deaf509e8f, Oct 24 2022, 14:43:23) [Clang 13.0.0 (clang-1300.0.29.30)]'
platform.system()='Darwin'
platform.release()='21.6.0'
platform.version()='Darwin Kernel Version 21.6.0: Thu Sep 29 20:12:57 PDT 2022; root:xnu-8020.240.7~1/RELEASE_X86_64'
platform.platform()='macOS-12.6.1-x86_64-i386-64bit'
platform.mac_ver()=('12.6.1', ('', '', ''), 'x86_64')

The result is the same for macos 11 and 12 && for python 3.8, 3.9 and 3.10. In all cases the macos version is wrongly 10.16. Only python 3.11 detects macos 12 and 11.

Then I saw that macos 11 started with kernel version 20, which can be detected with platform.release(), so I changed the check to use the kernel version instead and that worked then like it should locally on linux+py310 and in CI some jobs are green so I guess the rest will be too: https://github.com/rstcheck/rstcheck-core/actions/runs/3499574430.

@viraptor Could you test the test-me-macos-10 branch.

tjni commented 1 year ago

I stumbled upon this today in nixpkgs and decided to take a look.

First of all, @Cielquan, thank you for the time you have already put into this issue and for creating the test branch. It does not fix the issue in nixpkgs, because I suspect it's not related to macOS version at all.

The nix package manager builds everything from source. This means that, prior to building rstcheck-core, it builds clang from source and specifies it in the CXX environment variable. On the other hand, when building on macOS normally, the check_cpp method will invoke g++, which invokes Apple's clang compiler. It seems that Apple's reports -Wnewline-eof when invoked with -pedantic but nixpkg's does not.

I suggest that we side step this (controversial) corner case of the C specification by adding a newline to the end of the code block source prior to syntax checking. What do you think?

viraptor commented 1 year ago

Sorry @Cielquan, I totally missed your last message :( Confirming what tjni said - the mentioned branch did not fix the issue unfortunately.

(happy to do more testing as required)

Cielquan commented 1 year ago

I will hopefully have time to look at this again within the next 2 weeks. I will report back then.