Closed CleanCut closed 5 months ago
@CleanCut can you point me to the issue with Python 3.11 on macOS ? I pulled the 3.5.0 branch on our side and the GH action stage is passing as expected.
It's a problem that pops up on my local mac when I run make release
. Dang, that makes it nearly impossible for you to debug if it doesn't do it in CI as well. Sorry about that.
I'll see what I can figure out the next time I have some free time. Dunno exactly when that will be.
I've switched my local virtualenv to a fresh Python 3.11.6 and on the 3.5.0 branch. I got this error:
make release
...
source venv-installed/bin/activate; green -vvvv green
Green 3.5.0, Coverage 7.3.2, Python 3.11.6
...
Ran 233 tests in 9.006s using 10 processes
OK (passes=231, skips=2)
(test-installed) completed
make test-versions
# Run the in-place stub under all python versions in the path
./test_versions
Identified python versions: 3.11 3.12
...
[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip
pyenv: python3.12: command not found
The `python3.12' command exists in these Python versions:
3.12.0
Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
Please install pip under python3.12
make[1]: *** [test-versions] Error 5
make: *** [test-local] Error 2
The workaround was for me was to temporarily force PYTHON_VERSIONS=3.11
in the test_versions
script.
Then I got this, which is what I suppose you are referring to:
Ran 233 tests in 6.038s using 10 processes
Coverage of 35% is below minimum level of 100%
FAILED (passes=231, skips=2)
Coverage of 35% is below minimum level of 100%
make: *** [sanity-checks] Error 1
I get loads of these errors:
/Users/stephane/.pyenv/versions/green311/lib/python3.11/site-packages/coverage/report_core.py:115: CoverageWarning: Couldn't parse '/private/var/folders/z7/tq_3ftkd2wv74hzkjgvv2kcr0000gp/T/tmpqwwtok45/tmpcwan2e4g/tmpyje6dije/tmp_a2qbahv/existing_module.py': No source for code: '/private/var/folders/z7/tq_3ftkd2wv74hzkjgvv2kcr0000gp/T/tmpqwwtok45/tmpcwan2e4g/tmpyje6dije/tmp_a2qbahv/existing_module.py'. (couldnt-parse)
The problem seems to be that the temp folder has been cleaned up by the time coverage.py is trying to read from it.
I'm able to see the 35% with just make test-coverage
.
---------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 14610 9519 35%
Ran 233 tests in 5.667s using 10 processes
OK (passes=231, skips=2)
(test-coverage) completed
I'll look into it.
I simplified the command to reproduce to: PYTHONPATH="." python3 -m green.cmdline -r -vvvv green
And got the coverage errors. Rolling back coverage.py to 6.5.0 from September 2022 seems to fix the issue. The latest version is 7.3.2.
With 6.5.0 I get:
Name Stmts Miss Cover Missing
-----------------------------------------------------
green/cmdline.py 44 0 100%
green/command.py 38 0 100%
green/config.py 0 0 100%
green/djangorunner.py 60 0 100%
green/exceptions.py 2 0 100%
green/junit.py 119 0 100%
green/loader.py 261 0 100%
green/output.py 101 0 100%
green/process.py 53 0 100%
green/result.py 380 0 100%
green/runner.py 70 0 100%
green/suite.py 88 0 100%
green/version.py 0 0 100%
-----------------------------------------------------
TOTAL 1216 0 100%
Ran 233 tests in 6.260s using 10 processes
OK (passes=231, skips=2)
With 6.5.0 (the last 6.x) it works, with 7.0.0 it fails. So the issue is that we are incompatible with coverage 7.x. I'll have to read the release notes and see what we need to do in order to be forward compatible.
I got some results. It turns out that for coverage 7.x to report just like it did with prior versions we need to pass what should be included. In our own wrapper for green we actually always pass --include-pattern=
and this seems to do the trick here.
I think there is a larger 'compatibility' bug to address which is to ensure we pass a new parameter when calling coverage.coverage()
in process.py
, but since this is not a direct regression in the green codebase I would just fix the release code of green to pass the pattern, then have a bug for proper coverage 7 compatibility.
> PYTHONPATH="." python3 -m green.cmdline -r -vvvv --include-pattern='green/*' green
Green 3.5.0, Coverage 7.3.2, Python 3.11.6
...
Name Stmts Miss Cover Missing
------------------------------------------------------
green/cmdline.py 44 0 100%
green/command.py 38 0 100%
green/config.py 0 0 100%
green/djangorunner.py 60 0 100%
green/exceptions.py 2 0 100%
green/junit.py 119 0 100%
green/loader.py 261 0 100%
green/output.py 101 0 100%
green/process.py 53 0 100%
green/result.py 380 0 100%
green/runner.py 70 0 100%
green/suite.py 88 0 100%
green/test/__init__.py 0 0 100%
green/version.py 0 0 100%
------------------------------------------------------
TOTAL 1216 0 100%
Ran 233 tests in 5.616s using 10 processes
OK (passes=231, skips=2)
Setting the coverage source allowed coverage.py to focus only on the files in the green directory. This works with v6.x and v7.x.
Note that I put the option in pyproject.toml, but you need either python 3.11+ (toml was added to the standard library in 3.11), or have coverage[toml]'
installed on older python releases. See https://coverage.readthedocs.io/en/7.3.2/config.html#configuration-reference.
I thing this short fix is good enough to get green v3.5.0 out.
I took a look at the GH actions to figure out why the coverage works and this seems to be because the coverage action uses coveralls which explicitly requires coverage 6.x.
https://github.com/clumio-oss/green/actions/runs/7119558135/job/19384933814#step:8:15
Collecting coverage!=6.0.*,!=6.1,!=6.1.1,<7.0,>=4.1 (from coveralls)
Downloading coverage-6.5.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (217 kB)
I managed to steal a few minutes during lunch today. You nearly got us there! Now we're getting past that and crashing on the pypy3
test. I have to get back to my day job, so I haven't been able to dig into this, but I hope it's as simple as installing some toml package in the pypy3
virtual environment. 🤞🏻
+ PYTHONPATH=.
+ venvpypy3/bin/python -m green.cmdline -k green
..........................................ssssssss.................................................................s................................................................EE............................s.....................s
Error in green.test.test_result.TestGreenTestRunCoverage.test_coverage
File "/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/unittest/case.py", line 624, in run
self.setUp()
File "/Users/nathan/proj/green/green/test/test_result.py", line 1096, in setUp
include=self.args.include_patterns,
File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/control.py", line 292, in __init__
context=context,
File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/config.py", line 597, in read_coverage_config
config_read = config.from_file(fname, warn, our_file=our_file)
File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/config.py", line 297, in from_file
files_read = cp.read(filename)
File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/tomlconfig.py", line 67, in read
raise ConfigError(msg.format(filename))
coverage.exceptions.ConfigError: Can't read 'pyproject.toml' without TOML support. Install with [toml] extra
Error in green.test.test_result.TestGreenTestRunCoverage.test_quiet_coverage
File "/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/local/Cellar/pypy3/7.3.9/libexec/lib-python/3/unittest/case.py", line 624, in run
self.setUp()
File "/Users/nathan/proj/green/green/test/test_result.py", line 1096, in setUp
include=self.args.include_patterns,
File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/control.py", line 292, in __init__
context=context,
File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/config.py", line 597, in read_coverage_config
config_read = config.from_file(fname, warn, our_file=our_file)
File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/config.py", line 297, in from_file
files_read = cp.read(filename)
File "/Users/nathan/proj/green/venvpypy3/site-packages/coverage/tomlconfig.py", line 67, in read
raise ConfigError(msg.format(filename))
coverage.exceptions.ConfigError: Can't read 'pyproject.toml' without TOML support. Install with [toml] extra
Ran 233 tests in 4.151s using 32 processes
FAILED (errors=2, passes=220, skips=11)
+ exit 3
make[1]: *** [test-versions] Error 3
make: *** [test-local] Error 2
Ah this is because coverage.py requires to be installed as coverage[toml]
on python versions prior to 3.11. This is because toml
is not a standard library as of Python 3.11. See my prior comment with the link to the coverage.py documentation.
This requirement is documented here: https://coverage.readthedocs.io/en/7.3.2/config.html#configuration-reference
pypy3 tests error and hang. (sigh) Maybe I should just drop support for pypy3. Thoughts?
I do not use pypy since it is always a few years behind cPython. Currently pypy supports 3.9, 3.10 and still 2.7 somehow. Fortunately a quick pyenv install -l
shows that pypy3.6-7.3.3
to pypy3.10-7.3.13
are available to install. I'll try these 2 versions locally and see if I can reproduce. Which command are they failing on, I tried the GH Action CI on our side and all of this seems to work, maybe we could add pypy as a target in the CI Actions. I would only target the linux version though.
It seems to work with pypy 3.10.13:
> python --version
Python 3.10.13 (f1607341da97ff5a1e93430b6e8c4af0ad1aa019, Sep 28 2023, 20:47:55)
[PyPy 7.3.13 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2.5)]
> make test-coverage
./g 3 -r -vvvv green
Green 3.5.0, Coverage 7.3.4, Python 3.10.13
...
Ran 233 tests in 10.399s using 10 processes
OK (passes=231, skips=2)
(test-coverage) completed
I do get an error with pypy 3.9.17:
> python --version
Python 3.9.17 (3f3f2298ddc56db44bbdb4551ce992d8e9401646, Jun 15 2023, 11:14:28)
[PyPy 7.3.12 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2.5)]
> make test-coverage
./g 3 -r -vvvv green
Green 3.5.0, Coverage 7.3.4, Python 3.9.17
Traceback (most recent call last):
File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/runpy.py", line 200, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/stephane/repos/clumio-oss/green/green/cmdline.py", line 103, in <module>
sys.exit(main())
File "/Users/stephane/repos/clumio-oss/green/green/cmdline.py", line 99, in main
return _main(argv, testing)
File "/Users/stephane/repos/clumio-oss/green/green/cmdline.py", line 67, in _main
result = run(test_suite, stream, args, testing)
File "/Users/stephane/repos/clumio-oss/green/green/runner.py", line 91, in run
parallel_targets = toParallelTargets(suite, args.targets)
File "/Users/stephane/repos/clumio-oss/green/green/loader.py", line 336, in toParallelTargets
proto_test_list = toProtoTestList(suite)
File "/Users/stephane/repos/clumio-oss/green/green/loader.py", line 320, in toProtoTestList
toProtoTestList(i, test_list, doing_completions)
File "/Users/stephane/repos/clumio-oss/green/green/loader.py", line 312, in toProtoTestList
getattr(suite, exception_method.strip())()
File "/Users/stephane/repos/clumio-oss/green/green/loader.py", line 78, in testFailure
raise ImportError(message)
ImportError: Failed to import green.test.test_djangorunner computed from filename /Users/stephane/repos/clumio-oss/green/green/test/test_djangorunner.py
Traceback (most recent call last):
File "/Users/stephane/repos/clumio-oss/green/green/loader.py", line 52, in loadFromModuleFilename
__import__(dotted_module)
File "/Users/stephane/repos/clumio-oss/green/green/test/test_djangorunner.py", line 8, in <module>
from green import djangorunner
File "<frozen importlib._bootstrap>", line 1058, in _handle_fromlist
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/Users/stephane/repos/clumio-oss/green/green/djangorunner.py", line 73, in <module>
from django.test.runner import DiscoverRunner
File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/site-packages/django/test/__init__.py", line 3, in <module>
from django.test.client import AsyncClient, AsyncRequestFactory, Client, RequestFactory
File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/site-packages/django/test/client.py", line 12, in <module>
from asgiref.sync import sync_to_async
File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/site-packages/asgiref/sync.py", line 130, in <module>
class AsyncToSync(Generic[_P, _R]):
File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/typing.py", line 277, in inner
return func(*args, **kwds)
File "/Users/stephane/.pyenv/versions/pypy3.9-7.3.12/lib/pypy3.9/typing.py", line 997, in __class_getitem__
raise TypeError(
TypeError: Parameters to Generic[...] must all be type variables
make: *** [test-coverage] Error 1
But it seems more of an issue of Django being incompatible with pypy and not a problem with green itself.
I would not make this a release blocker, pypy is not guaranteed to be 100% compatible, and I think the 3.5.0 branch is ok to release as it is. We can still claim that it was tested with pypy pypy3.10-7.3.13.
I tested targeting pypy in GH Actions and other than pypy 3.6 and 3.7 not being installable, everything looks good.
I've updated the 3.5.0 branch to target pypy 3.8, 3.9 and 3.10 as they worked just fine. No need to drop support for pypy, I just think there is some local issue on your laptop. https://github.com/CleanCut/green/actions/runs/7310069963
Merging with the 3.12 errors since the branch is known to not be compatible. Main will be ok.
Based on the advice in #273, I have been working on prepping a 3.5.0 release which primarily represents a nice checkpoint release that is the last release supporting Python 3.6 and Python 3.7.
I branched off of ff8d950c059ffe1ad954919dea7ea2941b92935e for this release. We'll need to resolve some conflicts with
main
AFTER release and before merging this intomain
.I've run into a bunch of blockers:
venv
is built-in and should be preferred over installingvirtualenv
green-
pypy3
can't install one ofblack
's dependenciestest_finalizer
test was not succeeding on macOSmake release
) on Python 3.11 on macOS -- I haven't drilled into this, yet.@sodul I have invited you as a collaborator. It won't let me view your permissions until you accept, but I intend to give you write permission so you can create branches directly on the repository, and push commits to existing branches like this one if you want to (hint, hint 😉 😆).