search

python / cpython

The Python programming language
https://www.python.org
Other
62.35k stars 29.94k forks source link

test_peg_generator leaks references #105063

Open Eclips4 opened 1 year ago

Eclips4 commented 1 year ago

Tried on curent main branch

Traceback ```python PS C:\Users\KIRILL-1\CLionProjects\cpython> ./python -m test -v test_peg_generator Running Debug|x64 interpreter... == CPython 3.13.0a0 (heads/main:1668b41dc4, May 29 2023, 18:02:21) [MSC v.1933 64 bit (AMD64)] == Windows-10-10.0.19043-SP0 little-endian == Python build: debug == cwd: C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ == CPU count: 16 == encodings: locale=cp1251, FS=utf-8 0:00:00 Run tests sequentially 0:00:00 [1/1] test_peg_generator Run: C:\Users\KIRILL-1\CLionProjects\cpython\PCbuild\amd64\python_d.exe -X dev -m venv venv Run: C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ\tempcwd\venv\Scripts\python_d.exe -X dev -m pip install C:\U sers\KIRILL-1\CLionProjects\cpython\Lib\test\setuptools-67.6.1-py3-none-any.whl C:\Users\KIRILL-1\CLionProjects\cpython\Lib\test\w heel-0.40.0-py3-none-any.whl Processing c:\users\kirill-1\clionprojects\cpython\lib\test\setuptools-67.6.1-py3-none-any.whl C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ\tempcwd\venv\Lib\site-packages\pip\_internal\metadata\importlib\_ dists.py:74: DeprecationWarning: Unimplemented abstract methods {'locate_file'} return cls(files, info_location) Processing c:\users\kirill-1\clionprojects\cpython\lib\test\wheel-0.40.0-py3-none-any.whl Installing collected packages: wheel, setuptools C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ\tempcwd\venv\Lib\site-packages\pip\_internal\metadata\importlib\_ dists.py:74: DeprecationWarning: Unimplemented abstract methods {'locate_file'} return cls(files, info_location) Successfully installed setuptools-67.6.1 wheel-0.40.0 C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ\tempcwd\venv\Lib\site-packages\pip\_internal\self_outdated_check. py:232: DeprecationWarning: datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware obje cts to represent datetimes in UTC: datetime.now(datetime.UTC). current_time=datetime.datetime.utcnow(), C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_11752æ\tempcwd\venv\Lib\site-packages\pip\_internal\utils\misc.py:130: D eprecationWarning: onerror argument is deprecated, use onexc instead shutil.rmtree(dir, ignore_errors=ignore_errors, onerror=rmtree_errorhandler) Traceback (most recent call last): File "", line 1, in File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\encodings\cp1251.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeEncodeError: 'charmap' codec can't encode character '\xe6' in position 63: character maps to setUpClass (test.test_peg_generator.test_c_parser.TestCParser) ... ERROR test_advance_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_advance_left_recursion) ... ok test_alternatives (test.test_peg_generator.test_first_sets.TestFirstSets.test_alternatives) ... ok test_epsilon_production_in_start_rule (test.test_peg_generator.test_first_sets.TestFirstSets.test_epsilon_production_in_start_rule ) ... ok test_gather (test.test_peg_generator.test_first_sets.TestFirstSets.test_gather) ... ok test_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_left_recursion) ... ok test_multiple_nullable_rules (test.test_peg_generator.test_first_sets.TestFirstSets.test_multiple_nullable_rules) ... ok test_mutual_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_mutual_left_recursion) ... ok test_nasty_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_nasty_left_recursion) ... ok test_negative_lookahead (test.test_peg_generator.test_first_sets.TestFirstSets.test_negative_lookahead) ... ok test_nullable_rule (test.test_peg_generator.test_first_sets.TestFirstSets.test_nullable_rule) ... ok test_optional_after (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_after) ... ok test_optional_before (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_before) ... ok test_optional_literal (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_literal) ... ok test_optional_operator (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_operator) ... ok test_optionals (test.test_peg_generator.test_first_sets.TestFirstSets.test_optionals) ... ok test_positive_lookahead (test.test_peg_generator.test_first_sets.TestFirstSets.test_positive_lookahead) ... ok test_repeat_0 (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_0) ... ok test_repeat_0_with_group (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_0_with_group) ... ok test_repeat_1 (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_1) ... ok test_repeat_1_with_group (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_1_with_group) ... ok test_repeat_with_separator (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_with_separator) ... ok test_rule_with_collision_after_some_other_rules (test.test_peg_generator.test_grammar_validator.TestPegen.test_rule_with_collision _after_some_other_rules) ... ok test_rule_with_no_collision (test.test_peg_generator.test_grammar_validator.TestPegen.test_rule_with_no_collision) ... ok test_rule_with_simple_collision (test.test_peg_generator.test_grammar_validator.TestPegen.test_rule_with_simple_collision) ... ok test_deep_nested_rule (test.test_peg_generator.test_pegen.TestGrammarVisualizer.test_deep_nested_rule) ... ok test_multiple_rules (test.test_peg_generator.test_pegen.TestGrammarVisualizer.test_multiple_rules) ... ok test_simple_rule (test.test_peg_generator.test_pegen.TestGrammarVisualizer.test_simple_rule) ... ok test_advanced_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_advanced_left_recursive) ... ok test_alt_optional_operator (test.test_peg_generator.test_pegen.TestPegen.test_alt_optional_operator) ... ok test_bad_token_reference (test.test_peg_generator.test_pegen.TestPegen.test_bad_token_reference) ... ok test_cut (test.test_peg_generator.test_pegen.TestPegen.test_cut) ... ok test_dangling_reference (test.test_peg_generator.test_pegen.TestPegen.test_dangling_reference) ... ok test_expr_grammar (test.test_peg_generator.test_pegen.TestPegen.test_expr_grammar) ... ok test_forced (test.test_peg_generator.test_pegen.TestPegen.test_forced) ... ok test_forced_with_group (test.test_peg_generator.test_pegen.TestPegen.test_forced_with_group) ... ok test_gather (test.test_peg_generator.test_pegen.TestPegen.test_gather) ... ok test_invalid_rule_name (test.test_peg_generator.test_pegen.TestPegen.test_invalid_rule_name) ... ok test_invalid_variable_name (test.test_peg_generator.test_pegen.TestPegen.test_invalid_variable_name) ... ok test_invalid_variable_name_in_temporal_rule (test.test_peg_generator.test_pegen.TestPegen.test_invalid_variable_name_in_temporal_r ule) ... ok test_left_recursion_too_complex (test.test_peg_generator.test_pegen.TestPegen.test_left_recursion_too_complex) ... ok test_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_left_recursive) ... ok test_locations_in_alt_action_and_group (test.test_peg_generator.test_pegen.TestPegen.test_locations_in_alt_action_and_group) ... o k test_long_rule_str (test.test_peg_generator.test_pegen.TestPegen.test_long_rule_str) ... ok test_lookahead (test.test_peg_generator.test_pegen.TestPegen.test_lookahead) ... ok test_missing_start (test.test_peg_generator.test_pegen.TestPegen.test_missing_start) ... ok test_mutually_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_mutually_left_recursive) ... ok test_named_lookahead_error (test.test_peg_generator.test_pegen.TestPegen.test_named_lookahead_error) ... ok test_nasty_mutually_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_nasty_mutually_left_recursive) ... ok test_nullable (test.test_peg_generator.test_pegen.TestPegen.test_nullable) ... ok test_opt_sequence (test.test_peg_generator.test_pegen.TestPegen.test_opt_sequence) ... ok test_optional_literal (test.test_peg_generator.test_pegen.TestPegen.test_optional_literal) ... ok test_optional_operator (test.test_peg_generator.test_pegen.TestPegen.test_optional_operator) ... ok test_parse_grammar (test.test_peg_generator.test_pegen.TestPegen.test_parse_grammar) ... ok test_python_expr (test.test_peg_generator.test_pegen.TestPegen.test_python_expr) ... ok test_repeat_0_complex (test.test_peg_generator.test_pegen.TestPegen.test_repeat_0_complex) ... ok test_repeat_0_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_0_simple) ... ok test_repeat_1_complex (test.test_peg_generator.test_pegen.TestPegen.test_repeat_1_complex) ... ok test_repeat_1_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_1_simple) ... ok test_repeat_with_sep_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_with_sep_simple) ... ok test_soft_keyword (test.test_peg_generator.test_pegen.TestPegen.test_soft_keyword) ... ok test_start_leader (test.test_peg_generator.test_pegen.TestPegen.test_start_leader) ... ok test_typed_rules (test.test_peg_generator.test_pegen.TestPegen.test_typed_rules) ... ok test_unreachable_explicit (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_explicit) ... ok test_unreachable_implicit1 (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_implicit1) ... ok test_unreachable_implicit2 (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_implicit2) ... ok test_unreachable_implicit3 (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_implicit3) ... ok ====================================================================== ERROR: setUpClass (test.test_peg_generator.test_c_parser.TestCParser) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\test\test_peg_generator\test_c_parser.py", line 95, in setUpClass sitepackages = subprocess.check_output( ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\subprocess.py", line 466, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['C:\\Users\\KIRILL-1\\CLionProjects\\cpython\\build\\test_python_11752æ\\tempcwd\\venv\\S cripts\\python_d.exe', '-c', "import sysconfig; print(sysconfig.get_path('platlib'))"]' returned non-zero exit status 1. ---------------------------------------------------------------------- Ran 66 tests in 22.073s FAILED (errors=1) test test_peg_generator failed test_peg_generator failed (1 error) == Tests result: FAILURE == 1 test failed: test_peg_generator Total duration: 22.3 sec Tests result: FAILURE ```

Linked PRs

Eclips4 commented 1 year ago

seems to be introduced in #104798 cc @lysnikolaou

Eclips4 commented 1 year ago

I think, this traceback more helpful:

Traceback ```python3 PS C:\Users\KIRILL-1\CLionProjects\cpython> ./python -m test -v test_peg_generator -m test_c_parser -m TestCParser Running Debug|x64 interpreter... == CPython 3.13.0a0 (heads/main:1668b41dc4, May 29 2023, 18:02:21) [MSC v.1933 64 bit (AMD64)] == Windows-10-10.0.19043-SP0 little-endian == Python build: debug == cwd: C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_16468æ == CPU count: 16 == encodings: locale=cp1251, FS=utf-8 0:00:00 Run tests sequentially 0:00:00 [1/1] test_peg_generator Run: C:\Users\KIRILL-1\CLionProjects\cpython\PCbuild\amd64\python_d.exe -X dev -m venv venv Run: C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_16468æ\tempcwd\venv\Scripts\python_d.exe -X dev -m pip install C:\U sers\KIRILL-1\CLionProjects\cpython\Lib\test\setuptools-67.6.1-py3-none-any.whl C:\Users\KIRILL-1\CLionProjects\cpython\Lib\test\w heel-0.40.0-py3-none-any.whl Processing c:\users\kirill-1\clionprojects\cpython\lib\test\setuptools-67.6.1-py3-none-any.whl C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_16468æ\tempcwd\venv\Lib\site-packages\pip\_internal\metadata\importlib\_ dists.py:74: DeprecationWarning: Unimplemented abstract methods {'locate_file'} return cls(files, info_location) Processing c:\users\kirill-1\clionprojects\cpython\lib\test\wheel-0.40.0-py3-none-any.whl Installing collected packages: wheel, setuptools C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_16468æ\tempcwd\venv\Lib\site-packages\pip\_internal\metadata\importlib\_ dists.py:74: DeprecationWarning: Unimplemented abstract methods {'locate_file'} return cls(files, info_location) Successfully installed setuptools-67.6.1 wheel-0.40.0 C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_16468æ\tempcwd\venv\Lib\site-packages\pip\_internal\self_outdated_check. py:232: DeprecationWarning: datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware obje cts to represent datetimes in UTC: datetime.now(datetime.UTC). current_time=datetime.datetime.utcnow(), C:\Users\KIRILL-1\CLionProjects\cpython\build\test_python_16468æ\tempcwd\venv\Lib\site-packages\pip\_internal\utils\misc.py:130: D eprecationWarning: onerror argument is deprecated, use onexc instead shutil.rmtree(dir, ignore_errors=ignore_errors, onerror=rmtree_errorhandler) Traceback (most recent call last): File "", line 1, in File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\encodings\cp1251.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeEncodeError: 'charmap' codec can't encode character '\xe6' in position 63: character maps to setUpClass (test.test_peg_generator.test_c_parser.TestCParser) ... ERROR ====================================================================== ERROR: setUpClass (test.test_peg_generator.test_c_parser.TestCParser) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\test\test_peg_generator\test_c_parser.py", line 95, in setUpClass sitepackages = subprocess.check_output( ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\subprocess.py", line 466, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['C:\\Users\\KIRILL-1\\CLionProjects\\cpython\\build\\test_python_16468æ\\tempcwd\\venv\\S cripts\\python_d.exe', '-c', "import sysconfig; print(sysconfig.get_path('platlib'))"]' returned non-zero exit status 1. ---------------------------------------------------------------------- Ran 0 tests in 22.299s FAILED (errors=1) test test_peg_generator failed test_peg_generator failed (1 error) == Tests result: FAILURE == 1 test failed: test_peg_generator Total duration: 22.6 sec Tests result: FAILURE ```
erlend-aasland commented 1 year ago

Thanks, Lysandros is already working on this since yesterday. BTW, the failure is not Windows specific: https://buildbot.python.org/all/#/builders/320/builds/730/steps/5/logs/stdio

erlend-aasland commented 1 year ago

seems to be introduced in #104798

FTR, it is verified that #104798 introduced this.

Eclips4 commented 1 year ago

Thanks, Lysandros is already working on this since yesterday. BTW, the failure is not Windows specific: https://buildbot.python.org/all/#/builders/320/builds/730/steps/5/logs/stdio

Indeed. I couldn't reproduce it on my WSL (Ubuntu 20.04), and thought that it's related to Windows system. So, thanks for this point and removing os-windows label. UPD: I was running test as ./python -m test -v test_peg_generator on Linux, which doesn't cause test fail.

Eclips4 commented 1 year ago

I'm somewhat unsure about changing the issue title. Seems there two issues: 1 - which described in this issue - test fail (Windows issue) 2 - refleak test fail, which reproducible on Linux (but on Windows we cannot get refleak test results, because of test failing) So, seems that there two problems, which probably will be solved by one PR

erlend-aasland commented 1 year ago

Enclosing my response in <details> in order to not bring too much clutter and noise into this issue:

Kirill, please see CI on `main`. You can see it using the GitHUb UI by clicking the "Code" tab. When in the "Code" tab, you can explore the CI run of the most recent commit: Screenshot 2023-05-29 at 23 25 50 When you click the red cross or green check mark, you can inspect the CI run for the commit: Screenshot 2023-05-29 at 23 27 25 As you can see, both the 32-bit and 64-bit Windows CI are completely green, which implies there is no failure on [current](39f6a0489fcc815a578d27dfee2feea003c896f8) `main`. Perhaps the failed test you are seeing is because of an issue with your local setup? Did you try a fresh build? If no, please do so. In case of test failures, you should _always_ try `git clear -fdx` (in case of in-tree builds), or `rm -rf ` first. However, what we _know_ from our CI, is that there's a _reference leak_.
Eclips4 commented 1 year ago

Thanks Erlend for your answer! I know about it all. After one situation with asyncio(eager-task merging) I've git clone and rebuilt interpreter every time when I find some fail. So, there's a some proof of my words:

```python PS C:\Users\KIRILL-1\test_cpython> git clone https://github.com/python/cpython Cloning into 'cpython'... remote: Enumerating objects: 952218, done. remote: Counting objects: 100% (1521/1521), done. remote: Compressing objects: 100% (585/585), done. remote: Total 952218 (delta 1071), reused 1286 (delta 936), pack-reused 950697 Receiving objects: 100% (952218/952218), 555.63 MiB | 7.43 MiB/s, done. Resolving deltas: 100% (756558/756558), done. Updating files: 100% (4393/4393), done. PS C:\Users\KIRILL-1\test_cpython> cd cpython PS C:\Users\KIRILL-1\test_cpython\cpython> pcbuild/build.bat -c Debug Using py -3.11 (found 3.11 with py.exe) Fetching external libraries... Fetching bzip2-1.0.8... Fetching sqlite-3.42.0.0... Fetching xz-5.2.5... Fetching zlib-1.2.13... Fetching external binaries... Fetching libffi-3.4.4... Fetching openssl-bin-1.1.1t... Fetching tcltk-8.6.13.0... Finished. Using "D:\VStudio\MSBuild\Current\Bin\msbuild.exe" (found in the Visual Studio installation) Using py -3.11 (found 3.11 with py.exe) C:\Users\KIRILL-1\test_cpython\cpython>"D:\VStudio\MSBuild\Current\Bin\msbuild.exe" "C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\pcbuild.proj" /t:Build /m /nologo /v:m /clp:summary /p:Configuration=Debug /p:Platform=x64 /p:IncludeExternals=true /p:IncludeCTypes=true /p:IncludeSSL=true /p:IncludeTkinter=true /p:UseTestMarker= /p:GIT="C:\Program Files\Git\cmd\git.exe" _freeze_module.c config_minimal.c atexitmodule.c faulthandler.c gcmodule.c getbuildinfo.c getpath_noop.c posixmodule.c signalmodule.c timemodule.c _tracemalloc.c _iomodule.c bufferedio.c bytesio.c fileio.c iobase.c stringio.c textio.c winconsoleio.c abstract.c boolobject.c bytearrayobject.c bytes_methods.c bytesobject.c call.c capsule.c cellobject.c classobject.c codeobject.c complexobject.c descrobject.c dictobject.c enumobject.c exceptions.c fileobject.c floatobject.c frameobject.c funcobject.c genericaliasobject.c genobject.c interpreteridobject.c iterobject.c listobject.c longobject.c memoryobject.c methodobject.c moduleobject.c namespaceobject.c object.c obmalloc.c odictobject.c picklebufobject.c rangeobject.c setobject.c sliceobject.c structseq.c tupleobject.c typeobject.c typevarobject.c unicodectype.c unicodeobject.c unionobject.c weakrefobject.c myreadline.c parser.c peg_api.c pegen.c pegen_errors.c action_helpers.c string_parser.c token.c tokenizer.c invalid_parameter_handler.c msvcrtmodule.c winreg.c _warnings.c asdl.c assemble.c ast.c ast_opt.c ast_unparse.c bltinmodule.c bootstrap_hash.c ceval.c codecs.c compile.c context.c dtoa.c dynamic_annotations.c dynload_win.c errors.c fileutils.c flowgraph.c formatter_unicode.c frame.c future.c getargs.c getcompiler.c getcopyright.c getopt.c getplatform.c getversion.c ceval_gil.c hamt.c hashtable.c import.c importdl.c initconfig.c intrinsics.c instrumentation.c legacy_tracing.c marshal.c modsupport.c mysnprintf.c mystrtoul.c pathconfig.c perf_trampoline.c preconfig.c pyarena.c pyctype.c pyfpe.c pyhash.c pylifecycle.c pymath.c pystate.c pystrcmp.c pystrhex.c pystrtod.c Python-ast.c pythonrun.c Python-tokenize.c pytime.c specialize.c structmember.c suggestions.c symtable.c thread.c traceback.c tracemalloc.c sysmodule.c _freeze_module.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\win32\_freeze_module_d.exe Updated files: importlib._bootstrap.h, importlib._bootstrap_external.h, zipimport.h, abc.h, codecs.h, io.h, _collections_abc.h, _sitebuiltins.h, genericpath.h, ntpath.h, posixpath.h, os.h, site.h, stat.h, importlib.util.h, importlib.machinery.h, runpy.h, __hello__.h, __phello__.h, __phello__.ham.h, __phello__.ham.eggs.h, __phello__.spam.h, frozen_only.h Updated files: getpath.h Killing any running python_d.exe instances... Regenerate pycore_ast.h pycore_ast_state.h Python-ast.c C:\Users\KIRILL-1\test_cpython\cpython\Python\Python-ast.c, C:\Users\KIRILL-1\test_cpython\cpython\Include\internal\pycore_ast. h, C:\Users\KIRILL-1\test_cpython\cpython\Include\internal\pycore_ast_state.h regenerated. Regenerate opcode.h pycore_opcode.h opcode_targets.h Include\opcode.h regenerated from Lib\opcode.py Jump table written into Python\opcode_targets.h Regenerate token-list.inc pycore_token.h token.c token.py Regenerate Global Objects # not changed: C:\Users\KIRILL-1\test_cpython\cpython\Include\internal\pycore_global_strings.h # not changed: C:\Users\KIRILL-1\test_cpython\cpython\Include\internal\pycore_runtime_init_generated.h # not changed: C:\Users\KIRILL-1\test_cpython\cpython\Include\internal\pycore_unicodeobject_generated.h # not changed: C:\Users\KIRILL-1\test_cpython\cpython\Include\internal\pycore_global_objects_fini_generated.h Generated sources are up to date Getting build info from "C:\Program Files\Git\cmd\git.exe" Building heads/main:d593074494 main getpath.c _abc.c _bisectmodule.c blake2module.c blake2b_impl.c blake2s_impl.c _codecsmodule.c _collectionsmodule.c _contextvarsmodule.c _csv.c _functoolsmodule.c Hacl_Hash_MD5.c Hacl_Hash_SHA1.c Hacl_Hash_SHA2.c Hacl_Hash_SHA3.c _heapqmodule.c _json.c _localemodule.c _lsprof.c _pickle.c _randommodule.c sre.c _stat.c _struct.c _weakref.c arraymodule.c atexitmodule.c cmathmodule.c _datetimemodule.c errnomodule.c faulthandler.c gcmodule.c itertoolsmodule.c main.c mathmodule.c md5module.c mmapmodule.c _opcode.c _operator.c posixmodule.c rotatingtree.c sha1module.c sha2module.c sha3module.c signalmodule.c _statisticsmodule.c symtablemodule.c _threadmodule.c _tracemalloc.c _typingmodule.c timemodule.c xxsubtype.c _xxsubinterpretersmodule.c _xxinterpchannelsmodule.c fileio.c bytesio.c stringio.c bufferedio.c iobase.c textio.c winconsoleio.c _iomodule.c _codecs_cn.c _codecs_hk.c _codecs_iso2022.c _codecs_jp.c _codecs_kr.c _codecs_tw.c multibytecodec.c _winapi.c abstract.c boolobject.c bytearrayobject.c bytes_methods.c bytesobject.c call.c capsule.c cellobject.c classobject.c codeobject.c complexobject.c descrobject.c dictobject.c enumobject.c exceptions.c fileobject.c floatobject.c frameobject.c funcobject.c genericaliasobject.c genobject.c interpreteridobject.c iterobject.c listobject.c longobject.c memoryobject.c methodobject.c moduleobject.c namespaceobject.c object.c obmalloc.c odictobject.c picklebufobject.c rangeobject.c setobject.c sliceobject.c structseq.c tupleobject.c typeobject.c typevarobject.c unicodectype.c unicodeobject.c unionobject.c weakrefobject.c myreadline.c tokenizer.c token.c pegen.c pegen_errors.c action_helpers.c parser.c string_parser.c peg_api.c invalid_parameter_handler.c winreg.c config.c msvcrtmodule.c pyhash.c _warnings.c asdl.c assemble.c ast.c ast_opt.c ast_unparse.c bltinmodule.c bootstrap_hash.c ceval.c codecs.c compile.c context.c dynamic_annotations.c dynload_win.c errors.c fileutils.c flowgraph.c formatter_unicode.c frame.c frozen.c future.c getargs.c getcompiler.c getcopyright.c getopt.c getplatform.c getversion.c ceval_gil.c hamt.c hashtable.c import.c importdl.c initconfig.c intrinsics.c instrumentation.c legacy_tracing.c marshal.c modsupport.c mysnprintf.c mystrtoul.c pathconfig.c perf_trampoline.c preconfig.c pyarena.c pyctype.c pyfpe.c pylifecycle.c pymath.c pytime.c pystate.c pystrcmp.c pystrhex.c pystrtod.c dtoa.c Python-ast.c Python-tokenize.c pythonrun.c specialize.c suggestions.c structmember.c symtable.c thread.c traceback.c tracemalloc.c deepfreeze.c zlibmodule.c adler32.c compress.c crc32.c infback.c inffast.c inflate.c inftrees.c trees.c uncompr.c zutil.c dl_nt.c binascii.c getbuildinfo.c sysmodule.c deflate.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\python313_d.lib и объект C:\Users\KIRILL-1\test_cp ython\cpython\PCbuild\amd64\python313_d.exp pythoncore.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\python313_d.dll _ctypes_test.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_ctypes_test_d.lib и объект C:\Users\KIRILL-1\test _cpython\cpython\PCbuild\amd64\_ctypes_test_d.exp _ctypes_test.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_ctypes_test_d.pyd _testbuffer.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testbuffer_d.lib и объект C:\Users\KIRILL-1\test_ cpython\cpython\PCbuild\amd64\_testbuffer_d.exp _testbuffer.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testbuffer_d.pyd python3dll.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\python3_d.lib и объект C:\Users\KIRILL-1\test_cpyt hon\cpython\PCbuild\amd64\python3_d.exp python3dll.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\python3_d.dll _testcapimodule.c getargs.c vectorcall.c vectorcall_limited.c heaptype.c heaptype_relative.c unicode.c pytime.c datetime.c docstring.c mem.c watchers.c float.c long.c structmember.c exceptions.c code.c buffer.c pyos.c immortal.c gc.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testcapi_d.lib и объект C:\Users\KIRILL-1\test_cp ython\cpython\PCbuild\amd64\_testcapi_d.exp _testcapi.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testcapi_d.pyd _testinternalcapi.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testinternalcapi_d.lib и объект C:\Users\KIRILL-1 \test_cpython\cpython\PCbuild\amd64\_testinternalcapi_d.exp _testinternalcapi.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testinternalcapi_d.pyd _testembed.c _testembed.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testembed_d.exe _testimportmultiple.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testimportmultiple_d.lib и объект C:\Users\KIRILL -1\test_cpython\cpython\PCbuild\amd64\_testimportmultiple_d.exp _testimportmultiple.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testimportmultiple_d.pyd _testmultiphase.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testmultiphase_d.lib и объект C:\Users\KIRILL-1\t est_cpython\cpython\PCbuild\amd64\_testmultiphase_d.exp _testmultiphase.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testmultiphase_d.pyd _testsinglephase.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testsinglephase_d.lib и объект C:\Users\KIRILL-1\ test_cpython\cpython\PCbuild\amd64\_testsinglephase_d.exp _testsinglephase.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testsinglephase_d.pyd _testconsole.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testconsole_d.lib и объект C:\Users\KIRILL-1\test _cpython\cpython\PCbuild\amd64\_testconsole_d.exp _testconsole.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_testconsole_d.pyd launcher2.c pyshellext.cpp winsound.c launcher.c launcher2.c _uuidmodule.c Performing Custom Build Tools _queuemodule.c _elementtree.c selectmodule.c xmlparse.c _asynciomodule.c xmlrole.c unicodedata.c overlapped.c xmltok.c pyexpat.c launcher.c multiprocessing.c _zoneinfo.c xmlparse.c semaphore.c xmlrole.c xmltok.c _wmimodule.cpp Assembling: C:\Users\KIRILL-1\test_cpython\cpython\Modules\_decimal\libmpdec\vcdiv64.asm _decimal.c basearith.c constants.c context.c convolute.c crt.c difradix2.c fnt.c fourstep.c io.c mpalloc.c mpdecimal.c numbertheory.c sixstep.c transpose.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_queue_d.lib и объект C:\Users\KIRILL-1\test_cpyth on\cpython\PCbuild\amd64\_queue_d.exp _ctypes.c callbacks.c callproc.c cfield.c malloc_closure.c stgdict.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\select_d.lib и объект C:\Users\KIRILL-1\test_cpyth on\cpython\PCbuild\amd64\select_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_asyncio_d.lib и объект C:\Users\KIRILL-1\test_cpy thon\cpython\PCbuild\amd64\_asyncio_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\winsound_d.lib и объект C:\Users\KIRILL-1\test_cpy thon\cpython\PCbuild\amd64\winsound_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_overlapped_d.lib и объект C:\Users\KIRILL-1\test_ cpython\cpython\PCbuild\amd64\_overlapped_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_zoneinfo_d.lib и объект C:\Users\KIRILL-1\test_cp ython\cpython\PCbuild\amd64\_zoneinfo_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_multiprocessing_d.lib и объект C:\Users\KIRILL-1\ test_cpython\cpython\PCbuild\amd64\_multiprocessing_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_elementtree_d.lib и объект C:\Users\KIRILL-1\test _cpython\cpython\PCbuild\amd64\_elementtree_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\unicodedata_d.lib и объект C:\Users\KIRILL-1\test_ cpython\cpython\PCbuild\amd64\unicodedata_d.exp _queue.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_queue_d.pyd Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\pyexpat_d.lib и объект C:\Users\KIRILL-1\test_cpyt hon\cpython\PCbuild\amd64\pyexpat_d.exp select.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\select_d.pyd Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\pyshellext_d.lib и объект C:\Users\KIRILL-1\test_c python\cpython\PCbuild\amd64\pyshellext_d.exp winsound.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\winsound_d.pyd _overlapped.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_overlapped_d.pyd _bz2module.c blocksort.c bzlib.c compress.c crctable.c decompress.c huffman.c randtable.c _multiprocessing.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_multiprocessing_d.pyd sqlite3.c tuklib_cpucores.c _hashopenssl.c tuklib_physmem.c check.c crc32_fast.c crc32_table.c crc64_fast.c crc64_table.c sha256.c alone_decoder.c alone_encoder.c auto_decoder.c block_buffer_decoder.c block_buffer_encoder.c block_decoder.c block_encoder.c block_header_decoder.c socketmodule.c _tkinter.c tkappinit.c _asyncio.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_asyncio_d.pyd Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_decimal_d.lib и объект C:\Users\KIRILL-1\test_cpy thon\cpython\PCbuild\amd64\_decimal_d.exp block_header_encoder.c block_util.c common.c easy_buffer_encoder.c easy_decoder_memusage.c easy_encoder.c easy_encoder_memusage.c easy_preset.c filter_buffer_decoder.c filter_buffer_encoder.c filter_common.c filter_decoder.c filter_encoder.c filter_flags_decoder.c filter_flags_encoder.c hardware_cputhreads.c hardware_physmem.c index.c index_decoder.c index_encoder.c index_hash.c outqueue.c stream_buffer_decoder.c stream_buffer_encoder.c stream_decoder.c stream_encoder.c stream_encoder_mt.c stream_flags_common.c stream_flags_decoder.c stream_flags_encoder.c vli_decoder.c vli_encoder.c vli_size.c _zoneinfo.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_zoneinfo_d.pyd Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_socket_d.lib и объект C:\Users\KIRILL-1\test_cpyt hon\cpython\PCbuild\amd64\_socket_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_hashlib_d.lib и объект C:\Users\KIRILL-1\test_cpy thon\cpython\PCbuild\amd64\_hashlib_d.exp delta_common.c delta_decoder.c delta_encoder.c fastpos_table.c lzma2_decoder.c lzma2_encoder.c lzma_decoder.c lzma_encoder.c lzma_encoder_optimum_fast.c lzma_encoder_optimum_normal.c lzma_encoder_presets.c lz_decoder.c lz_encoder.c lz_encoder_mf.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_bz2_d.lib и объект C:\Users\KIRILL-1\test_cpython \cpython\PCbuild\amd64\_bz2_d.exp price_table.c arm.c armthumb.c ia64.c powerpc.c simple_coder.c simple_decoder.c simple_encoder.c sparc.c x86.c unicodedata.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\unicodedata_d.pyd pyshellext.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\pyshellext_d.dll _hashlib.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_hashlib_d.pyd _bz2.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_bz2_d.pyd pyexpat.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\pyexpat_d.pyd _elementtree.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_elementtree_d.pyd _decimal.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_decimal_d.pyd _socket.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_socket_d.pyd _ssl.c venvlauncher.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\venvlauncher_d.exe venvwlauncher.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\venvwlauncher_d.exe pywlauncher.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\pyw_d.exe pylauncher.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\py_d.exe Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\sqlite3_d.lib и объект C:\Users\KIRILL-1\test_cpyt hon\cpython\PCbuild\amd64\sqlite3_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_tkinter_d.lib и объект C:\Users\KIRILL-1\test_cpy thon\cpython\PCbuild\amd64\_tkinter_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_ctypes_d.lib и объект C:\Users\KIRILL-1\test_cpyt hon\cpython\PCbuild\amd64\_ctypes_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_wmi_d.lib и объект C:\Users\KIRILL-1\test_cpython \cpython\PCbuild\amd64\_wmi_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_uuid_d.lib и объект C:\Users\KIRILL-1\test_cpytho n\cpython\PCbuild\amd64\_uuid_d.exp sqlite3.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\sqlite3_d.dll _tkinter.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_tkinter_d.pyd _ctypes.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_ctypes_d.pyd _uuid.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_uuid_d.pyd _wmi.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_wmi_d.pyd connection.c cursor.c microprotocols.c prepare_protocol.c row.c statement.c util.c blob.c liblzma.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\liblzma_d.lib _lzmamodule.c applink.c Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_sqlite3_d.lib и объект C:\Users\KIRILL-1\test_cpy thon\cpython\PCbuild\amd64\_sqlite3_d.exp _sqlite3.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_sqlite3_d.pyd Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_lzma_d.lib и объект C:\Users\KIRILL-1\test_cpytho n\cpython\PCbuild\amd64\_lzma_d.exp Создается библиотека C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_ssl_d.lib и объект C:\Users\KIRILL-1\test_cpython \cpython\PCbuild\amd64\_ssl_d.exp _lzma.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_lzma_d.pyd _ssl.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\_ssl_d.pyd python.c python.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\python_d.exe C:\Windows\SYSTEM32\ucrtbased.dll is version 10.0.19041.685 Regenerate test_frozenmain.h Programs\test_frozenmain.h written Wrote C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\LICENSE.txt WinMain.c pythonw.vcxproj -> C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\pythonw_d.exe Сборка успешно завершена. Предупреждений: 0 Ошибок: 0 Прошло времени 00:00:52.12 PS C:\Users\KIRILL-1\test_cpython\cpython> ./python -m test -v test_peg_generator Running Debug|x64 interpreter... == CPython 3.13.0a0 (heads/main:d593074494, May 30 2023, 00:53:45) [MSC v.1933 64 bit (AMD64)] == Windows-10-10.0.19043-SP0 little-endian == Python build: debug == cwd: C:\Users\KIRILL-1\test_cpython\cpython\build\test_python_22060æ == CPU count: 16 == encodings: locale=cp1251, FS=utf-8 0:00:00 Run tests sequentially 0:00:00 [1/1] test_peg_generator Run: C:\Users\KIRILL-1\test_cpython\cpython\PCbuild\amd64\python_d.exe -X dev -m venv venv Run: C:\Users\KIRILL-1\test_cpython\cpython\build\test_python_22060æ\tempcwd\venv\Scripts\python_d.exe -X dev -m pip install C:\Users\KIRILL-1\test_cpython\cpython\Lib\test\setuptools-67.6.1-py3-none-any.whl C:\Users\KIRILL-1\test_cpython\cpython\Lib\test\wheel-0.40.0-py3-none-any.whl Processing c:\users\kirill-1\test_cpython\cpython\lib\test\setuptools-67.6.1-py3-none-any.whl C:\Users\KIRILL-1\test_cpython\cpython\build\test_python_22060æ\tempcwd\venv\Lib\site-packages\pip\_internal\metadata\importlib\_dists.py:74: DeprecationWarning: Unimplemented abstract methods {'locate_file'} return cls(files, info_location) Processing c:\users\kirill-1\test_cpython\cpython\lib\test\wheel-0.40.0-py3-none-any.whl Installing collected packages: wheel, setuptools C:\Users\KIRILL-1\test_cpython\cpython\build\test_python_22060æ\tempcwd\venv\Lib\site-packages\pip\_internal\metadata\importlib\_dists.py:74: DeprecationWarning: Unimplemented abstract methods {'locate_file'} return cls(files, info_location) Successfully installed setuptools-67.6.1 wheel-0.40.0 C:\Users\KIRILL-1\test_cpython\cpython\build\test_python_22060æ\tempcwd\venv\Lib\site-packages\pip\_internal\self_outdated_check.py:232: DeprecationWarning: datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.now(datetime.UTC). current_time=datetime.datetime.utcnow(), C:\Users\KIRILL-1\test_cpython\cpython\build\test_python_22060æ\tempcwd\venv\Lib\site-packages\pip\_internal\utils\misc.py:130: DeprecationWarning: onerror argument is deprecated, use onexc instead shutil.rmtree(dir, ignore_errors=ignore_errors, onerror=rmtree_errorhandler) Traceback (most recent call last): File "", line 1, in File "C:\Users\KIRILL-1\test_cpython\cpython\Lib\encodings\cp1251.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeEncodeError: 'charmap' codec can't encode character '\xe6' in position 62: character maps to setUpClass (test.test_peg_generator.test_c_parser.TestCParser) ... ERROR test_advance_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_advance_left_recursion) ... ok test_alternatives (test.test_peg_generator.test_first_sets.TestFirstSets.test_alternatives) ... ok test_epsilon_production_in_start_rule (test.test_peg_generator.test_first_sets.TestFirstSets.test_epsilon_production_in_start_rule) ... ok test_gather (test.test_peg_generator.test_first_sets.TestFirstSets.test_gather) ... ok test_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_left_recursion) ... ok test_multiple_nullable_rules (test.test_peg_generator.test_first_sets.TestFirstSets.test_multiple_nullable_rules) ... ok test_mutual_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_mutual_left_recursion) ... ok test_nasty_left_recursion (test.test_peg_generator.test_first_sets.TestFirstSets.test_nasty_left_recursion) ... ok test_negative_lookahead (test.test_peg_generator.test_first_sets.TestFirstSets.test_negative_lookahead) ... ok test_nullable_rule (test.test_peg_generator.test_first_sets.TestFirstSets.test_nullable_rule) ... ok test_optional_after (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_after) ... ok test_optional_before (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_before) ... ok test_optional_literal (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_literal) ... ok test_optional_operator (test.test_peg_generator.test_first_sets.TestFirstSets.test_optional_operator) ... ok test_optionals (test.test_peg_generator.test_first_sets.TestFirstSets.test_optionals) ... ok test_positive_lookahead (test.test_peg_generator.test_first_sets.TestFirstSets.test_positive_lookahead) ... ok test_repeat_0 (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_0) ... ok test_repeat_0_with_group (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_0_with_group) ... ok test_repeat_1 (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_1) ... ok test_repeat_1_with_group (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_1_with_group) ... ok test_repeat_with_separator (test.test_peg_generator.test_first_sets.TestFirstSets.test_repeat_with_separator) ... ok test_rule_with_collision_after_some_other_rules (test.test_peg_generator.test_grammar_validator.TestPegen.test_rule_with_collision_after_some_other_rules) ... ok test_rule_with_no_collision (test.test_peg_generator.test_grammar_validator.TestPegen.test_rule_with_no_collision) ... ok test_rule_with_simple_collision (test.test_peg_generator.test_grammar_validator.TestPegen.test_rule_with_simple_collision) ... ok test_deep_nested_rule (test.test_peg_generator.test_pegen.TestGrammarVisualizer.test_deep_nested_rule) ... ok test_multiple_rules (test.test_peg_generator.test_pegen.TestGrammarVisualizer.test_multiple_rules) ... ok test_simple_rule (test.test_peg_generator.test_pegen.TestGrammarVisualizer.test_simple_rule) ... ok test_advanced_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_advanced_left_recursive) ... ok test_alt_optional_operator (test.test_peg_generator.test_pegen.TestPegen.test_alt_optional_operator) ... ok test_bad_token_reference (test.test_peg_generator.test_pegen.TestPegen.test_bad_token_reference) ... ok test_cut (test.test_peg_generator.test_pegen.TestPegen.test_cut) ... ok test_dangling_reference (test.test_peg_generator.test_pegen.TestPegen.test_dangling_reference) ... ok test_expr_grammar (test.test_peg_generator.test_pegen.TestPegen.test_expr_grammar) ... ok test_forced (test.test_peg_generator.test_pegen.TestPegen.test_forced) ... ok test_forced_with_group (test.test_peg_generator.test_pegen.TestPegen.test_forced_with_group) ... ok test_gather (test.test_peg_generator.test_pegen.TestPegen.test_gather) ... ok test_invalid_rule_name (test.test_peg_generator.test_pegen.TestPegen.test_invalid_rule_name) ... ok test_invalid_variable_name (test.test_peg_generator.test_pegen.TestPegen.test_invalid_variable_name) ... ok test_invalid_variable_name_in_temporal_rule (test.test_peg_generator.test_pegen.TestPegen.test_invalid_variable_name_in_temporal_rule) ... ok test_left_recursion_too_complex (test.test_peg_generator.test_pegen.TestPegen.test_left_recursion_too_complex) ... ok test_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_left_recursive) ... ok test_locations_in_alt_action_and_group (test.test_peg_generator.test_pegen.TestPegen.test_locations_in_alt_action_and_group) ... ok test_long_rule_str (test.test_peg_generator.test_pegen.TestPegen.test_long_rule_str) ... ok test_lookahead (test.test_peg_generator.test_pegen.TestPegen.test_lookahead) ... ok test_missing_start (test.test_peg_generator.test_pegen.TestPegen.test_missing_start) ... ok test_mutually_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_mutually_left_recursive) ... ok test_named_lookahead_error (test.test_peg_generator.test_pegen.TestPegen.test_named_lookahead_error) ... ok test_nasty_mutually_left_recursive (test.test_peg_generator.test_pegen.TestPegen.test_nasty_mutually_left_recursive) ... ok test_nullable (test.test_peg_generator.test_pegen.TestPegen.test_nullable) ... ok test_opt_sequence (test.test_peg_generator.test_pegen.TestPegen.test_opt_sequence) ... ok test_optional_literal (test.test_peg_generator.test_pegen.TestPegen.test_optional_literal) ... ok test_optional_operator (test.test_peg_generator.test_pegen.TestPegen.test_optional_operator) ... ok test_parse_grammar (test.test_peg_generator.test_pegen.TestPegen.test_parse_grammar) ... ok test_python_expr (test.test_peg_generator.test_pegen.TestPegen.test_python_expr) ... ok test_repeat_0_complex (test.test_peg_generator.test_pegen.TestPegen.test_repeat_0_complex) ... ok test_repeat_0_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_0_simple) ... ok test_repeat_1_complex (test.test_peg_generator.test_pegen.TestPegen.test_repeat_1_complex) ... ok test_repeat_1_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_1_simple) ... ok test_repeat_with_sep_simple (test.test_peg_generator.test_pegen.TestPegen.test_repeat_with_sep_simple) ... ok test_soft_keyword (test.test_peg_generator.test_pegen.TestPegen.test_soft_keyword) ... ok test_start_leader (test.test_peg_generator.test_pegen.TestPegen.test_start_leader) ... ok test_typed_rules (test.test_peg_generator.test_pegen.TestPegen.test_typed_rules) ... ok test_unreachable_explicit (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_explicit) ... ok test_unreachable_implicit1 (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_implicit1) ... ok test_unreachable_implicit2 (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_implicit2) ... ok test_unreachable_implicit3 (test.test_peg_generator.test_pegen.TestPegen.test_unreachable_implicit3) ... ok ====================================================================== ERROR: setUpClass (test.test_peg_generator.test_c_parser.TestCParser) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Users\KIRILL-1\test_cpython\cpython\Lib\test\test_peg_generator\test_c_parser.py", line 95, in setUpClass sitepackages = subprocess.check_output( ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\KIRILL-1\test_cpython\cpython\Lib\subprocess.py", line 466, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\KIRILL-1\test_cpython\cpython\Lib\subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['C:\\Users\\KIRILL-1\\test_cpython\\cpython\\build\\test_python_22060æ\\tempcwd\\venv\\Scripts\\python_d.exe', '-c', "import sysconfig; print(sysconfig.get_path('platlib'))"]' returned non-zero exit status 1. ---------------------------------------------------------------------- Ran 66 tests in 25.305s FAILED (errors=1) test test_peg_generator failed test_peg_generator failed (1 error) == Tests result: FAILURE == 1 test failed: test_peg_generator Total duration: 26.5 sec Tests result: FAILURE PS C:\Users\KIRILL-1\test_cpython\cpython> ```
erlend-aasland commented 1 year ago

@Eclips4, try with a UTF-8 setup.

Eclips4 commented 1 year ago

@Eclips4, try with a UTF-8 setup.

Seems that with UTF-8 encoding test passes. But, cp1251 is a common encoding for Windows.

sunmy2019 commented 1 year ago

(Copied from #105087) Previously, test_peg_generator is skipped -- distutils has been removed in Python 3.12

Now it's added back in https://github.com/python/cpython/pull/104798 and is leaking references.

Recent builtbot results: https://buildbot.python.org/all/#/builders/259/builds/775

test_peg_generator leaked [68, 68, 68] references, sum=204
test_peg_generator leaked [52, 52, 52] memory blocks, sum=156

Seems all ref leaks come from Tools/peg_generator/pegen/build.py : compile_c_extension

But no matter how many times we call this function, the leaks are the same as if it was called once.

This behavior looks like a cache was created and forgotten.

lysnikolaou commented 1 year ago

FYI we still haven't been able to find a solution to this. We've bisected the problem down to a call to setuptools.Distribution. I've opened an issue on this on pypa/setuptools#3938 to figure out what the correct course of action is.

sunmy2019 commented 1 year ago

Hi, I tracked down part of the ref leaks to this cache:

https://github.com/python/cpython/blob/c6ca368867bd68d44f333df840aa85d425a51410/Lib/importlib/_functools.py#L6-L85

Used Here: https://github.com/python/cpython/blob/c6ca368867bd68d44f333df840aa85d425a51410/Lib/importlib/metadata.py#L689-L691

Setting 

-    cache_wrapper = cache_wrapper or functools.lru_cache()
+    cache_wrapper = cache_wrapper or functools.lru_cache(maxsize=0)

reduced the ref leaks.

sunmy2019 commented 1 year ago

Part of the reference seems (not sure) to come from https://github.com/python/cpython/blob/3fe7d7c020a8b2d395a58bfafbe689ee36f7fe30/Modules/_ctypes/callproc.c#L1877-L1926

string LP_... and string _type_ seem to be leaked.

sunmy2019 commented 1 year ago

The rest ref leaks all came from here

https://github.com/pypa/setuptools/blob/b545fc778583f644d6c331773dbe0ea53bfa41af/setuptools/_distutils/dir_util.py#L12

# cache for by mkpath() -- in addition to cheapening redundant calls,
# eliminates redundant "creating /foo/bar/baz" messages in dry-run mode
_path_created = {}   # <-- not cleared

def mkpath(name, mode=0o777, verbose=1, dry_run=0):  # noqa: C901
    """Create a directory and any missing ancestor directories.

    If the directory already exists (or if 'name' is the empty string, which
    means the current directory, which of course exists), then do nothing.
    Raise DistutilsFileError if unable to create some directory along the way
    (eg. some sub-path exists, but is a file rather than a directory).
    If 'verbose' is true, print a one-line summary of each mkdir to stdout.
    Return the list of directories actually created.

    os.makedirs is not used because:

    a) It's new to Python 1.5.2, and
    b) it blows up if the directory already exists (in which case it should
       silently succeed).
    """

That should be all for the ref leaks on Linux.

sunmy2019 commented 1 year ago

I will leave the fix to someone else. You can add me as the co-author of the fix if you wish to fix this.

lysnikolaou commented 1 year ago

@sunmy2019 Thanks a lot for spending time on this! I'll work on opening separate issues if needed and fixing those leaks, probably early next week.

erlend-aasland commented 1 year ago

@lysnikolaou, any updates on this? Perhaps we should skip rerunning these tests for now?

lysnikolaou commented 1 year ago

Unfortunately I didn't have the time to look into this further.

Perhaps we should skip rerunning these tests for now?

Yup, that's probably a good idea.

ambv commented 1 year ago

Keeping this open until a real fix is reintroduced.