search

rocky / python-xdis

Python cross-version bytecode library and disassembler
GNU General Public License v2.0
292 stars 95 forks source link

[Windows] make check issues #147

Closed greenozon closed 2 months ago

greenozon commented 2 months ago

There are some issues that stop running of TC under Windows

after fixing Access Denied one that was stopping TC run on a very early phase, there is another one now, that hits later on:

c:\Dev\python\python-xdis-orig>make check
python -m pytest pytest
========================================================================================= test session starts =========================================================================================
platform win32 -- Python 3.12.5, pytest-8.3.2, pluggy-1.5.0
rootdir: c:\Dev\python\python-xdis-orig
configfile: pyproject.toml
collected 35 items

pytest\test_bytecode.py ..                                                                                                                                                                       [  5%]
pytest\test_codetype.py .                                                                                                                                                                        [  8%]
pytest\test_cross_dis.py .                                                                                                                                                                       [ 11%]
pytest\test_disasm.py sssssssss                                                                                                                                                                  [ 37%]
pytest\test_instructions.py ..                                                                                                                                                                   [ 42%]
pytest\test_load_file.py s                                                                                                                                                                       [ 45%]
pytest\test_magic.py .                                                                                                                                                                           [ 48%]
pytest\test_opcode.py .                                                                                                                                                                          [ 51%]
pytest\test_stack_effect.py ..                                                                                                                                                                   [ 57%]
pytest\test_std.py ............s..                                                                                                                                                               [100%]

========================================================================================== warnings summary ===========================================================================================
pytest/test_codetype.py::test_codeType2Portable
  c:\Dev\python\python-xdis-orig\xdis\codetype\__init__.py:44: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_table_field = "co_lnotab" if hasattr(code, "co_lnotab") else "co_linetable"

pytest/test_codetype.py::test_codeType2Portable
  c:\Dev\python\python-xdis-orig\xdis\codetype\__init__.py:45: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_table = getattr(code, line_table_field)

pytest/test_std.py::test_write_bytecode_file
pytest/test_std.py::test_write_bytecode_bad_timestamp_type
  c:\Dev\python\python-xdis-orig\pytest\test_std.py:214: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    fn_code.co_lnotab,  # In general, You should adjust this

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= 24 passed, 11 skipped, 4 warnings in 0.29s ==============================================================================
make[1]: Entering directory '/c/Dev/python/python-xdis-orig/test'
python test_pyenvlib.py --simple
Sat Aug 17 17:01:27 2024
Processed 24 files, total
Sat Aug 17 17:01:27 2024
0.00500011 seconds
Sat Aug 17 17:01:27 2024
Processed 1 files, total
Sat Aug 17 17:01:27 2024
0.0120006 seconds
python test_pythonlib.py \
          --bytecode-1.0 --bytecode-1.1 --bytecode-1.2 --bytecode-1.3 \
                  --bytecode-1.4 --bytecode-1.5 --bytecode-1.6 \
          --bytecode-2.1 --bytecode-2.2 --bytecode-2.3 \
                  --bytecode-2.4 --bytecode-2.5 \
          --bytecode-2.5dropbox \
                  --bytecode-2.6 --bytecode-2.7 \
          --bytecode-3.0 --bytecode-3.1 \
                  --bytecode-3.2 --bytecode-3.3 \
                  --bytecode-3.4 --bytecode-3.5 \
          --bytecode-3.6 --bytecode-3.7 --bytecode-3.8 \
          --bytecode-3.9 --bytecode-3.10 \
          --bytecode-2.7pypy --bytecode-3.2pypy \
                  --bytecode-pypy35 --bytecode-pypy36 --bytecode-pypy37 --bytecode-pypy38
Sat Aug 17 17:01:27 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.0
Sat Aug 17 17:01:27 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.1
Sat Aug 17 17:01:27 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.2
Sat Aug 17 17:01:27 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.3
Sat Aug 17 17:01:28 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.4
Sat Aug 17 17:01:28 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.5
Sat Aug 17 17:01:28 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.6
Sat Aug 17 17:01:28 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.1
Sat Aug 17 17:01:28 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.2
Sat Aug 17 17:01:28 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.3
Traceback (most recent call last):
  File "C:\Dev\python\python-xdis-orig\test\test_pythonlib.py", line 292, in <module>
    do_tests(src_dir, pattern, target_dir, test_opts)
  File "C:\Dev\python\python-xdis-orig\test\test_pythonlib.py", line 210, in do_tests
    disassemble_file(infile, output)
  File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\site-packages\xdis\disasm.py", line 371, in disassemble_file
    disco(
  File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\site-packages\xdis\disasm.py", line 163, in disco
    real_out.write(format_code_info(co, version_tuple, is_graal=is_graal) + "\n")
  File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\encodings\cp1251.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\xe7' in position 216: character maps to <undefined>
make[1]: *** [Makefile:13: check] Error 1
make[1]: Leaving directory '/c/Dev/python/python-xdis-orig/test'
make: *** [Makefile:37: check] Error 2

My gut feeling is that there should be some treatment applied using the "utf-8" encoding Yeah, I agree, it works out of the box under any Linux, but we are special guys, the WIndows ones :)

greenozon commented 2 months ago

the workaround without touching code base might be this before runing TCs

set PYTHONUTF8=1
make check

eg:

c:\Dev\python\python-xdis-orig>set PYTHONUTF8=1

c:\Dev\python\python-xdis-orig>make check
python -m pytest pytest
========================================================================================= test session starts =========================================================================================
platform win32 -- Python 3.12.5, pytest-8.3.2, pluggy-1.5.0
rootdir: c:\Dev\python\python-xdis-orig
configfile: pyproject.toml
collected 35 items

pytest\test_bytecode.py ..                                                                                                                                                                       [  5%]
pytest\test_codetype.py .                                                                                                                                                                        [  8%]
pytest\test_cross_dis.py .                                                                                                                                                                       [ 11%]
pytest\test_disasm.py sssssssss                                                                                                                                                                  [ 37%]
pytest\test_instructions.py ..                                                                                                                                                                   [ 42%]
pytest\test_load_file.py s                                                                                                                                                                       [ 45%]
pytest\test_magic.py .                                                                                                                                                                           [ 48%]
pytest\test_opcode.py .                                                                                                                                                                          [ 51%]
pytest\test_stack_effect.py ..                                                                                                                                                                   [ 57%]
pytest\test_std.py ............s..                                                                                                                                                               [100%]

========================================================================================== warnings summary ===========================================================================================
pytest/test_codetype.py::test_codeType2Portable
  c:\Dev\python\python-xdis-orig\xdis\codetype\__init__.py:44: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_table_field = "co_lnotab" if hasattr(code, "co_lnotab") else "co_linetable"

pytest/test_codetype.py::test_codeType2Portable
  c:\Dev\python\python-xdis-orig\xdis\codetype\__init__.py:45: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_table = getattr(code, line_table_field)

pytest/test_std.py::test_write_bytecode_file
pytest/test_std.py::test_write_bytecode_bad_timestamp_type
  c:\Dev\python\python-xdis-orig\pytest\test_std.py:214: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    fn_code.co_lnotab,  # In general, You should adjust this

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= 24 passed, 11 skipped, 4 warnings in 0.29s ==============================================================================
make[1]: Entering directory '/c/Dev/python/python-xdis-orig/test'
python test_pyenvlib.py --simple
Sat Aug 17 19:19:32 2024
Processed 24 files, total
Sat Aug 17 19:19:32 2024
0.00500035 seconds
Sat Aug 17 19:19:32 2024
Processed 1 files, total
Sat Aug 17 19:19:32 2024
0.0120006 seconds
python test_pythonlib.py \
          --bytecode-1.0 --bytecode-1.1 --bytecode-1.2 --bytecode-1.3 \
                  --bytecode-1.4 --bytecode-1.5 --bytecode-1.6 \
          --bytecode-2.1 --bytecode-2.2 --bytecode-2.3 \
                  --bytecode-2.4 --bytecode-2.5 \
          --bytecode-2.5dropbox \
                  --bytecode-2.6 --bytecode-2.7 \
          --bytecode-3.0 --bytecode-3.1 \
                  --bytecode-3.2 --bytecode-3.3 \
                  --bytecode-3.4 --bytecode-3.5 \
          --bytecode-3.6 --bytecode-3.7 --bytecode-3.8 \
          --bytecode-3.9 --bytecode-3.10 \
          --bytecode-2.7pypy --bytecode-3.2pypy \
                  --bytecode-pypy35 --bytecode-pypy36 --bytecode-pypy37 --bytecode-pypy38
Sat Aug 17 19:19:32 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.0
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.1
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.2
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.3
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.4
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.5
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.6
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.1
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.2
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.3
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.4
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.5
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.5dropbox
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.6
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.7
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.0
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.1
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.2
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.3
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.4
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.5
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.6
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.7
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.8
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.9
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.10
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.7pypy
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.2pypy
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_pypy35
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_pypy36
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_pypy37
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_pypy38
make[1]: Leaving directory '/c/Dev/python/python-xdis-orig/test'

c:\Dev\python\python-xdis-orig>echo %ERRORLEVEL%
0

the quesiton is: do you think there is a need to make code changes for this regard?

rocky commented 2 months ago

the workaround without touching code base might be this before runing TCs

set PYTHONUTF8=1
make check

eg:

c:\Dev\python\python-xdis-orig>set PYTHONUTF8=1

c:\Dev\python\python-xdis-orig>make check
python -m pytest pytest
========================================================================================= test session starts =========================================================================================
platform win32 -- Python 3.12.5, pytest-8.3.2, pluggy-1.5.0
rootdir: c:\Dev\python\python-xdis-orig
configfile: pyproject.toml
collected 35 items

pytest\test_bytecode.py ..                                                                                                                                                                       [  5%]
pytest\test_codetype.py .                                                                                                                                                                        [  8%]
pytest\test_cross_dis.py .                                                                                                                                                                       [ 11%]
pytest\test_disasm.py sssssssss                                                                                                                                                                  [ 37%]
pytest\test_instructions.py ..                                                                                                                                                                   [ 42%]
pytest\test_load_file.py s                                                                                                                                                                       [ 45%]
pytest\test_magic.py .                                                                                                                                                                           [ 48%]
pytest\test_opcode.py .                                                                                                                                                                          [ 51%]
pytest\test_stack_effect.py ..                                                                                                                                                                   [ 57%]
pytest\test_std.py ............s..                                                                                                                                                               [100%]

========================================================================================== warnings summary ===========================================================================================
pytest/test_codetype.py::test_codeType2Portable
  c:\Dev\python\python-xdis-orig\xdis\codetype\__init__.py:44: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_table_field = "co_lnotab" if hasattr(code, "co_lnotab") else "co_linetable"

pytest/test_codetype.py::test_codeType2Portable
  c:\Dev\python\python-xdis-orig\xdis\codetype\__init__.py:45: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_table = getattr(code, line_table_field)

pytest/test_std.py::test_write_bytecode_file
pytest/test_std.py::test_write_bytecode_bad_timestamp_type
  c:\Dev\python\python-xdis-orig\pytest\test_std.py:214: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    fn_code.co_lnotab,  # In general, You should adjust this

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= 24 passed, 11 skipped, 4 warnings in 0.29s ==============================================================================
make[1]: Entering directory '/c/Dev/python/python-xdis-orig/test'
python test_pyenvlib.py --simple
Sat Aug 17 19:19:32 2024
Processed 24 files, total
Sat Aug 17 19:19:32 2024
0.00500035 seconds
Sat Aug 17 19:19:32 2024
Processed 1 files, total
Sat Aug 17 19:19:32 2024
0.0120006 seconds
python test_pythonlib.py \
          --bytecode-1.0 --bytecode-1.1 --bytecode-1.2 --bytecode-1.3 \
                  --bytecode-1.4 --bytecode-1.5 --bytecode-1.6 \
          --bytecode-2.1 --bytecode-2.2 --bytecode-2.3 \
                  --bytecode-2.4 --bytecode-2.5 \
          --bytecode-2.5dropbox \
                  --bytecode-2.6 --bytecode-2.7 \
          --bytecode-3.0 --bytecode-3.1 \
                  --bytecode-3.2 --bytecode-3.3 \
                  --bytecode-3.4 --bytecode-3.5 \
          --bytecode-3.6 --bytecode-3.7 --bytecode-3.8 \
          --bytecode-3.9 --bytecode-3.10 \
          --bytecode-2.7pypy --bytecode-3.2pypy \
                  --bytecode-pypy35 --bytecode-pypy36 --bytecode-pypy37 --bytecode-pypy38
Sat Aug 17 19:19:32 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.0
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.1
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.2
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.3
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.4
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.5
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_1.6
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.1
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.2
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.3
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.4
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.5
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.5dropbox
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.6
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.7
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.0
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.1
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.2
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.3
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.4
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.5
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.6
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.7
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.8
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.9
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.10
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_2.7pypy
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_3.2pypy
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_pypy35
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_pypy36
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_pypy37
Sat Aug 17 19:19:33 2024
Source directory:  C:\Dev\python\python-xdis-orig\test\bytecode_pypy38
make[1]: Leaving directory '/c/Dev/python/python-xdis-orig/test'

c:\Dev\python\python-xdis-orig>echo %ERRORLEVEL%
0

the quesiton is: do you think there is a need to make code changes for this regard?

If it is possible to set this from inside the Makefile that would be best. See https://earthly.dev/blog/makefile-variables/ for some ideas.

greenozon commented 2 months ago

thanks for the nice and educational material, I've refreshed my make knowledge and to my curiousity figured out new syntax that is var :::= value
Immediate Assignment, but it's very modern feature (added in make version 4.4)

The tricky thing in current design of Makefile is each test target changes the current dir into test\ (using -C test make argument) and spins up new make process so the simple assignment on top of makefile

var=value does not work in this case

for addressing this issue I did the export var=value approach and things started to work

export PYTHONUTF8=1

With GNU make, variables marked with export are only available to [the shells launched for] recipe commands (commands that are part of rules),

greenozon commented 2 months ago

One more question that touches the TCc run:

currently in modern python being used in the OS we do see these guys:

========================================================================================== warnings summary ===========================================================================================
pytest/test_codetype.py::test_codeType2Portable
  c:\Dev\python\python-xdis-my\python-xdis\xdis\codetype\__init__.py:44: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_table_field = "co_lnotab" if hasattr(code, "co_lnotab") else "co_linetable"

pytest/test_codetype.py::test_codeType2Portable
  c:\Dev\python\python-xdis-my\python-xdis\xdis\codetype\__init__.py:45: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_table = getattr(code, line_table_field)

pytest/test_std.py::test_write_bytecode_file
pytest/test_std.py::test_write_bytecode_bad_timestamp_type
  c:\Dev\python\python-xdis-my\python-xdis\pytest\test_std.py:214: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    fn_code.co_lnotab,  # In general, You should adjust this

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

do you think it makes sense to hide those DeprecationWarnings ?

as far as I understand you can't just refactor those warnings into modern features as you run TC on older pythons, right?

rocky commented 2 months ago

thanks for the nice and educational material, I've refreshed my make knowledge and to my curiousity figured out new syntax that is var :::= value Immediate Assignment, but it's very modern feature (added in make version 4.4)

If immediate variables work, let's use that. GNU Make 4.4 is about 4 years old. I don't see a reason to try to support a version of GNU make that is before 4.4.

The tricky thing in current design of Makefile is each test target changes the current dir into test\ (using -C test make argument) and spins up new make process so the simple assignment on top of makefile

If I have it correct, this is only failing for pytests, not the disassembly tests in the tests directory.

var=value does not work in this case

for addressing this issue I did the export var=value approach and things started to work

export PYTHONUTF8=1

With GNU make, variables marked with export are only available to [the shells launched for] recipe commands (commands that are part of rules),

rocky commented 2 months ago

One more question that touches the TCc run:

currently in modern python being used in the OS we do see these guys:

========================================================================================== warnings summary ===========================================================================================
pytest/test_codetype.py::test_codeType2Portable
  c:\Dev\python\python-xdis-my\python-xdis\xdis\codetype\__init__.py:44: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_table_field = "co_lnotab" if hasattr(code, "co_lnotab") else "co_linetable"

pytest/test_codetype.py::test_codeType2Portable
  c:\Dev\python\python-xdis-my\python-xdis\xdis\codetype\__init__.py:45: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    line_table = getattr(code, line_table_field)

pytest/test_std.py::test_write_bytecode_file
pytest/test_std.py::test_write_bytecode_bad_timestamp_type
  c:\Dev\python\python-xdis-my\python-xdis\pytest\test_std.py:214: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
    fn_code.co_lnotab,  # In general, You should adjust this

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

do you think it makes sense to hide those DeprecationWarnings ?

No, please don't hide the deprecation warning but instead fix them.

as far as I understand you can't just refactor those warnings into modern features as you run TC on older pythons, right?

Handling this is a bit more difficult than the suggested fix because the code object might sometimes have co_lnotab and sometimes it may have co_linetable. So I think one needs to test the code object to using the hasattr() function.

Supporting older versions of Python is done via different github branches; the master branch is for Python 3.11 up. For Python 3.6 to 3.10 the branch names is python-3.6-to-3.10 and so on.

greenozon commented 2 months ago

happy Sunday!

If immediate variables work, let's use that. GNU Make 4.4 is about 4 years old. I don't see a reason to try to support a version of GNU make that is before 4.4.

Well, it doesn't help... PYTHONUTF8 :::= 1 it is more about how $var expanding works, etc

If using this approach we hit the old good case:

...
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\xe7' in position 216: character maps to <undefined>

re

If I have it correct, this is only failing for pytests, not the disassembly tests in the tests directory.

yes, thats correct I was thinking where to put the utf8 var, as there are options for sure the general intention was that it might happen that in the future new TC might hit the case again, so having that var on a wider scope potentially covers future pitfalls, but if you don't like polluting env with extra stuff not required, lets limit the scope

PS: PR updated

greenozon commented 2 months ago

going back to temp dir on Windows: you are right, the .pyc files are gone (in case there is no exception during TC run) but now there is another leftovers observed: the empty dirs as follows:

image

name pattern: py-dis-xxxxxxxx

expectation: empty dirs are cleaned up (removed) after TC run

rocky commented 2 months ago

When there is a failure, you have to remove those files individually.