kloczek
commented
11 months ago Yeah and other thing is DeprecationWarning about use pkg_resources.
Open kloczek opened 11 months ago
kloczek
commented
11 months ago Yeah and other thing is DeprecationWarning about use pkg_resources.
kloczek
commented
11 months ago After add test_evm.py to --ignore list pytest stil is failing
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/capstone-4.0.2-15.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/capstone-4.0.2-15.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network' --ignore test_evm.py
============================= test session starts ==============================
platform linux -- Python 3.8.18, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/tkloczko/rpmbuild/BUILD/capstone-4.0.2/bindings/python
collected 19 items
test_arm.py . [ 5%]
test_arm64.py . [ 10%]
test_basic.py F. [ 21%]
test_customized_mnem.py . [ 26%]
test_detail.py . [ 31%]
test_lite.py .. [ 42%]
test_m680x.py . [ 47%]
test_m68k.py . [ 52%]
test_mips.py . [ 57%]
test_ppc.py . [ 63%]
test_skipdata.py E. [ 73%]
test_sparc.py . [ 78%]
test_systemz.py . [ 84%]
test_tms320c64x.py . [ 89%]
test_x86.py . [ 94%]
test_xcore.py . [100%]
==================================== ERRORS ====================================
___________________________ ERROR at setup of testcb ___________________________
file /home/tkloczko/rpmbuild/BUILD/capstone-4.0.2/bindings/python/test_skipdata.py, line 21
def testcb(buffer, size, offset, userdata):
E fixture 'buffer' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.
/home/tkloczko/rpmbuild/BUILD/capstone-4.0.2/bindings/python/test_skipdata.py:21
=================================== FAILURES ===================================
_____________________________ test_cs_disasm_quick _____________________________
def test_cs_disasm_quick():
for arch, mode, code, comment, syntax in all_tests:
print('*' * 40)
print("Platform: %s" % comment)
print("Disasm:"),
print(to_hex(code))
for insn in cs_disasm_quick(arch, mode, code, 0x1000):
> print("0x%x:\t%s\t%s" % (insn.address, insn.mnemonic, insn.op_str))
test_basic.py:75:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <[CsError(7) raised in repr()] CsInsn object at 0x7f06413f3a00>
name = 'mnemonic'
def __getattr__(self, name):
if not self._cs._detail:
> raise CsError(CS_ERR_DETAIL)
E capstone.CsError: Details are unavailable (CS_ERR_DETAIL)
capstone/__init__.py:667: CsError
----------------------------- Captured stdout call -----------------------------
****************************************
Platform: X86 16bit (Intel syntax)
Disasm:
0x8d 0x4c 0x32 0x08 0x01 0xd8 0x81 0xc6 0x34 0x12 0x00 0x00
=============================== warnings summary ===============================
capstone/__init__.py:267
/home/tkloczko/rpmbuild/BUILD/capstone-4.0.2/bindings/python/capstone/__init__.py:267: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
ERROR test_skipdata.py::testcb
FAILED test_basic.py::test_cs_disasm_quick - capstone.CsError: Details are un...
=============== 1 failed, 17 passed, 1 warning, 1 error in 0.61s ===============Cannot find which one module provides buffer pytest fixture
kabeor
commented
11 months ago Hi, v4.0.2 is not maintained any more, plz using v5 branch.
Yeah and other thing is
DeprecationWarningabout usepkg_resources.
This is a known issue and will be fixed soon, see #2005.
kloczek
commented
11 months ago I've started working on that. Here is the proposed patch
--- a/bindings/python/capstone/__init__.py
+++ b/bindings/python/capstone/__init__.py
@@ -1,5 +1,8 @@
# Capstone Python bindings, by Nguyen Anh Quynnh <aquynh@gmail.com>
-import os, sys
+import os
+import sys
+import sysconfig
+
from platform import system
_python2 = sys.version_info[0] < 3
if _python2:
@@ -376,8 +379,6 @@
import ctypes, ctypes.util
from os.path import split, join, dirname
-import distutils.sysconfig
-import pkg_resources
import inspect
if not hasattr(sys.modules[__name__], '__file__'):
@@ -407,17 +408,15 @@
# Loading attempts, in order
# - user-provided environment variable
-# - pkg_resources can get us the path to the local libraries
+# - python's lib directory
# - we can get the path to the local libraries by parsing our filename
# - global load
-# - python's lib directory
# - last-gasp attempt at some hardcoded paths on darwin and linux
_path_list = [os.getenv('LIBCAPSTONE_PATH', None),
- pkg_resources.resource_filename(__name__, 'lib'),
+ sysconfig.get_config_var('LIBDIR'),
join(split(__file__)[0], 'lib'),
'',
- distutils.sysconfig.get_python_lib(),
"/usr/local/lib/" if sys.platform == 'darwin' else '/usr/lib64']
for _path in _path_list:However with that patch pytest still is failing
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/capstone-5.0.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/capstone-5.0.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.18, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1
collected 35 items / 2 errors
========================================================================================== ERRORS ===========================================================================================
___________________________________________________________________________ ERROR collecting suite/test_corpus.py ___________________________________________________________________________
/usr/lib/python3.8/site-packages/_pytest/python.py:622: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:567: in import_path
importlib.import_module(module_name)
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
???
<frozen importlib._bootstrap>:991: in _find_and_load
???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:671: in _load_unlocked
???
/usr/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:169: in exec_module
source_stat, co = _rewrite_test(fn, self.config)
/usr/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:351: in _rewrite_test
tree = ast.parse(source, filename=strfn)
/usr/lib64/python3.8/ast.py:47: in parse
return compile(source, filename, mode, flags,
E File "/home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_corpus.py", line 137
E print "skipping", hex_code
E ^
E SyntaxError: Missing parentheses in call to 'print'. Did you mean print("skipping", hex_code)?
________________________________________________________________ ERROR collecting suite/regress/test_arm64_ldr_registers.py _________________________________________________________________
/usr/lib/python3.8/site-packages/_pytest/runner.py:341: in from_call
result: Optional[TResult] = func()
/usr/lib/python3.8/site-packages/_pytest/runner.py:372: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/usr/lib/python3.8/site-packages/_pytest/python.py:536: in collect
self._inject_setup_module_fixture()
/usr/lib/python3.8/site-packages/_pytest/python.py:550: in _inject_setup_module_fixture
self.obj, ("setUpModule", "setup_module")
/usr/lib/python3.8/site-packages/_pytest/python.py:315: in obj
self._obj = obj = self._getobj()
/usr/lib/python3.8/site-packages/_pytest/python.py:533: in _getobj
return self._importtestmodule()
/usr/lib/python3.8/site-packages/_pytest/python.py:622: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:567: in import_path
importlib.import_module(module_name)
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
???
<frozen importlib._bootstrap>:991: in _find_and_load
???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:671: in _load_unlocked
???
/usr/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
exec(co, module.__dict__)
suite/regress/test_arm64_ldr_registers.py:5: in <module>
_python3 = sys.version_info.major == 3
E NameError: name 'sys' is not defined
================================================================================== short test summary info ==================================================================================
ERROR suite/test_corpus.py
ERROR suite/regress/test_arm64_ldr_registers.py - NameError: name 'sys' is not defined
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================================================== 2 errors in 1.45s =====================================================================================Looks like there are some python 2.x leftovers. I've added patch to remove python 2.x syntax:
--- a/suite/test_corpus.py (original)
+++ b/suite/test_corpus.py (refactored)
@@ -134,11 +134,11 @@
try:
hex_data = hex_code.strip().decode('hex')
except:
- print "skipping", hex_code
+ print("skipping", hex_code)
fout = open("fuzz/corpus/%s_%s" % (os.path.basename(fname), hex_code), 'w')
if (arch, mode) not in mc_modes:
- print "fail", arch, mode
- fout.write(unichr(mc_modes[(arch, mode)]))
+ print("fail", arch, mode)
+ fout.write(chr(mc_modes[(arch, mode)]))
fout.write(hex_data)
fout.close()
--- a/suite/regress/test_arm64_ldr_registers.py
+++ b/suite/regress/test_arm64_ldr_registers.py
@@ -2,9 +2,6 @@
from capstone import *
from capstone.arm64 import *
-_python3 = sys.version_info.major == 3
-
-
class SubRegTest(unittest.TestCase):
PATTERNS = [
@@ -21,10 +18,7 @@
self.cs.detail = True
for pattern, asm in self.PATTERNS:
- if _python3:
- l = list(self.cs.disasm(bytes.fromhex(pattern), 0))
- else:
- l = list(self.cs.disasm(bytearray.fromhex(pattern), 0))
+ l = list(self.cs.disasm(bytes.fromhex(pattern), 0))
self.assertTrue(len(l) == 1)
_, expected_reg_written, expected_reg_read = asm.split()However even with that patch pytest still fails
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/capstone-5.0.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/capstone-5.0.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.18, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1
collected 38 items
bindings/python/test_arm.py . [ 2%]
bindings/python/test_arm64.py . [ 5%]
bindings/python/test_basic.py F.. [ 13%]
bindings/python/test_bpf.py . [ 15%]
bindings/python/test_customized_mnem.py . [ 18%]
bindings/python/test_detail.py . [ 21%]
bindings/python/test_evm.py . [ 23%]
bindings/python/test_lite.py .. [ 28%]
bindings/python/test_m680x.py . [ 31%]
bindings/python/test_m68k.py . [ 34%]
bindings/python/test_mips.py . [ 36%]
bindings/python/test_mos65xx.py . [ 39%]
bindings/python/test_ppc.py . [ 42%]
bindings/python/test_riscv.py . [ 44%]
bindings/python/test_sh.py . [ 47%]
bindings/python/test_skipdata.py E. [ 52%]
bindings/python/test_sparc.py . [ 55%]
bindings/python/test_systemz.py . [ 57%]
bindings/python/test_tms320c64x.py . [ 60%]
bindings/python/test_tricore.py . [ 63%]
bindings/python/test_wasm.py . [ 65%]
bindings/python/test_x86.py . [ 68%]
bindings/python/test_xcore.py . [ 71%]
suite/test_corpus.py E [ 73%]
suite/test_corpus3.py E [ 76%]
suite/test_mc.py E [ 78%]
suite/regress/test_arm64_bra.py .. [ 84%]
suite/regress/test_arm64_ldr_registers.py FF [ 89%]
suite/regress/test_arm64_mov.py .. [ 94%]
suite/regress/test_arm64_pac.py .. [100%]
========================================================================================== ERRORS ===========================================================================================
_________________________________________________________________________________ ERROR at setup of testcb __________________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/bindings/python/test_skipdata.py, line 21
def testcb(buffer, size, offset, userdata):
E fixture 'buffer' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.
/home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/bindings/python/test_skipdata.py:21
________________________________________________________________________________ ERROR at setup of test_file ________________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_corpus.py, line 7
def test_file(fname):
E fixture 'fname' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.
/home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_corpus.py:7
________________________________________________________________________________ ERROR at setup of test_file ________________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_corpus3.py, line 10
def test_file(fname):
E fixture 'fname' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.
/home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_corpus3.py:10
________________________________________________________________________________ ERROR at setup of test_file ________________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_mc.py, line 67
def test_file(fname):
E fixture 'fname' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.
/home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_mc.py:67
========================================================================================= FAILURES ==========================================================================================
___________________________________________________________________________________ test_cs_disasm_quick ____________________________________________________________________________________
def test_cs_disasm_quick():
for arch, mode, code, comment, syntax in all_tests:
print('*' * 40)
print("Platform: %s" % comment)
print("Disasm:"),
print(to_hex(code))
for insn in cs_disasm_quick(arch, mode, code, 0x1000):
> print("0x%x:\t%s\t%s" % (insn.address, insn.mnemonic, insn.op_str))
bindings/python/test_basic.py:82:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <[CsError(7) raised in repr()] CsInsn object at 0x7fc3a7fa3940>, name = 'mnemonic'
def __getattr__(self, name):
if not self._cs._detail:
> raise CsError(CS_ERR_DETAIL)
E capstone.CsError: Details are unavailable (CS_ERR_DETAIL)
bindings/python/capstone/__init__.py:798: CsError
----------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------
****************************************
Platform: X86 16bit (Intel syntax)
Disasm:
0x8d 0x4c 0x32 0x08 0x01 0xd8 0x81 0xc6 0x34 0x12 0x00 0x00
_________________________________________________________________________________ SubRegTest.test_operands __________________________________________________________________________________
self = <test_arm64_ldr_registers.SubRegTest testMethod=test_operands>
def test_operands(self):
"""Check that the `operands` API provides correct data"""
for inst, asm, expected_regs in self.insts:
ops = inst.operands
> self.assertEqual(len(ops), 2)
E AssertionError: 1 != 2
suite/regress/test_arm64_ldr_registers.py:48: AssertionError
_________________________________________________________________________________ SubRegTest.test_registers _________________________________________________________________________________
self = <test_arm64_ldr_registers.SubRegTest testMethod=test_registers>
def test_registers(self):
"""Check that the `regs_access` API provides correct data"""
for inst, asm, expected_regs in self.insts:
# Check that the instruction writes the first register operand and reads the second
for i, decoded_regs in enumerate(map(lambda l: list(map(self.cs.reg_name, l)), inst.regs_access())):
> self.assertEqual(len(decoded_regs), 1, "%s has %d %s registers instead of 1" % (asm, len(decoded_regs), ["read", "written"][i]))
E AssertionError: 0 != 1 : ldr x1, [x2] has 0 read registers instead of 1
suite/regress/test_arm64_ldr_registers.py:40: AssertionError
================================================================================== short test summary info ==================================================================================
ERROR bindings/python/test_skipdata.py::testcb
ERROR suite/test_corpus.py::test_file
ERROR suite/test_corpus3.py::test_file
ERROR suite/test_mc.py::test_file
FAILED bindings/python/test_basic.py::test_cs_disasm_quick - capstone.CsError: Details are unavailable (CS_ERR_DETAIL)
FAILED suite/regress/test_arm64_ldr_registers.py::SubRegTest::test_operands - AssertionError: 1 != 2
FAILED suite/regress/test_arm64_ldr_registers.py::SubRegTest::test_registers - AssertionError: 0 != 1 : ldr x1, [x2] has 0 read registers instead of 1
========================================================================== 3 failed, 31 passed, 4 errors in 1.41s ===========================================================================Cannot find what provides those missing pytest fixtures 🤔
Please let me know if you want above patch es as PR.
XVilka
commented
11 months ago @kloczek patch looks good, IMHO. Please send it as a PR to the next branch. Then it can be cherry-picked to the potential 5.0.2 release: https://github.com/capstone-engine/capstone/issues/2081
I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
python3 -sBm build -w --no-isolationbuildwith--no-isolationI'm using during all processes only locally installed modulescut off from access to the public network(pytest is executed with-m "not network")Here is pytest output:
Here is list of installed modules in build env