kloczek
commented
1 year ago Yeah and other thing is DeprecationWarning about use pkg_resources.
Closed kloczek closed 2 months ago
kloczek
commented
1 year ago Yeah and other thing is DeprecationWarning about use pkg_resources.
kloczek
commented
1 year ago After add test_evm.py to --ignore list pytest stil is failing
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/capstone-4.0.2-15.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/capstone-4.0.2-15.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network' --ignore test_evm.py
============================= test session starts ==============================
platform linux -- Python 3.8.18, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/tkloczko/rpmbuild/BUILD/capstone-4.0.2/bindings/python
collected 19 items
test_arm.py . [ 5%]
test_arm64.py . [ 10%]
test_basic.py F. [ 21%]
test_customized_mnem.py . [ 26%]
test_detail.py . [ 31%]
test_lite.py .. [ 42%]
test_m680x.py . [ 47%]
test_m68k.py . [ 52%]
test_mips.py . [ 57%]
test_ppc.py . [ 63%]
test_skipdata.py E. [ 73%]
test_sparc.py . [ 78%]
test_systemz.py . [ 84%]
test_tms320c64x.py . [ 89%]
test_x86.py . [ 94%]
test_xcore.py . [100%]
==================================== ERRORS ====================================
___________________________ ERROR at setup of testcb ___________________________
file /home/tkloczko/rpmbuild/BUILD/capstone-4.0.2/bindings/python/test_skipdata.py, line 21
def testcb(buffer, size, offset, userdata):
E fixture 'buffer' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.
/home/tkloczko/rpmbuild/BUILD/capstone-4.0.2/bindings/python/test_skipdata.py:21
=================================== FAILURES ===================================
_____________________________ test_cs_disasm_quick _____________________________
def test_cs_disasm_quick():
for arch, mode, code, comment, syntax in all_tests:
print('*' * 40)
print("Platform: %s" % comment)
print("Disasm:"),
print(to_hex(code))
for insn in cs_disasm_quick(arch, mode, code, 0x1000):
> print("0x%x:\t%s\t%s" % (insn.address, insn.mnemonic, insn.op_str))
test_basic.py:75:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <[CsError(7) raised in repr()] CsInsn object at 0x7f06413f3a00>
name = 'mnemonic'
def __getattr__(self, name):
if not self._cs._detail:
> raise CsError(CS_ERR_DETAIL)
E capstone.CsError: Details are unavailable (CS_ERR_DETAIL)
capstone/__init__.py:667: CsError
----------------------------- Captured stdout call -----------------------------
****************************************
Platform: X86 16bit (Intel syntax)
Disasm:
0x8d 0x4c 0x32 0x08 0x01 0xd8 0x81 0xc6 0x34 0x12 0x00 0x00
=============================== warnings summary ===============================
capstone/__init__.py:267
/home/tkloczko/rpmbuild/BUILD/capstone-4.0.2/bindings/python/capstone/__init__.py:267: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
ERROR test_skipdata.py::testcb
FAILED test_basic.py::test_cs_disasm_quick - capstone.CsError: Details are un...
=============== 1 failed, 17 passed, 1 warning, 1 error in 0.61s ===============Cannot find which one module provides buffer pytest fixture
kabeor
commented
1 year ago Hi, v4.0.2 is not maintained any more, plz using v5 branch.
Yeah and other thing is
DeprecationWarningabout usepkg_resources.
This is a known issue and will be fixed soon, see #2005.
kloczek
commented
1 year ago I've started working on that. Here is the proposed patch
--- a/bindings/python/capstone/__init__.py
+++ b/bindings/python/capstone/__init__.py
@@ -1,5 +1,8 @@
# Capstone Python bindings, by Nguyen Anh Quynnh <aquynh@gmail.com>
-import os, sys
+import os
+import sys
+import sysconfig
+
from platform import system
_python2 = sys.version_info[0] < 3
if _python2:
@@ -376,8 +379,6 @@
import ctypes, ctypes.util
from os.path import split, join, dirname
-import distutils.sysconfig
-import pkg_resources
import inspect
if not hasattr(sys.modules[__name__], '__file__'):
@@ -407,17 +408,15 @@
# Loading attempts, in order
# - user-provided environment variable
-# - pkg_resources can get us the path to the local libraries
+# - python's lib directory
# - we can get the path to the local libraries by parsing our filename
# - global load
-# - python's lib directory
# - last-gasp attempt at some hardcoded paths on darwin and linux
_path_list = [os.getenv('LIBCAPSTONE_PATH', None),
- pkg_resources.resource_filename(__name__, 'lib'),
+ sysconfig.get_config_var('LIBDIR'),
join(split(__file__)[0], 'lib'),
'',
- distutils.sysconfig.get_python_lib(),
"/usr/local/lib/" if sys.platform == 'darwin' else '/usr/lib64']
for _path in _path_list:However with that patch pytest still is failing
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/capstone-5.0.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/capstone-5.0.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.18, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1
collected 35 items / 2 errors
========================================================================================== ERRORS ===========================================================================================
___________________________________________________________________________ ERROR collecting suite/test_corpus.py ___________________________________________________________________________
/usr/lib/python3.8/site-packages/_pytest/python.py:622: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:567: in import_path
importlib.import_module(module_name)
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
???
<frozen importlib._bootstrap>:991: in _find_and_load
???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:671: in _load_unlocked
???
/usr/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:169: in exec_module
source_stat, co = _rewrite_test(fn, self.config)
/usr/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:351: in _rewrite_test
tree = ast.parse(source, filename=strfn)
/usr/lib64/python3.8/ast.py:47: in parse
return compile(source, filename, mode, flags,
E File "/home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_corpus.py", line 137
E print "skipping", hex_code
E ^
E SyntaxError: Missing parentheses in call to 'print'. Did you mean print("skipping", hex_code)?
________________________________________________________________ ERROR collecting suite/regress/test_arm64_ldr_registers.py _________________________________________________________________
/usr/lib/python3.8/site-packages/_pytest/runner.py:341: in from_call
result: Optional[TResult] = func()
/usr/lib/python3.8/site-packages/_pytest/runner.py:372: in <lambda>
call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/usr/lib/python3.8/site-packages/_pytest/python.py:536: in collect
self._inject_setup_module_fixture()
/usr/lib/python3.8/site-packages/_pytest/python.py:550: in _inject_setup_module_fixture
self.obj, ("setUpModule", "setup_module")
/usr/lib/python3.8/site-packages/_pytest/python.py:315: in obj
self._obj = obj = self._getobj()
/usr/lib/python3.8/site-packages/_pytest/python.py:533: in _getobj
return self._importtestmodule()
/usr/lib/python3.8/site-packages/_pytest/python.py:622: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/usr/lib/python3.8/site-packages/_pytest/pathlib.py:567: in import_path
importlib.import_module(module_name)
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
???
<frozen importlib._bootstrap>:991: in _find_and_load
???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:671: in _load_unlocked
???
/usr/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
exec(co, module.__dict__)
suite/regress/test_arm64_ldr_registers.py:5: in <module>
_python3 = sys.version_info.major == 3
E NameError: name 'sys' is not defined
================================================================================== short test summary info ==================================================================================
ERROR suite/test_corpus.py
ERROR suite/regress/test_arm64_ldr_registers.py - NameError: name 'sys' is not defined
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================================================== 2 errors in 1.45s =====================================================================================Looks like there are some python 2.x leftovers. I've added patch to remove python 2.x syntax:
--- a/suite/test_corpus.py (original)
+++ b/suite/test_corpus.py (refactored)
@@ -134,11 +134,11 @@
try:
hex_data = hex_code.strip().decode('hex')
except:
- print "skipping", hex_code
+ print("skipping", hex_code)
fout = open("fuzz/corpus/%s_%s" % (os.path.basename(fname), hex_code), 'w')
if (arch, mode) not in mc_modes:
- print "fail", arch, mode
- fout.write(unichr(mc_modes[(arch, mode)]))
+ print("fail", arch, mode)
+ fout.write(chr(mc_modes[(arch, mode)]))
fout.write(hex_data)
fout.close()
--- a/suite/regress/test_arm64_ldr_registers.py
+++ b/suite/regress/test_arm64_ldr_registers.py
@@ -2,9 +2,6 @@
from capstone import *
from capstone.arm64 import *
-_python3 = sys.version_info.major == 3
-
-
class SubRegTest(unittest.TestCase):
PATTERNS = [
@@ -21,10 +18,7 @@
self.cs.detail = True
for pattern, asm in self.PATTERNS:
- if _python3:
- l = list(self.cs.disasm(bytes.fromhex(pattern), 0))
- else:
- l = list(self.cs.disasm(bytearray.fromhex(pattern), 0))
+ l = list(self.cs.disasm(bytes.fromhex(pattern), 0))
self.assertTrue(len(l) == 1)
_, expected_reg_written, expected_reg_read = asm.split()However even with that patch pytest still fails
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/capstone-5.0.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/capstone-5.0.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.8.18, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1
collected 38 items
bindings/python/test_arm.py . [ 2%]
bindings/python/test_arm64.py . [ 5%]
bindings/python/test_basic.py F.. [ 13%]
bindings/python/test_bpf.py . [ 15%]
bindings/python/test_customized_mnem.py . [ 18%]
bindings/python/test_detail.py . [ 21%]
bindings/python/test_evm.py . [ 23%]
bindings/python/test_lite.py .. [ 28%]
bindings/python/test_m680x.py . [ 31%]
bindings/python/test_m68k.py . [ 34%]
bindings/python/test_mips.py . [ 36%]
bindings/python/test_mos65xx.py . [ 39%]
bindings/python/test_ppc.py . [ 42%]
bindings/python/test_riscv.py . [ 44%]
bindings/python/test_sh.py . [ 47%]
bindings/python/test_skipdata.py E. [ 52%]
bindings/python/test_sparc.py . [ 55%]
bindings/python/test_systemz.py . [ 57%]
bindings/python/test_tms320c64x.py . [ 60%]
bindings/python/test_tricore.py . [ 63%]
bindings/python/test_wasm.py . [ 65%]
bindings/python/test_x86.py . [ 68%]
bindings/python/test_xcore.py . [ 71%]
suite/test_corpus.py E [ 73%]
suite/test_corpus3.py E [ 76%]
suite/test_mc.py E [ 78%]
suite/regress/test_arm64_bra.py .. [ 84%]
suite/regress/test_arm64_ldr_registers.py FF [ 89%]
suite/regress/test_arm64_mov.py .. [ 94%]
suite/regress/test_arm64_pac.py .. [100%]
========================================================================================== ERRORS ===========================================================================================
_________________________________________________________________________________ ERROR at setup of testcb __________________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/bindings/python/test_skipdata.py, line 21
def testcb(buffer, size, offset, userdata):
E fixture 'buffer' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.
/home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/bindings/python/test_skipdata.py:21
________________________________________________________________________________ ERROR at setup of test_file ________________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_corpus.py, line 7
def test_file(fname):
E fixture 'fname' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.
/home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_corpus.py:7
________________________________________________________________________________ ERROR at setup of test_file ________________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_corpus3.py, line 10
def test_file(fname):
E fixture 'fname' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.
/home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_corpus3.py:10
________________________________________________________________________________ ERROR at setup of test_file ________________________________________________________________________________
file /home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_mc.py, line 67
def test_file(fname):
E fixture 'fname' not found
> available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
> use 'pytest --fixtures [testpath]' for help on them.
/home/tkloczko/rpmbuild/BUILD/capstone-5.0.1/suite/test_mc.py:67
========================================================================================= FAILURES ==========================================================================================
___________________________________________________________________________________ test_cs_disasm_quick ____________________________________________________________________________________
def test_cs_disasm_quick():
for arch, mode, code, comment, syntax in all_tests:
print('*' * 40)
print("Platform: %s" % comment)
print("Disasm:"),
print(to_hex(code))
for insn in cs_disasm_quick(arch, mode, code, 0x1000):
> print("0x%x:\t%s\t%s" % (insn.address, insn.mnemonic, insn.op_str))
bindings/python/test_basic.py:82:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <[CsError(7) raised in repr()] CsInsn object at 0x7fc3a7fa3940>, name = 'mnemonic'
def __getattr__(self, name):
if not self._cs._detail:
> raise CsError(CS_ERR_DETAIL)
E capstone.CsError: Details are unavailable (CS_ERR_DETAIL)
bindings/python/capstone/__init__.py:798: CsError
----------------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------------
****************************************
Platform: X86 16bit (Intel syntax)
Disasm:
0x8d 0x4c 0x32 0x08 0x01 0xd8 0x81 0xc6 0x34 0x12 0x00 0x00
_________________________________________________________________________________ SubRegTest.test_operands __________________________________________________________________________________
self = <test_arm64_ldr_registers.SubRegTest testMethod=test_operands>
def test_operands(self):
"""Check that the `operands` API provides correct data"""
for inst, asm, expected_regs in self.insts:
ops = inst.operands
> self.assertEqual(len(ops), 2)
E AssertionError: 1 != 2
suite/regress/test_arm64_ldr_registers.py:48: AssertionError
_________________________________________________________________________________ SubRegTest.test_registers _________________________________________________________________________________
self = <test_arm64_ldr_registers.SubRegTest testMethod=test_registers>
def test_registers(self):
"""Check that the `regs_access` API provides correct data"""
for inst, asm, expected_regs in self.insts:
# Check that the instruction writes the first register operand and reads the second
for i, decoded_regs in enumerate(map(lambda l: list(map(self.cs.reg_name, l)), inst.regs_access())):
> self.assertEqual(len(decoded_regs), 1, "%s has %d %s registers instead of 1" % (asm, len(decoded_regs), ["read", "written"][i]))
E AssertionError: 0 != 1 : ldr x1, [x2] has 0 read registers instead of 1
suite/regress/test_arm64_ldr_registers.py:40: AssertionError
================================================================================== short test summary info ==================================================================================
ERROR bindings/python/test_skipdata.py::testcb
ERROR suite/test_corpus.py::test_file
ERROR suite/test_corpus3.py::test_file
ERROR suite/test_mc.py::test_file
FAILED bindings/python/test_basic.py::test_cs_disasm_quick - capstone.CsError: Details are unavailable (CS_ERR_DETAIL)
FAILED suite/regress/test_arm64_ldr_registers.py::SubRegTest::test_operands - AssertionError: 1 != 2
FAILED suite/regress/test_arm64_ldr_registers.py::SubRegTest::test_registers - AssertionError: 0 != 1 : ldr x1, [x2] has 0 read registers instead of 1
========================================================================== 3 failed, 31 passed, 4 errors in 1.41s ===========================================================================Cannot find what provides those missing pytest fixtures 🤔
Please let me know if you want above patch es as PR.
XVilka
commented
1 year ago @kloczek patch looks good, IMHO. Please send it as a PR to the next branch. Then it can be cherry-picked to the potential 5.0.2 release: https://github.com/capstone-engine/capstone/issues/2081
Rot127
commented
2 months ago Should be fixed by https://github.com/capstone-engine/capstone/pull/2456 and several previous PRs removing Python bindings and reworking the wheel build (https://github.com/capstone-engine/capstone/pull/2400).
I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
python3 -sBm build -w --no-isolationbuildwith--no-isolationI'm using during all processes only locally installed modulescut off from access to the public network(pytest is executed with-m "not network")Here is pytest output:
Here is list of installed modules in build env