Open felixonmars opened 6 years ago
@felixonmars if this is still an issue, can you please check against the latest 1.3.8 release?
Yes, it's still present in 1.3.7/1.3.8 releases.
Full logs w/versions: https://paste.xinu.at/HRD/
@felixonmars my first guess would be something is borked with the python-lxml
dependency. If I resort to the manylinux
wheel, e.g.
check() {
cd python-xmlsec-$pkgver
pip install --force-reinstall lxml
python setup.py pytest
}
in the PKGBUILD
, the tests pass. I will try looking further into it.
It seems the lxml here has some issues with runpy, that it fails the second time running the same example:
doc/source/examples {0}$ python
Python 3.8.3 (default, May 17 2020, 18:15:42)
[GCC 10.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import runpy
>>> runpy.run_path('encrypt.py')
b'<xenc:CipherValue xmlns:xenc="http://www.w3.org/2001/04/xmlenc#" xmlns:dsig="http://www.w3.org/2000/09/xmldsig#">kMAH68us7GEVk42MEU8DZ/FunGVNfZD0wj+ObupHtw2oIhiGbooNnjmDORGlhRtl\nGAj/DzVP3UP7INPgP9STkyqPRazgI3Q5T8DlLQwZhXqQpBG0xQ/sezBhXykCBa6j\np97qXLKT/xn5XslxH3iEN74WwIjmUGkDS3e3O1rM3Ab3bWQKPgho1Hwxaeq+8UJK\nIVsEq4gIHsxn70o64MWE6s5X3NT+zPx1BoVT2MujRwzpywLjX6K9muNn5IDpETwt\nZ9AqnN9babR9FU1jWNBOPr/7auOQ1gvDaqVGsKcnqUINr3W9RKTPNtTsle6tBovg\nET7FbEBj19aXeueDWFNOxQ==</xenc:CipherValue>\n'
{'__name__': '<run_path>', '__doc__': None, '__package__': '', '__loader__': None, '__spec__': None, '__file__': 'encrypt.py', '__cached__': None, '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '__package__': '', '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': ModuleSpec(name='builtins', loader=<class '_frozen_importlib.BuiltinImporter'>), '__build_class__': <built-in function __build_class__>, '__import__': <built-in function __import__>, 'abs': <built-in function abs>, 'all': <built-in function all>, 'any': <built-in function any>, 'ascii': <built-in function ascii>, 'bin': <built-in function bin>, 'breakpoint': <built-in function breakpoint>, 'callable': <built-in function callable>, 'chr': <built-in function chr>, 'compile': <built-in function compile>, 'delattr': <built-in function delattr>, 'dir': <built-in function dir>, 'divmod': <built-in function divmod>, 'eval': <built-in function eval>, 'exec': <built-in function exec>, 'format': <built-in function format>, 'getattr': <built-in function getattr>, 'globals': <built-in function globals>, 'hasattr': <built-in function hasattr>, 'hash': <built-in function hash>, 'hex': <built-in function hex>, 'id': <built-in function id>, 'input': <built-in function input>, 'isinstance': <built-in function isinstance>, 'issubclass': <built-in function issubclass>, 'iter': <built-in function iter>, 'len': <built-in function len>, 'locals': <built-in function locals>, 'max': <built-in function max>, 'min': <built-in function min>, 'next': <built-in function next>, 'oct': <built-in function oct>, 'ord': <built-in function ord>, 'pow': <built-in function pow>, 'print': <built-in function print>, 'repr': <built-in function repr>, 'round': <built-in function round>, 'setattr': <built-in function setattr>, 'sorted': <built-in function sorted>, 'sum': <built-in function sum>, 'vars': <built-in function vars>, 'None': None, 'Ellipsis': Ellipsis, 'NotImplemented': NotImplemented, 'False': False, 'True': True, 'bool': <class 'bool'>, 'memoryview': <class 'memoryview'>, 'bytearray': <class 'bytearray'>, 'bytes': <class 'bytes'>, 'classmethod': <class 'classmethod'>, 'complex': <class 'complex'>, 'dict': <class 'dict'>, 'enumerate': <class 'enumerate'>, 'filter': <class 'filter'>, 'float': <class 'float'>, 'frozenset': <class 'frozenset'>, 'property': <class 'property'>, 'int': <class 'int'>, 'list': <class 'list'>, 'map': <class 'map'>, 'object': <class 'object'>, 'range': <class 'range'>, 'reversed': <class 'reversed'>, 'set': <class 'set'>, 'slice': <class 'slice'>, 'staticmethod': <class 'staticmethod'>, 'str': <class 'str'>, 'super': <class 'super'>, 'tuple': <class 'tuple'>, 'type': <class 'type'>, 'zip': <class 'zip'>, '__debug__': True, 'BaseException': <class 'BaseException'>, 'Exception': <class 'Exception'>, 'TypeError': <class 'TypeError'>, 'StopAsyncIteration': <class 'StopAsyncIteration'>, 'StopIteration': <class 'StopIteration'>, 'GeneratorExit': <class 'GeneratorExit'>, 'SystemExit': <class 'SystemExit'>, 'KeyboardInterrupt': <class 'KeyboardInterrupt'>, 'ImportError': <class 'ImportError'>, 'ModuleNotFoundError': <class 'ModuleNotFoundError'>, 'OSError': <class 'OSError'>, 'EnvironmentError': <class 'OSError'>, 'IOError': <class 'OSError'>, 'EOFError': <class 'EOFError'>, 'RuntimeError': <class 'RuntimeError'>, 'RecursionError': <class 'RecursionError'>, 'NotImplementedError': <class 'NotImplementedError'>, 'NameError': <class 'NameError'>, 'UnboundLocalError': <class 'UnboundLocalError'>, 'AttributeError': <class 'AttributeError'>, 'SyntaxError': <class 'SyntaxError'>, 'IndentationError': <class 'IndentationError'>, 'TabError': <class 'TabError'>, 'LookupError': <class 'LookupError'>, 'IndexError': <class 'IndexError'>, 'KeyError': <class 'KeyError'>, 'ValueError': <class 'ValueError'>, 'UnicodeError': <class 'UnicodeError'>, 'UnicodeEncodeError': <class 'UnicodeEncodeError'>, 'UnicodeDecodeError': <class 'UnicodeDecodeError'>, 'UnicodeTranslateError': <class 'UnicodeTranslateError'>, 'AssertionError': <class 'AssertionError'>, 'ArithmeticError': <class 'ArithmeticError'>, 'FloatingPointError': <class 'FloatingPointError'>, 'OverflowError': <class 'OverflowError'>, 'ZeroDivisionError': <class 'ZeroDivisionError'>, 'SystemError': <class 'SystemError'>, 'ReferenceError': <class 'ReferenceError'>, 'MemoryError': <class 'MemoryError'>, 'BufferError': <class 'BufferError'>, 'Warning': <class 'Warning'>, 'UserWarning': <class 'UserWarning'>, 'DeprecationWarning': <class 'DeprecationWarning'>, 'PendingDeprecationWarning': <class 'PendingDeprecationWarning'>, 'SyntaxWarning': <class 'SyntaxWarning'>, 'RuntimeWarning': <class 'RuntimeWarning'>, 'FutureWarning': <class 'FutureWarning'>, 'ImportWarning': <class 'ImportWarning'>, 'UnicodeWarning': <class 'UnicodeWarning'>, 'BytesWarning': <class 'BytesWarning'>, 'ResourceWarning': <class 'ResourceWarning'>, 'ConnectionError': <class 'ConnectionError'>, 'BlockingIOError': <class 'BlockingIOError'>, 'BrokenPipeError': <class 'BrokenPipeError'>, 'ChildProcessError': <class 'ChildProcessError'>, 'ConnectionAbortedError': <class 'ConnectionAbortedError'>, 'ConnectionRefusedError': <class 'ConnectionRefusedError'>, 'ConnectionResetError': <class 'ConnectionResetError'>, 'FileExistsError': <class 'FileExistsError'>, 'FileNotFoundError': <class 'FileNotFoundError'>, 'IsADirectoryError': <class 'IsADirectoryError'>, 'NotADirectoryError': <class 'NotADirectoryError'>, 'InterruptedError': <class 'InterruptedError'>, 'PermissionError': <class 'PermissionError'>, 'ProcessLookupError': <class 'ProcessLookupError'>, 'TimeoutError': <class 'TimeoutError'>, 'open': <built-in function open>, 'quit': Use quit() or Ctrl-D (i.e. EOF) to exit, 'exit': Use exit() or Ctrl-D (i.e. EOF) to exit, 'copyright': Copyright (c) 2001-2020 Python Software Foundation.
All Rights Reserved.
Copyright (c) 2000 BeOpen.com.
All Rights Reserved.
Copyright (c) 1995-2001 Corporation for National Research Initiatives.
All Rights Reserved.
Copyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam.
All Rights Reserved., 'credits': Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands
for supporting Python development. See www.python.org for more information., 'license': Type license() to see the full license text, 'help': Type help() for interactive help, or help(object) for help about object., '_': None}, 'etree': <module 'lxml.etree' from '/usr/lib/python3.8/site-packages/lxml/etree.cpython-38-x86_64-linux-gnu.so'>, 'xmlsec': <module 'xmlsec' from '/usr/lib/python3.8/site-packages/xmlsec.cpython-38-x86_64-linux-gnu.so'>, 'manager': <xmlsec.KeysManager object at 0x7ff0fbae5b30>, 'key': <xmlsec.Key object at 0x7ff0f94395b0>, 'template': <Element Envelope at 0x7ff0f943ae40>, 'enc_data': <Element {http://www.w3.org/2001/04/xmlenc#}EncryptedData at 0x7ff0f9433440>, 'key_info': <Element {http://www.w3.org/2000/09/xmldsig#}KeyInfo at 0x7ff0f943ae80>, 'enc_key': <Element {http://www.w3.org/2001/04/xmlenc#}EncryptedKey at 0x7ff0f943aec0>, 'data': <Element Data at 0x7ff0f943af80>, 'enc_ctx': <xmlsec.EncryptionContext object at 0x7ff0f94395f0>, 'enc_method': <Element {http://www.w3.org/2001/04/xmlenc#}EncryptionMethod at 0x7ff0f943e040>, 'cipher_value': <Element {http://www.w3.org/2001/04/xmlenc#}CipherValue at 0x7ff0f943afc0>}
>>> runpy.run_path('encrypt.py')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,
File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "encrypt.py", line 8, in <module>
template = etree.parse('enc1-doc.xml').getroot()
File "src/lxml/etree.pyx", line 3521, in lxml.etree.parse
File "src/lxml/parser.pxi", line 1839, in lxml.etree._parseDocument
File "src/lxml/parser.pxi", line 1865, in lxml.etree._parseDocumentFromURL
File "src/lxml/parser.pxi", line 1769, in lxml.etree._parseDocFromFile
File "src/lxml/parser.pxi", line 1163, in lxml.etree._BaseParser._parseDocFromFile
File "src/lxml/parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc
File "src/lxml/parser.pxi", line 711, in lxml.etree._handleParseResult
File "src/lxml/parser.pxi", line 651, in lxml.etree._raiseParseError
File "b'enc1-doc.xml'", line 0
lxml.etree.XMLSyntaxError
Using subprocess.Popen
here instead of runpy makes the tests pass.
Here is my findings. Put together a xml file and run this:
from lxml import etree
import ctypes
xmlsec = ctypes.CDLL('/usr/lib/libxmlsec1.so')
xmlsec.xmlSecInit()
etree.parse('enc1-doc.xml')
print('ok')
etree.parse('enc1-doc.xml')
It reproduces. The second parse
call only triggers a stat
call but no file open
and read
calls.
lxml, xmlsec and libxml2 are all coming from my Arch Linux.
It looks like the lxml wheel from PyPI (installed by pip) has its own libxml2, but xmlsec dynamically links to the system one, so xmlsec changes one libxml2 while lxml uses another, avoiding this issue. (But I don't know if it's OK to have two libxml2 libraries like this.)
I don't know how to use the C libraries so no reproduction tries without Python.
@lilydjwg your comment was of an immense help and it brought me on the right track, thank you!
I found out that this issue is caused by lxml
reusing the XML parser object: e.g.
from lxml import etree
import xmlsec
etree.parse('doc.xml', parser=etree.XMLParser())
etree.parse('doc.xml', parser=etree.XMLParser())
works, but
from lxml import etree
import xmlsec
p = etree.XMLParser()
etree.parse('doc.xml', parser=p)
etree.parse('doc.xml', parser=p)
raises. The same issue arises when calling etree.parse('doc.xml')
- the default parser stored globally is reused.
The workaround to this issue is thus simple - resetting the default parser between the load_xml
invocations should be enough to fix the failing tests. However, this doesn't fix the underlying issue #96 yet. I'll poke around some more and open an issue against lxml
launchpad if won't be able to resolve it myself.
In the meantime, a simple patch like e.g.
diff --git a/tests/base.py b/tests/base.py
index b05de1d..5ec356f 100644
--- a/tests/base.py
+++ b/tests/base.py
@@ -94,6 +94,7 @@ class TestMemoryLeaks(unittest.TestCase):
def load_xml(self, name, xpath=None):
"""returns xml.etree"""
+ etree.set_default_parser(parser=etree.XMLParser())
root = etree.parse(self.path(name)).getroot()
if xpath is None:
return root
diff --git a/tests/test_doc_examples.py b/tests/test_doc_examples.py
index 2fc490f..53d2377 100644
--- a/tests/test_doc_examples.py
+++ b/tests/test_doc_examples.py
@@ -42,3 +42,5 @@ def test_doc_example(example):
"""
with cd(example.parent):
runpy.run_path(str(example))
+ from lxml import etree
+ etree.set_default_parser(parser=etree.XMLParser())
will fix the test run, but I'm sure this is not what you're after in the end.
For reference: https://bugs.launchpad.net/lxml/+bug/1880251
@hoefling, Thanks for the above solution. All tests are passing after upgrading lxml to 4.5.2 in amd64 platform, but in ARM64/AArch64 platform even after upgrading lxml, I am getting same lxml syntax error and all tests are passing after incorporating above mentioned changes.
It will be helpful if the above mentioned changes are added or suggest if I am missing any installation of any other dependencies.
@odidev if you're getting the same error with lxml==4.5.2
, then the issue is not completely resolved on lxml side (I also think there's more to do as the example I posted in this comment fails with 4.5.2). I'd open a new issue with a reproduction example at lxml's launchpad.
Follow-up issue: https://bugs.launchpad.net/lxml/+bug/1887848
There are still three failures here with 1.3.13 and lxml 4.9.1:
============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0
rootdir: /build/python-xmlsec/src/python-xmlsec-1.3.13
plugins: hypothesis-6.17.0
collected 297 items / 1 skipped
tests/test_xmlsec.py . [ 0%]
tests/test_constants.py ................................................ [ 16%]
........................................................................ [ 40%]
.......... [ 44%]
tests/test_doc_examples.py .FF.F. [ 46%]
tests/test_ds.py ........................................ [ 59%]
tests/test_enc.py ........................ [ 67%]
tests/test_keys.py ......................................... [ 81%]
tests/test_main.py ............. [ 85%]
tests/test_templates.py .................................. [ 97%]
tests/test_tree.py ........ [100%]
=================================== FAILURES ===================================
_________________________ test_doc_example[encrypt.py] _________________________
example = PosixPath('/build/python-xmlsec/src/python-xmlsec-1.3.13/doc/source/examples/encrypt.py')
@pytest.mark.parametrize('example', examples, ids=lambda p: p.name)
def test_doc_example(example):
"""
Verify example scripts included in the docs are up to date.
Execute each script in :file:`docs/source/examples`,
not raising any errors is good enough.
"""
with cd(example.parent):
> runpy.run_path(str(example))
tests/test_doc_examples.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.10/runpy.py:289: in run_path
return _run_module_code(code, init_globals, run_name,
/usr/lib/python3.10/runpy.py:96: in _run_module_code
_run_code(code, mod_globals, init_globals,
/usr/lib/python3.10/runpy.py:86: in _run_code
exec(code, run_globals)
doc/source/examples/encrypt.py:8: in <module>
template = etree.parse('enc1-doc.xml').getroot()
src/lxml/etree.pyx:3538: in lxml.etree.parse
???
src/lxml/parser.pxi:1876: in lxml.etree._parseDocument
???
src/lxml/parser.pxi:1902: in lxml.etree._parseDocumentFromURL
???
src/lxml/parser.pxi:1805: in lxml.etree._parseDocFromFile
???
src/lxml/parser.pxi:1177: in lxml.etree._BaseParser._parseDocFromFile
???
src/lxml/parser.pxi:615: in lxml.etree._ParserContext._handleParseResultDoc
???
src/lxml/parser.pxi:725: in lxml.etree._handleParseResult
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E File "b'enc1-doc.xml'", line 0
E lxml.etree.XMLSyntaxError: <no detail available>
src/lxml/parser.pxi:665: XMLSyntaxError
----------------------------- Captured stderr call -----------------------------
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=58:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='enc1-doc.xml'; xml error: 0: NULL
__________________________ test_doc_example[sign.py] ___________________________
example = PosixPath('/build/python-xmlsec/src/python-xmlsec-1.3.13/doc/source/examples/sign.py')
@pytest.mark.parametrize('example', examples, ids=lambda p: p.name)
def test_doc_example(example):
"""
Verify example scripts included in the docs are up to date.
Execute each script in :file:`docs/source/examples`,
not raising any errors is good enough.
"""
with cd(example.parent):
> runpy.run_path(str(example))
tests/test_doc_examples.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.10/runpy.py:289: in run_path
return _run_module_code(code, init_globals, run_name,
/usr/lib/python3.10/runpy.py:96: in _run_module_code
_run_code(code, mod_globals, init_globals,
/usr/lib/python3.10/runpy.py:86: in _run_code
exec(code, run_globals)
doc/source/examples/sign.py:5: in <module>
template = etree.parse('sign1-tmpl.xml').getroot()
src/lxml/etree.pyx:3538: in lxml.etree.parse
???
src/lxml/parser.pxi:1876: in lxml.etree._parseDocument
???
src/lxml/parser.pxi:1902: in lxml.etree._parseDocumentFromURL
???
src/lxml/parser.pxi:1805: in lxml.etree._parseDocFromFile
???
src/lxml/parser.pxi:1177: in lxml.etree._BaseParser._parseDocFromFile
???
src/lxml/parser.pxi:615: in lxml.etree._ParserContext._handleParseResultDoc
???
src/lxml/parser.pxi:725: in lxml.etree._handleParseResult
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E File "b'sign1-tmpl.xml'", line 0
E lxml.etree.XMLSyntaxError: <no detail available>
src/lxml/parser.pxi:665: XMLSyntaxError
----------------------------- Captured stderr call -----------------------------
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=58:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='sign1-tmpl.xml'; xml error: 0: NULL
_________________________ test_doc_example[verify.py] __________________________
example = PosixPath('/build/python-xmlsec/src/python-xmlsec-1.3.13/doc/source/examples/verify.py')
@pytest.mark.parametrize('example', examples, ids=lambda p: p.name)
def test_doc_example(example):
"""
Verify example scripts included in the docs are up to date.
Execute each script in :file:`docs/source/examples`,
not raising any errors is good enough.
"""
with cd(example.parent):
> runpy.run_path(str(example))
tests/test_doc_examples.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3.10/runpy.py:289: in run_path
return _run_module_code(code, init_globals, run_name,
/usr/lib/python3.10/runpy.py:96: in _run_module_code
_run_code(code, mod_globals, init_globals,
/usr/lib/python3.10/runpy.py:86: in _run_code
exec(code, run_globals)
doc/source/examples/verify.py:5: in <module>
template = etree.parse('sign1-res.xml').getroot()
src/lxml/etree.pyx:3538: in lxml.etree.parse
???
src/lxml/parser.pxi:1876: in lxml.etree._parseDocument
???
src/lxml/parser.pxi:1902: in lxml.etree._parseDocumentFromURL
???
src/lxml/parser.pxi:1805: in lxml.etree._parseDocFromFile
???
src/lxml/parser.pxi:1177: in lxml.etree._BaseParser._parseDocFromFile
???
src/lxml/parser.pxi:615: in lxml.etree._ParserContext._handleParseResultDoc
???
src/lxml/parser.pxi:725: in lxml.etree._handleParseResult
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E File "b'sign1-res.xml'", line 0
E lxml.etree.XMLSyntaxError: <no detail available>
src/lxml/parser.pxi:665: XMLSyntaxError
----------------------------- Captured stderr call -----------------------------
func=xmlSecNoXxeExternalEntityLoader:file=xmlsec.c:line=58:obj=unknown:subj=xmlSecNoXxeExternalEntityLoader:error=5:libxml2 library function failed:illegal external entity='sign1-res.xml'; xml error: 0: NULL
=========================== short test summary info ============================
FAILED tests/test_doc_examples.py::test_doc_example[encrypt.py] - File "b'e...
FAILED tests/test_doc_examples.py::test_doc_example[sign.py] - File "b'sign...
FAILED tests/test_doc_examples.py::test_doc_example[verify.py] - File "b'si...
=================== 3 failed, 294 passed, 1 skipped in 1.30s ===================
Looks like there's no activity on the follow-up lxml issue.
I am trying to package python-xmlsec for Arch, but unfortunately most tests are failing here. Versions:
Python 3.6.6/3.7.0 libxml2 2.9.8 xmlsec 1.2.26
Cflags:
-DXMLSEC_CRYPTO_DYNAMIC_LOADING=1 -D__XMLSEC_FUNCTION__=__func__ -DXMLSEC_NO_SIZE_T -DXMLSEC_NO_GOST=1 -DXMLSEC_NO_GOST2012=1 -DXMLSEC_DL_LIBLTDL=1 -I/usr/include/xmlsec1 -I/usr/include/libxml2
Example test failure (others are very similar):