search

potassco / clingo

šŸ¤” A grounder and solver for logic programs.
https://potassco.org/clingo
MIT License
601 stars 79 forks source link

Some python tests break: Failed to import test module: test_application #390

Closed yurivict closed 1 year ago

yurivict commented 2 years ago 
===>  Testing for py39-clingo-5.6.0
===>   py39-clingo-5.6.0 depends on file: /usr/local/bin/python3.9 - found
cd /disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo/clingo/tests && /usr/bin/env XDG_DATA_HOME=/disk-samsung/freebsd-ports/math/py-clingo/work-py39  XDG_CONFIG_HOME=/disk-samsung/freebsd-ports/math/py-clingo/work-py39  XDG_CACHE_HOME=/disk-samsung/freebsd-ports/math/py-clingo/work-py39/.cache  HOME=/disk-samsung/freebsd-ports/math/py-clingo/work-py39 PATH=/usr/local/libexec/ccache:/disk-samsung/freebsd-ports/math/py-clingo/work-py39/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin PKG_CONFIG_LIBDIR=/disk-samsung/freebsd-ports/math/py-clingo/work-py39/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig NO_PIE=yes MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES LDSHARED="cc -shared" PYTHONDONTWRITEBYTECODE= PYTHONOPTIMIZE= PREFIX=/usr/local  LOCALBASE=/usr/local  CC="cc" CFLAGS="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -fno-strict-aliasing "  CPP="cpp" CPPFLAGS="-fno-omit-frame-pointer"  LDFLAGS=" -fstack-protector-strong " LIBS=""  CXX="c++" CXXFLAGS="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -fno-strict-aliasing -fno-omit-frame-pointer  "  MANPREFIX="/usr/local" CCACHE_DIR="/tmp/.ccache" BSD_INSTALL_PROGRAM="install  -s -m 555"  BSD_INSTALL_LIB="install  -s -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" PYTHONPATH=/disk-samsung/freebsd-ports/math/py-clingo/work-py39/stage/usr/local/lib/python3.9/site-packages /usr/local/bin/python3.9 -m unittest  -v
test_application (unittest.loader._FailedTest) ... ERROR
test_aspif (unittest.loader._FailedTest) ... ERROR
test_ast (unittest.loader._FailedTest) ... ERROR
test_symbolic_atom (test_atoms.TestAtoms)
Test symbolic atom. ... ok
test_symbolic_atoms (test_atoms.TestAtoms)
Test symbolic atoms. ... ok
test_theory_atom (test_atoms.TestAtoms)
Test theory atom. ... ok
test_theory_element (test_atoms.TestAtoms)
Test theory element. ... ok
test_theory_term (test_atoms.TestAtoms)
Test theory term. ... ok
test_adding_theory (test_backend.TestBackend)
Test theory related functions in backend. ... ok
test_backend (test_backend.TestBackend)
Test backend via observer. ... ok
test_theory (test_backend.TestBackend)
Test observer via grounding. ... ok
test_theory_with_guard (test_backend.TestBackend)
Test observer via grounding. ... ok
test_config (test_conf.TestConfig)
Test configuration. ... ok
test_simple_stats (test_conf.TestConfig)
Test simple statistics. ... ok
test_user_stats (test_conf.TestConfig)
Test user statistics. ... ok
test_default (test_control.TestControl)
Test grounding with context and parameters. ... ok
test_error_handling (test_control.TestControl)
Test basic error handling during solving. ... __cxa_thread_call_dtors: dtr 0x83dbafa40 from unloaded dso, skipping
__cxa_thread_call_dtors: dtr 0x83db5a880 from unloaded dso, skipping
ok
test_ground (test_control.TestControl)
Test grounding with context and parameters. ... ok
test_ground_error (test_control.TestControl)
Test grounding with context and parameters. ... ok
test_lower (test_control.TestControl)
Test lower bounds reported during optimization. ... ok
test_propagator (unittest.loader._FailedTest) ... ERROR
test_solving (unittest.loader._FailedTest) ... ERROR
test_cmp (test_symbol.TestSymbol)
Test hashing and comparisons. ... ok
test_function (test_symbol.TestSymbol)
Test functions. ... ok
test_infsup (test_symbol.TestSymbol)
Test infimum and supremum. ... ok
test_match (test_symbol.TestSymbol)
Test symbol matching. ... ok
test_number (test_symbol.TestSymbol)
Test numbers. ... ok
test_parse (test_symbol.TestSymbol)
Test term parsing. ... ok
test_repr (test_symbol.TestSymbol)
Test representation of symbols. ... ok
test_str (test_symbol.TestSymbol)
Test string representation of symbols. ... ok
test_string (test_symbol.TestSymbol)
Test strings. ... ok

======================================================================
ERROR: test_application (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_application
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo/clingo/tests/test_application.py", line 12, in <module>
    from .util import _MCB
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: test_aspif (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_aspif
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo/clingo/tests/test_aspif.py", line 7, in <module>
    from ..control import Control
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: test_ast (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_ast
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo/clingo/tests/test_ast.py", line 10, in <module>
    from .. import ast
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: test_propagator (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_propagator
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo/clingo/tests/test_propagator.py", line 10, in <module>
    from .util import _MCB, _check_sat, _p
ImportError: attempted relative import with no known parent package

======================================================================
ERROR: test_solving (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_solving
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/local/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo/clingo/tests/test_solving.py", line 8, in <module>
    from . util import _MCB, _check_sat, _p
ImportError: attempted relative import with no known parent package

----------------------------------------------------------------------
Ran 31 tests in 0.038s

FAILED (errors=5)
__cxa_thread_call_dtors: dtr 0x83dbafa40 from unloaded dso, skipping
__cxa_thread_call_dtors: dtr 0x83db5a880 from unloaded dso, skipping
*** Error code 1

Version: 5.6.0 Python-3.9 clang-14 FreeBSD 13.1

rkaminsk commented 1 year ago

For some reason the relative imports fail. They could simply be replaced by absolute ones and the tests should work. It is still strange that this fails because the test module is a submodule of clingo.

What is more worrisome are the messages of form:

__cxa_thread_call_dtors: dtr <address> from unloaded dso, skipping

This means some memory might be leaked. I am not sure, I can do anything about it.

I found this discussion that seems related: https://www.spinics.net/lists/ceph-devel/msg42456.html.

I also found the following:

This is the backing feature to implement C++11 thread storage duration
specified by the thread_local keyword.  A destructor for given
thread-local object is registered to be executed at the thread
termination time using __cxa_thread_atexit().  Libc calls the
__cxa_thread_calls_dtors() during exit(3), before finalizers and
atexit functions, and libthr calls the function at the thread
termination time, after the stack unwinding and thread-specific key
destruction.

There are several uncertainties in the API which lacks a formal
specification.  Among them:
- is it allowed to register destructors during destructing;
    we allow, but limiting the nesting level.  If too many iterations
    detected, a diagnostic is issued to stderr and thread forcibly
    terminates for now.
- how to handle destructors which belong to an unloading dso;
    for now, we ignore destructor calls for such entries, and
    issue a diagnostic.  Linux does prevent dso unload until all
    threads with destructors from the dso terminated.
It is supposed that the diagnostics allow to detect real-world
applications relying on the above details and possibly adjust
our implementation.  Right now the choices were to provide the slim
API (but that rarely stands the practice test).

Tests are added to check generic functionality and to specify some of
the above implementation choices.

So this is something that should be addressed in BSDs libc implementation.

rkaminsk commented 1 year ago

I just saw that you are calling python -m unittest directly from the tests directory. It is not intended to be used like this. It should be called from the libpyclingo directory. Otherwise, imports won't work.

Using ctest, the tests are called like this:

Test command: /usr/bin/python3.8 "-m" "unittest"
Environment variables: 
 PYTHONPATH=<MODPATH>
Test timeout computed to be: 10000000
................................................................
----------------------------------------------------------------------
Ran 64 tests in 1.252s

OK

Note that the PYTHONPATH is set to the location where the _clingo.*.so python module is located.

rkaminsk commented 1 year ago

Adding the invalid label. I think everything is fine with clingo.

yurivict commented 1 year ago

It should be called from the libpyclingo directory.

It fails to find the 'clingo' module because libpyclingo has the subdirectory clingo which python picks up:

===>  Testing for py39-clingo-5.6.0
===>   py39-clingo-5.6.0 depends on file: /usr/local/bin/python3.9 - found
cd /disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo && /usr/bin/env XDG_DATA_HOME=/disk-samsung/freebsd-ports/math/py-clingo/work-py39  XDG_CONFIG_HOME=/disk-samsung/freebsd-ports/math/py-clingo/work-py39  XDG_CACHE_HOME=/disk-samsung/freebsd-ports/math/py-clingo/work-py39/.cache  HOME=/disk-samsung/freebsd-ports/math/py-clingo/work-py39 PATH=/usr/local/libexec/ccache:/disk-samsung/freebsd-ports/math/py-clingo/work-py39/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin PKG_CONFIG_LIBDIR=/disk-samsung/freebsd-ports/math/py-clingo/work-py39/.pkgconfig:/usr/local/libdata/pkgconfig:/usr/local/share/pkgconfig:/usr/libdata/pkgconfig NO_PIE=yes MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES LDSHARED="cc -shared" PYTHONDONTWRITEBYTECODE= PYTHONOPTIMIZE= PREFIX=/usr/local  LOCALBASE=/usr/local  CC="cc" CFLAGS="-O2 -pipe -fno-omit-frame-pointer  -fstack-protector-strong -fno-strict-aliasing "  CPP="cpp" CPPFLAGS="-fno-omit-frame-pointer"  LDFLAGS=" -fstack-protector-strong " LIBS=""  CXX="c++" CXXFLAGS="-O2 -pipe -fno-omit-frame-pointer -fstack-protector-strong -fno-strict-aliasing -fno-omit-frame-pointer  "  MANPREFIX="/usr/local" CCACHE_DIR="/tmp/.ccache" BSD_INSTALL_PROGRAM="install  -s -m 555"  BSD_INSTALL_LIB="install  -s -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" PYTHONPATH=/disk-samsung/freebsd-ports/math/py-clingo/work-py39/stage/usr/local/lib/python3.9/site-packages /usr/local/bin/python3.9 -m unittest  -v
clingo (unittest.loader._FailedTest) ... ERROR

======================================================================
ERROR: clingo (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: clingo
Traceback (most recent call last):
  File "/disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo/clingo/_internal.py", line 18, in <module>
    from _clingo import ffi as _ffi, lib as _lib # type: ignore # pylint: disable=no-name-in-module
ModuleNotFoundError: No module named '_clingo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/unittest/loader.py", line 470, in _find_test_path
    package = self._get_module_from_name(name)
  File "/usr/local/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo/clingo/__init__.py", line 100, in <module>
    from .core import *
  File "/disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo/clingo/core.py", line 8, in <module>
    from ._internal import _cb_error_panic, _ffi, _lib, _to_str
  File "/disk-samsung/freebsd-ports/math/py-clingo/work-py39/clingo-5.6.0/libpyclingo/clingo/_internal.py", line 20, in <module>
    from ._clingo import ffi as _ffi, lib as _lib # type: ignore # pylint: disable=no-name-in-module
ModuleNotFoundError: No module named 'clingo._clingo'

----------------------------------------------------------------------
Ran 1 test in 0.000s
rkaminsk commented 1 year ago

It fails to find the 'clingo' module because libpyclingo has the subdirectory clingo which python picks up:

Like I said, you have to point the PYTHONPATH to the C module. The test target takes care of this. You can just run

cmake --build <BUILDDIR> --target test
yurivict commented 1 year ago

Ok, thank you for your help. It works fine now.