Open dyershov opened 1 year ago
Yes, looks like the same bug.
My problem remains reproducing the error. Do you have more details, e.g. platform, compiler version, etc.?
A few updates. First, the bug persists even after installing cppyy
from sources, i.e., rebuilding the backend from scratch.
Second, I'm using Arch Linux with both gcc and clang installed:
Anything else?
@wlav, I found a related issue https://github.com/root-project/root/issues/12294#issuecomment-1435904083
BTW, is it possible to point cling to a custom toolchain, system and STL headers, and compiled objects?
The fix for root#12294 is already included in the release, so that can't be it.
For the toolchain, in principle yes: Cling picks up the compiler paths from the system compiler, so either change PATH
to point to the new chain or use -I
arguments in the EXTRA_CLING_ARGS
variable.
@wlav not sure if it helps, but I was testing a different variant from the example @dyershov provided.
import cppyy
cppyy.cppdef("""
template<typename T>
T call_int_int(T (*f)(T,T), T i1, T i2) {
return f(i1, i2);
}
int sumc(int a, int b)
{
return a+b;
}
""")
from cppyy.gbl import call_int_int, sumc
def add(a: int, b: int)->int:
return a + b
print(call_int_int[int](sumc, 3, 7)) # prints 10 as expected
print(call_int_int[int](add, 3, 7)) # errors out, trace below
Line 19 seems weird. Tmpl_name="<" ?
#19 AddTypeName (tmpl_name="<", tn=<optimized out>, arg=<optimized out>, pref=pref
entry=CPyCppyy::Utility::kNone, pcnt=pcnt
entry=0x0) at src/Utility.cxx:509
Currently using:
cppyy==3.0.0
cppyy-backend==1.14.11
cppyy-cling==6.28.0
CPyCppyy==1.12.13
gcc 12.2.0
Ubuntu 23.04
Nice, a fully reproducible crash. :) Thanks! Fixed in repo. (Problem wasn't that traceback line 19, but the code assuming a string type while it had class int
.)
Now I'm getting a different error. Shouldn't the Python int have been converted to C++ int before getting to the following prototype? Not sure when do these conversions happen, so I'm guessing.
func.prototype.file:1:1: error: expected a type
<class 'int'> (*)(<class 'int'>, <class 'int'>),int,int
^
...
TClass::GetClass: Header Parsing - The representation of int(*)(int,int) was not found in the type system. A lookup in the interpreter is about to be tried: this can cause parsing. This can be avoided selecting int(*)(int,int) in the linkdef/selection file.
cling JIT session error: Failed to materialize symbols: { (main, { _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZN16__cppyy_internal10fptr_wrap3Eii }) }
Traceback (most recent call last):
File "/home/gabriel/./Desktop/cpycppyy/test.py", line 21, in <module>
print(call_int_int[int](add, 3, 7)) # errors out, trace below
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Could not find "call_int_int<int>" (set cppyy.set_debug() for C++ errors):
int ::call_int_int(int(*)(int,int) f, int i1, int i2) =>
TypeError: could not convert argument 1
none of the 2 overloaded methods succeeded. Full details:
int ::call_int_int(int(*)(int,int) f, int i1, int i2) =>
TypeError: could not convert argument 1
int ::call_int_int(int(*)(int,int) f, int i1, int i2) =>
TypeError: could not convert argument 1
Interesting, as I didn't (and don't) get that result. However, I can see how that could happen. Presumed fix is in repo.
Prototype error went away, but the TypeErrors persist.
Right, because of cling JIT session error: Failed to materialize symbols
, which I can't reproduce. :(
Hmm. Testing a bit more, found something that may be of interest.
Running the same example:
10
cling JIT session error: Failed to materialize symbols: { (main, { _ZN16__cppyy_internal10fptr_wrap1Eii }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZN16__cppyy_internal10fptr_wrap2Eii }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZN16__cppyy_internal10fptr_wrap3Eii }) }
Traceback (most recent call last):
File "/home/gabriel/./Desktop/cpycppyy/test.py", line 66, in <module>
print(call_int_int[int](add, 3, 7)) # errors out, trace below
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
Tried getting the generated code for that function wrapper that failed to be materialized.
Then defined that manually.
It still fails the exact same way (expected), but the symbol materialization failure list is different.
When it is defined in the same cppdef as the sumc function, the list includes the sumc
function.
cling JIT session error: Failed to materialize symbols: { (main, { _Z4sumcii }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZN16__cppyy_internal10fptr_wrap1Eii }) }
When it is on a different cppdef call, it prints the original list.
Thought it would be nice to call the wrapper function directly and got the following.
IncrementalExecutor::executeFunction: symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE17_M_use_local_dataEv' unresolved while linking symbol '__cf_4'!
You are probably missing the definition of std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_use_local_data()
Maybe you need to load the corresponding shared library?
Traceback (most recent call last):
File "/home/gabriel/./Desktop/test.py", line 67, in <module>
print(__cppyy_internal.fptr_wrap99999( 3, 7))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cppyy.gbl.std.runtime_error: int __cppyy_internal::fptr_wrap99999(int arg0, int arg1) =>
runtime_error: failed to resolve function
GLIBCXX_USE_CXX11_ABI shenanigans?
I can reproduce this to differing extents in ubuntu and fedora docker containers. The ubuntu one is set up as follows:
# docker run -t -i --rm ubuntu bash
apt-get update && apt-get install -y python3 python3-pip python3-dev build-essential wget
python3 -V
pip3 -V
pip3 install cppyy
Now, running the examples from above and my Observer code from #189 works without warnings, but loading ogdf-python yields a similar problem:
# pip3 install ogdf-python ogdf-wheel
# python3 -c "import ogdf_python"
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-142, _ZN4ogdfL17s_ogdfInitializerE, _GLOBAL__sub_I_cling_module_142, $.cling-module-142.__inits.0, __cxx_global_var_initcling_module_142_ }) }
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-142 }) }
On fedora the issue is easier to reproduce
# docker run -t -i --rm fedora bash
sudo dnf install -y python3 python3-pip python3-devel gcc g++ cmake wget
python3 -V
pip3 -V
pip3 install cppyy
mv /usr/local/lib64/python3.11/site-packages/cppyy_backend/lib/libcppyy_backend.so /usr/local/lib/python3.11/site-packages/cppyy_backend/lib/libcppyy_backend.so # fix #191
python3 -c "import cppyy" # works
The example from above here yields:
>>> call_int_int(add, 3, 7)
cling JIT session error: Failed to materialize symbols: { (main, { _ZN16__cppyy_internal10fptr_wrap1Eii }) }
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: int ::call_int_int(int(*)(int,int) f, int i1, int i2) =>
TypeError: could not convert argument 1
Running my Observer code:
# wget https://gist.githubusercontent.com/N-Coder/c1fafdd5ff2aae1a134852416e9e3587/raw/f7189e70f5df36cba36cfc6da36332c965ddc125/test.cpp
# python3
Python 3.11.5 (main, Aug 28 2023, 00:00:00) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cppyy import *
>>> import cppyy.gbl as c
>>>
>>> include("test.cpp")
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { $.cling-module-142.__inits.0, _ZN10MyObserverD1Ev, _ZN10MyObserverD0Ev, _ZTV10MyObserver, _ZN8Observer7observeEP10Observable, _ZN10MyObserver12onRegisteredEP10Observable, _ZTI10MyObserver, long_lived_obs, _ZN8ObserverD1Ev, _ZN8Observer12onRegisteredEP10Observable, __orc_init_func.cling-module-142, _ZNSt7__cxx114listIP8ObserverSaIS2_EED1Ev, _GLOBAL__sub_I_cling_module_142, _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag, _ZTS10MyObserver, _ZN10MyObserverD2Ev, _ZTI8Observer, _ZTS8Observer, _ZTV8Observer, main, _ZN10MyObserverC2Ev, _ZN8ObserverD2Ev, _ZN8ObserverD0Ev, __cxx_global_var_initcling_module_142_, _ZN10MyObserver11onThingDoneERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE, __clang_call_terminate }) }
True
>>>
>>> class MyObserver(c.Observer):
... def onThingDone(self, msg):
... print(id(self), "onThingDone", msg)
... def onRegistered(self, prev):
... print(id(self), "onRegistered", "was", prev, "is", self.observed)
...
>>> obj = c.Observable()
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-142 }) }
>>> obs = MyObserver()
cling JIT session error: Failed to materialize symbols: { (main, { _ZN8Observer7observeEP10Observable }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZTVN16__cppyy_internal11Dispatcher1E }) }
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: none of the 2 overloaded methods succeeded. Full details:
none of the 3 overloaded methods succeeded. Full details:
Dispatcher1::Dispatcher1(const __cppyy_internal::Dispatcher1& other) =>
TypeError: takes at least 1 arguments (0 given)
Dispatcher1::Dispatcher1(const Observer& a0) =>
TypeError: takes at least 1 arguments (0 given)
__cppyy_internal::Dispatcher1 constructor failed
none of the 3 overloaded methods succeeded. Full details:
Dispatcher1::Dispatcher1(const __cppyy_internal::Dispatcher1& other) =>
TypeError: takes at least 1 arguments (0 given)
Dispatcher1::Dispatcher1(const Observer& a0) =>
TypeError: takes at least 1 arguments (0 given)
__cppyy_internal::Dispatcher1 constructor failed
>>> exit()
cling JIT session error: Failed to materialize symbols: { (main, { _ZN8Observer7observeEP10Observable }) }
And loading ogdf-python:
# python3 -c "import ogdf_python"
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __cxx_global_var_initcling_module_146_, _GLOBAL__sub_I_cling_module_146, __orc_init_func.cling-module-146, _ZN4ogdfL17s_ogdfInitializerE, $.cling-module-146.__inits.0 }) }
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-146 }) }
It seems that some fedora update now broke cppyy version 2.4, so now downgrading is also no longer an option and I currently cannot install a working cppyy version (2.4 due to below error, 3.0 due to the "Failed to materialize symbols" problems).
# docker run -t -i --rm fedora bash
sudo dnf update -y
sudo dnf install -y python3 python3-pip python3-devel gcc g++ cmake wget
pip3 install "cppyy<3"
mv /usr/local/lib64/python3.11/site-packages/cppyy_backend/lib/libcppyy_backend.so /usr/local/lib/python3.11/site-packages/cppyy_backend/lib/libcppyy_backend.so # fix #191
python3 -c "import cppyy"
There seem to be compile errors due to the "(Re-)building pre-compiled headers" and also some after "No precompiled header available (failed to build)" and lastly everything breaks down with
/usr/lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/cmath:93:3: error: redefinition of 'acos'
[...]
fatal error: too many errors emitted, stopping now [-ferror-limit=]
*** Break *** segmentation violation
Generating stack trace...
sh: line 1: which: command not found
/usr/bin/addr2line: '0x000055dea312b095': No such file
0x00007f3c9619f6c6 in CPyCppyy::CreateScopeProxy(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, _object*, unsigned int) at /usr/include/c++/13/bits/new_allocator.h:92 from /usr/local/lib64/python3.11/site-packages/libcppyy.cpython-311-x86_64-linux-gnu.so
@wlav found a workaround in https://github.com/sxs-collaboration/spectre/pull/5222/files#diff-093aadf224e5fee0d33ae1810f2f1c23304fb5ca398ba6b96c4e7918e0811729
It is apparently a bug caused by the PCH (I've been hitting a few with C++20, so I'm not surprised).
import cppyy
cppyy.cppdef("""
template std::__cxx11::basic_string<char,
std::char_traits<char>,
std::allocator<char>>::pointer std::__cxx11::basic_string<char, std::char_traits<char>,
std::allocator<char>>::_M_use_local_data();
template std::__cxx11::basic_string<wchar_t,
std::char_traits<wchar_t>,
std::allocator<wchar_t>>::pointer std::__cxx11::basic_string<wchar_t,
std::char_traits<wchar_t>,
std::allocator<wchar_t>>::_M_use_local_data();
template<typename T>
T call_int_int_temp(T (*f)(T,T), T i1, T i2) {
return f(i1, i2);
}
int sumc(int a, int b)
{
return a+b;
}
""")
from cppyy.gbl import call_int_int_temp, sumc
def add(a: int, b: int)->int:
return a + b
print(call_int_int_temp[int](sumc, 3, 7)) # prints 10 as expected
print(call_int_int_temp[int](add, 3, 7)) # works just fine
cppyy.cppdef(r"""
int call_int_int(int (*f)(int, int), int i1, int i2) {
return f(i1, i2);
}
""")
from cppyy.gbl import call_int_int
print(call_int_int(sumc, 3, 7)) # works fine
print(call_int_int(lambda x, y: x*y, 3, 7)) # works fine
Still checking other cases. Hopefully fixes them all. :D
Update: for all Python callbacks I've tried, this seems to do the trick. Other symbol materialization issues related to static initialization are a different problem.
I can not reproduce the problem with all provided docker images/examples, but where I can, I confirm that the fix found by @Gabrielcarvfer solves the issue (many, many, thanks for that!).
I simplified it to:
+ gInterpreter->Declare(
+ "#if defined(__GLIBCXX__)\n"
+ "template std::string::pointer std::string::_M_use_local_data();\n"
+ "template std::wstring::pointer std::wstring::_M_use_local_data();\n"
+ "#endif\n"
+ );
in src/clingwrapper.cxx
.
This has been a long run ... with this one, the Mac issue, and the gcc12 problem resolved (famous last words :) ), I can finally cut a release. Again, many thanks!
Just to add to my own optimistic message: likely the reason that it was not reproducible with some docker images, is that the two offending methods were introduced only with gcc12.
Should all be good now with release 3.1.0. Feel free to reopen if you find otherwise.
Bad news, I still get this with ogdf-python on my Fedora 38...
$ python -m ogdf_python
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { _ZN4ogdfL17s_ogdfInitializerE, __orc_init_func.cling-module-150, _GLOBAL__sub_I_cling_module_150, __cxx_global_var_initcling_module_150_, $.cling-module-150.__inits.0 }) }
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-150 }) }
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC2IS3_EEPKcRKS3_, _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev, _GLOBAL__sub_I_cling_module_162, $.cling-module-162.__inits.0, _ZN9get_macro16var_OGDF_VERSIONB5cxx11E, __cxx_global_var_initcling_module_162_, __orc_init_func.cling-module-162 }) }
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-162 }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZN9get_macro16var_OGDF_VERSIONB5cxx11E }) }
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/tmp/venv/lib64/python3.11/site-packages/ogdf_python/__main__.py", line 5, in <module>
pprint(get_ogdf_info(), sort_dicts=False)
^^^^^^^^^^^^^^^
File "/tmp/venv/lib64/python3.11/site-packages/ogdf_python/info.py", line 63, in get_ogdf_info
"ogdf_version": get_macro("OGDF_VERSION").strip('"'),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'strip'
$ python
Python 3.11.6 (main, Oct 3 2023, 00:00:00) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ogdf_python import *
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { _GLOBAL__sub_I_cling_module_150, $.cling-module-150.__inits.0, _ZN4ogdfL17s_ogdfInitializerE, __cxx_global_var_initcling_module_150_, __orc_init_func.cling-module-150 }) }
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-150 }) }
>>> G = ogdf.Graph()
>>> print(G)
<cppyy.gbl.ogdf.Graph object at 0x55a35712c770>
$ pip freeze
cppyy==3.1.0
cppyy-backend==1.15.0
cppyy-cling==6.30.0
CPyCppyy==1.12.14
ogdf-python==0.3.0
ogdf-wheel==2023.9
$ uname -a
Linux escher 6.5.10-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Nov 2 19:59:55 UTC 2023 x86_64 GNU/Linux
$ lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: Fedora
Description: Fedora release 38 (Thirty Eight)
Release: 38
Codename: ThirtyEight
Good news is that seems that even though these warnings, many things seem to work again, although that needs further testing. My Observer example from here is not working:
$ python
Python 3.11.6 (main, Oct 3 2023, 00:00:00) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cppyy import *
>>> import cppyy.gbl as c
>>>
>>> include("test.cpp")
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { _ZN10MyObserverC2Ev, _ZTV8Observer, _ZN10MyObserverD1Ev, _ZTI8Observer, _ZTV10MyObserver, _ZN8ObserverD0Ev, _ZN10MyObserverD0Ev, _ZNSt7__cxx114listIP8ObserverSaIS2_EED1Ev, main, __orc_init_func.cling-module-146, _ZN8ObserverD1Ev, _ZN8Observer12onRegisteredEP10Observable, _ZTI10MyObserver, __clang_call_terminate, _ZN8ObserverD2Ev, _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC2IS3_EEPKcRKS3_, __cxx_global_var_initcling_module_146_, _ZTS10MyObserver, _ZN10MyObserverD2Ev, _ZN10MyObserver11onThingDoneERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE, long_lived_obs, $.cling-module-146.__inits.0, _GLOBAL__sub_I_cling_module_146, _ZN10MyObserver12onRegisteredEP10Observable, _ZTS8Observer, _ZN8Observer7observeEP10Observable }) }
True
>>>
>>> class MyObserver(c.Observer):
... def onThingDone(self, msg):
... print(id(self), "onThingDone", msg)
... def onRegistered(self, prev):
... print(id(self), "onRegistered", "was", prev, "is", self.observed)
...
>>>
>>> obj = c.Observable()
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-146 }) }
>>> obs = MyObserver()
cling JIT session error: Failed to materialize symbols: { (main, { _ZN8Observer7observeEP10Observable }) }
cling JIT session error: Failed to materialize symbols: { (main, { _ZTVN16__cppyy_internal11Dispatcher1E }) }
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: none of the 2 overloaded methods succeeded. Full details:
none of the 3 overloaded methods succeeded. Full details:
Dispatcher1::Dispatcher1(const __cppyy_internal::Dispatcher1& other) =>
TypeError: takes at least 1 arguments (0 given)
Dispatcher1::Dispatcher1(const Observer& a0) =>
TypeError: takes at least 1 arguments (0 given)
__cppyy_internal::Dispatcher1 constructor failed
none of the 3 overloaded methods succeeded. Full details:
Dispatcher1::Dispatcher1(const __cppyy_internal::Dispatcher1& other) =>
TypeError: takes at least 1 arguments (0 given)
Dispatcher1::Dispatcher1(const Observer& a0) =>
TypeError: takes at least 1 arguments (0 given)
__cppyy_internal::Dispatcher1 constructor failed
>>> obs
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'obs' is not defined. Did you mean: 'obj'?
>>> exit()
cling JIT session error: Failed to materialize symbols: { (main, { _ZN8Observer7observeEP10Observable }) }
:(
Can you check the output of:
$ python
>>> import cppyy
>>> cppyy.cppexec("std::cerr << __GLIBCXX__ << std::endl;")
I'm curious (hoping, rather) whether it's something as simple as me having the cutoff incorrect of the GLIBCXX where the problem started.
Got an error on Ubuntu 22.04 with GCC 11.4.0
>>> import cppyy
input_line_10:2:45: error: explicit instantiation of '_M_use_local_data' does not refer to a function template, variable template, member function, member class, or static data member
template std::string::pointer std::string::_M_use_local_data();
^
input_line_10:3:46: error: explicit instantiation of '_M_use_local_data' does not refer to a function template, variable template, member function, member class, or static data member
template std::wstring::pointer std::wstring::_M_use_local_data();
^
>>> cppyy.cppexec("std::cerr << __GLIBCXX__ << std::endl;")
20230528
True
>>>
But it is working for me on Ubuntu 23.04
Python 3.11.4 (main, Jun 9 2023, 07:59:55) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cppyy
>>> cppyy.cppexec("std::cout << __GLIBCXX__ << std::endl;")
20230510
True
>>>
That error is coming from:
gInterpreter->Declare(
"#if defined(__GLIBCXX__) && __GLIBCXX__ >= 20220506\n"
"template std::string::pointer std::string::_M_use_local_data();\n"
"template std::wstring::pointer std::wstring::_M_use_local_data();\n"
"#endif\n"
);
where that __GLIBCXX__
is supposed to check for the presence of _M_use_local_data
, which is the cause of the symbols not materializing. I guess that the check isn't working as I expected to identify the presence of that method (it was introduced at some point in gcc11).
Looks like issue with std::allocator<char>
, not with that std::string
constructor itself (which just uses it). Also, this Fedora system has gcc13, so may have had further changes that the original report didn't fix.
In fact, if I remove all other workarounds and just add this at the top of the script (after importing cppyy), then things run fine for me on the Fedora docker image (don't know yet about the others):
cppyy.cppdef("template class std::allocator<char>;")
Edit: even better: I can add the above to the clingwrapper.cxx
file so that the compiler (as opposed to the JIT) instantiates it. The JIT then finds it as an ordinary linker symbol. It does not seem to require the same for std::string
and std::wstring
, but I can add these for good measure. (The advantage of doing this in the .cxx, is that there's no risk of errors for duplicate explicit instantiations, as there is for the JIT, and by using the standard classes rather than the implementation-specific methods, I don't need to do a version check.)
@wlav I could create a PR to set up GitHub Actions CI in this repo to continuously perform (at least smoke) tests on different OSes and versions (say latest and oldest supported Ubuntu, Debian and Fedora via Docker and also MacOs and Windows). Test coverage may be extendable but just checking for exceptions or warnings on startup or with a handful of simple use-cases (eg from the docs or from GitHub issues to check for regressions) might be a good start. It might be a little more challenging to bring all things together and in-sync with the split across different packages and repos that we have here, so I'd initially gear this towards checking (almost) released versions for problems on individual OSes or trying new things based on a released configuration instead of testing everything live alongside development. How would that sound to you? If you're open to this (you can still easily ignore the CI if you don't care for that) I'd happily draft a PR to set this up.
Oh, and the cppdef seems to have fixed it!
$ python
Python 3.11.6 (main, Oct 3 2023, 00:00:00) [GCC 13.2.1 20230728 (Red Hat 13.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cppyy
>>> cppyy.cppexec("std::cerr << __GLIBCXX__ << std::endl;")
20231011
True
>>> cppyy.cppdef("template class std::allocator<char>;")
True
>>> from cppyy import *
>>> import cppyy.gbl as c
>>>
>>> include("test.cpp")
0x7f48446e0000 construct
True
>>> class MyObserver(c.Observer):
... def onThingDone(self, msg):
... print(id(self), "onThingDone", msg)
... def onRegistered(self, prev):
... print(id(self), "onRegistered", "was", prev, "is", self.observed)
...
>>>
>>> obj = c.Observable()
>>> obs = MyObserver()
>>> exit()
0x7f48446e0000 destruct
$ # edit in your cppdef("template class std::allocator<char>;") into the ogdf_python loader...
$ python -m ogdf_python
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { $.cling-module-151.__inits.0, __cxx_global_var_initcling_module_151_, _ZN4ogdfL17s_ogdfInitializerE, _GLOBAL__sub_I_cling_module_151, __orc_init_func.cling-module-151 }) }
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-151 }) }
{'ogdf_path': '/tmp/venv/lib64/libOGDF.so', ...}
While there still is a warning for ogdf_python
, the library now seems to be loaded correctly and working. You don't happen to have similar fix ready for s_ogdfInitializer
, have you? My attempt at simply copying it from the source code lead to redefinition errors...
Edit: interestingly, turning off the declaration of static Initialization s_ogdfInitializer
in that header by doing #define OGDF_INSTALL
before loading it also fixed the remaining warnings (it now loads completely without warnings!). Unfortunately I'm not quite sure about all consequences of this, so I don't know whether this somewhat hacky fix is the way to go...
For CI, yes that may work, but it's a bit convoluted to build from source when not everything is released b/c of the way pip deals (or rather not deals) with dependencies. cppyy 4.0.0 (based on clang-repl) should be consolidated, but is some way off (upstream says that about half the tests succeed; I haven't worked on it myself yet). Building the full backend (which includes LLVM) is also costly and doesn't work on platforms such as manylinux which insists on using a pre-C++11 ABI. :( Maybe something that can run the full test-suite manually just prior to a release? (Note that the test-suite is only 100% error free on Linux and Mac; there are a handful of errors on Windows left.)
For the static initializer, there is this: https://github.com/cms-sw/cmssw/issues/43077#issuecomment-1781310510 so upstream is working on that (and they have to care alot about CMS software ;) ). (Yes, I've seen your edit, but there's something real still there.)
I'm going to cut a new release later today (a fix in clingwrapper is easy to release, which is why it exists as a separate package, but ironically enough this is the first time since its existence I'm making use of that). I just want to get this one done as well: https://github.com/wlav/cppyy/issues/191, since I'm now on this Fedora image anyway where it's easy to reproduce.
Release 3.1.1 works for me (famous last words) on that Fedora docker image ...
Things are looking very good now on Fedora. Install works flawlessly, I only get the warning from the ogdf initializer, and the Observer
example now exits cleanly with only the warning "sys:1: RuntimeWarning: Call attempted on deleted python-side proxy". So both issues are now resolved from my point of view.
As you are already building cppyy-cling wheels in CI and the rest is easy to install, I threw together this script which should work on debian, ubuntu and fedora docker containers to set up all components with their latest repo versions. Currently, this fails with the following error on all three systems:
File "/venv/lib64/python3.11/site-packages/cppyy_backend/loader.py", line 92, in load_cpp_backend
raise RuntimeError("could not load cppyy_backend library, details:\n%s" %
RuntimeError: could not load cppyy_backend library, details:
libcppyy_backend.cpython-311-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory
/venv/lib64/python3.11/site-packages/cppyy_backend/lib/libcppyy_backend.cpython-311-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory
/venv/lib64/python3.11/site-packages/cppyy_backend/lib/libcppyy_backend.so: undefined symbol: _ZN11CppyyLegacy10TClassEdit9CleanTypeB5cxx11EPKciPS2_
libcppyy_backend.so: cannot open shared object file: No such file or directory
So it seems the four components are currently somehow in an inconsistent state. Still, I'll set up a CI pipeline around this next and ensure that there are options to only run the versions from PyPi and to manually trigger such runs. I'll open a new issue for this once there is further progress...
My issue might be related to https://github.com/wlav/cppyy/issues/156
I installed
cppyy==3.0.0
inside a virtual environment usingvirtualenvwrapper
. The following code fromcppyy
examples fails:The error messages are: