Open saraedum opened 2 years ago
@wlav in earlier versions, the allDict.cxx.pch
was created inside the environment as e.g.,
/home/jule/proj/umamba/envs/test/lib/python3.7/site-packages/cppyy_backend/etc/allDict.cxx.pch.6.25.1
I don't really understand what this file is doing exactly, but it seems to depend on the headers installed into the conda environment so it should be created inside the environment and not be shared in the home directory it seems?
Is this something that we are doing wrong in our build of cppyy?
Can we set the CLING_STANDARD_PCH
when activating cppyy to point to a place inside the environment?
I don't really understand what this file is doing exactly
The PCH (Pre-Compiled Header) is LLVM bitcode of parsed headers. It (considerably) speeds up loading standard and system headers. It's not functionally necessary, although with it, cppyy can actually run w/o a local compiler install, which can be convenient for deployment (upstream does this, but with precompiled modules instead of a single PCH). You can actually add your own headers to allHeaders.txt and see a speedup for your own projects.
One problem with it, is that some features get "baked in." Notably, AVX, OpenMP, and CUDA support.
What changed in 2.3.0 (which now supports, experimentally, CUDA), that depending on the flags EXTRA_CLING_ARGS
envar (in particular -mavx
which is enabled based on CPU support for AVX), the PCH is labeled for these features.
My guess of what is going on here, is that there's a PCH with .avx
in the name from the Conda build, but locally does not support AVX and it gets switched off, or v.v. (OpenMP and CUDA are not automatic, although if CUDA is enabled, you get two PCHs). As of 2.3.0, that will cause a rebuild to guarantee consistency.
Yes, you can set CLING_STANDARD_PCH
to some file and that will overwrite other choices. However, that PCH may not work properly if that "up-to-date" file exists but was previously build with conflicting EXTRA_CLING_ARGS
. E.g. class sizes (such as in Eigen
for example) can differ depending on whether AVX is enabled, resulting in trouble (segfaults or worse) if a library built with -mavx
is used through Cling with a PCH w/o -mavx
.
Thanks for the explanation. I'll see if something has changed in our build as you indicate.
It's not functionally necessary
That's strange. I have a project here that crashes when running with CLING_STANDARD_PCH=none
, namely I get:
input_line_48:2:19: error: no type named 'TStreamerInfo' in namespace 'CppyyLegacy'
new CppyyLegacy::TStreamerInfo{}
The stacktrace (probably not too enlightening):
/tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/cysignals/signals.cpython-39-x86_64-linux-gnu.so(+0x6f4d)[0x7f4e244edf4d] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/cysignals/signals.cpython-39-x86_64-linux-gnu.so(+0x714c)[0x7f4e244ee14c] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/cysignals/signals.cpython-39-x86_64-linux-gnu.so(+0x972c)[0x7f4e244f072c] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f4e2efd0730] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/cppyy_backend/lib/libCoreLegacy.so(_ZN11CppyyLegacy6TClass4InitEPKcsPKSt9type_infoPNS_16TVirtualIsAProxyES2_S2_iiPNS_11ClassInfo_tEb+0x5ea)[0x7f4e2de8d3da] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/cppyy_backend/lib/libCoreLegacy.so(_ZN11CppyyLegacy6TClassC1EPKcsb+0x236)[0x7f4e2de94a36] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/cppyy_backend/lib/libCling.so(+0x40cb26)[0x7f4e2b89fb26] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/cppyy_backend/lib/libCoreLegacy.so(_ZN11CppyyLegacy6TClass8GetClassEPKcbb+0x557)[0x7f4e2de8ea87] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/cppyy_backend/lib/libcppyy_backend.so(_ZN5Cppyy8GetScopeERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x104)[0x7f4e26679c74] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so(_ZN8CPyCppyy14CreateExecutorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_10DimensionsE+0x489)[0x7f4e261d6f19] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so(_ZN8CPyCppyy9CPPMethod13InitExecutor_ERPNS_8ExecutorEPNS_11CallContextE+0x47)[0x7f4e2619ee37] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so(_ZN8CPyCppyy9CPPMethod10InitializeEPNS_11CallContextE+0x43)[0x7f4e261a0783] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so(_ZN8CPyCppyy9CPPMethod4CallERPNS_11CPPInstanceEPKP7_objectmS5_PNS_11CallContextE+0x201)[0x7f4e261a0fa1] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/libcppyy.cpython-39-x86_64-linux-gnu.so(+0x4930f)[0x7f4e261a730f] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyEval_EvalCodeWithName+0x47)[0x564d1eb08257] python(PyEval_EvalCodeEx+0x39)[0x564d1eb08299] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/categories/map.cpython-39-x86_64-linux-gnu.so(+0xc4ce)[0x7f4e2311a4ce] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/categories/map.cpython-39-x86_64-linux-gnu.so(+0xea52)[0x7f4e2311ca52] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/categories/map.cpython-39-x86_64-linux-gnu.so(+0x1d253)[0x7f4e2312b253] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/structure/coerce_actions.cpython-39-x86_64-linux-gnu.so(+0x1599a)[0x7f4e22e8799a] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/categories/action.cpython-39-x86_64-linux-gnu.so(+0xb328)[0x7f4e22eb0328] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/categories/action.cpython-39-x86_64-linux-gnu.so(+0x8751)[0x7f4e22ead751] python(+0x16cce5)[0x564d1ea49ce5] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/structure/coerce_actions.cpython-39-x86_64-linux-gnu.so(+0x20b14)[0x7f4e22e92b14] python(+0x153645)[0x564d1ea30645] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/structure/coerce_actions.cpython-39-x86_64-linux-gnu.so(+0x17af5)[0x7f4e22e89af5] python(+0x16cce5)[0x564d1ea49ce5] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/structure/parent.cpython-39-x86_64-linux-gnu.so(+0xcfaf)[0x7f4e23161faf] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/structure/parent.cpython-39-x86_64-linux-gnu.so(+0x37776)[0x7f4e2318c776] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/structure/parent.cpython-39-x86_64-linux-gnu.so(+0x1ed97)[0x7f4e23173d97] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/structure/coerce.cpython-39-x86_64-linux-gnu.so(+0x385b0)[0x7f4e22ff45b0] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/structure/coerce.cpython-39-x86_64-linux-gnu.so(+0x12835)[0x7f4e22fce835] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/structure/coerce.cpython-39-x86_64-linux-gnu.so(+0x2239c)[0x7f4e22fde39c] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/structure/element.cpython-39-x86_64-linux-gnu.so(+0x1865a)[0x7f4e2336865a] python(PyNumber_Multiply+0x157)[0x564d1ea7c217] python(_PyEval_EvalFrameDefault+0xeb3)[0x564d1eac91a3] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyEval_EvalCodeWithName+0x47)[0x564d1eb08257] python(PyEval_EvalCodeEx+0x39)[0x564d1eb08299] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/misc/cachefunc.cpython-39-x86_64-linux-gnu.so(+0x123de)[0x7f4e2e3ce3de] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/misc/cachefunc.cpython-39-x86_64-linux-gnu.so(+0x1d248)[0x7f4e2e3d9248] python(_PyObject_MakeTpCall+0x4e7)[0x564d1ea2b297] python(_PyEval_EvalFrameDefault+0x5301)[0x564d1eacd5f1] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/misc/cachefunc.cpython-39-x86_64-linux-gnu.so(+0xfbb2)[0x7f4e2e3cbbb2] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/misc/cachefunc.cpython-39-x86_64-linux-gnu.so(+0x12447)[0x7f4e2e3ce447] /tmp/ruth/micromamba/envs/flatsurvey-build/lib/python3.9/site-packages/sage/misc/cachefunc.cpython-39-x86_64-linux-gnu.so(+0x1d248)[0x7f4e2e3d9248] python(_PyObject_MakeTpCall+0x4e7)[0x564d1ea2b297] python(_PyEval_EvalFrameDefault+0x5301)[0x564d1eacd5f1] python(+0x1412b0)[0x564d1ea1e2b0] python(+0x1acd31)[0x564d1ea89d31] python(PyObject_GetAttr+0x26f)[0x564d1ea9b67f] python(_PyEval_EvalFrameDefault+0x6db)[0x564d1eac89cb] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(PyObject_Call+0x1c2)[0x564d1ea4a1a2] python(_PyEval_EvalFrameDefault+0x2fba)[0x564d1eacb2aa] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(PyObject_Call+0x1c2)[0x564d1ea4a1a2] python(_PyEval_EvalFrameDefault+0x2fba)[0x564d1eacb2aa] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(+0x1b7464)[0x564d1ea94464] python(_PyObject_MakeTpCall+0x33b)[0x564d1ea2b0eb] python(_PyEval_EvalFrameDefault+0x5614)[0x564d1eacd904] python(+0x140a80)[0x564d1ea1da80] python(+0x1ac00a)[0x564d1ea8900a] python(PyObject_Call+0xb8)[0x564d1ea4a098] python(_PyEval_EvalFrameDefault+0x2fba)[0x564d1eacb2aa] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x4cb7)[0x564d1eaccfa7] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x4cb7)[0x564d1eaccfa7] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x4cb7)[0x564d1eaccfa7] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x4cb7)[0x564d1eaccfa7] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x4cb7)[0x564d1eaccfa7] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x4cb7)[0x564d1eaccfa7] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x4cb7)[0x564d1eaccfa7] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x4cb7)[0x564d1eaccfa7] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x140a80)[0x564d1ea1da80] python(+0x1ac00a)[0x564d1ea8900a] python(_PyEval_EvalFrameDefault+0x11ba)[0x564d1eac94aa] python(_PyFunction_Vectorcall+0x19a)[0x564d1ea887da] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(PyObject_Call+0xb8)[0x564d1ea4a098] python(_PyEval_EvalFrameDefault+0x2fba)[0x564d1eacb2aa] python(+0x140a80)[0x564d1ea1da80] python(+0x1ac00a)[0x564d1ea8900a] python(PyObject_Call+0xb8)[0x564d1ea4a098] python(_PyEval_EvalFrameDefault+0x2fba)[0x564d1eacb2aa] python(+0x1412b0)[0x564d1ea1e2b0] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x60e)[0x564d1eac88fe] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(+0x1ac1e3)[0x564d1ea891e3] python(PyObject_Call+0x1c2)[0x564d1ea4a1a2] python(_PyEval_EvalFrameDefault+0x2fba)[0x564d1eacb2aa] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyObject_FastCallDictTstate+0x65)[0x564d1ea49dc5] python(_PyObject_Call_Prepend+0x6a)[0x564d1ea49f1a] python(+0x265f25)[0x564d1eb42f25] python(_PyObject_MakeTpCall+0x4e7)[0x564d1ea2b297] python(_PyEval_EvalFrameDefault+0x4de7)[0x564d1eacd0d7] python(+0x140a80)[0x564d1ea1da80] python(_PyEval_EvalCodeWithName+0x47)[0x564d1eb08257] python(PyEval_EvalCodeEx+0x39)[0x564d1eb08299] python(PyEval_EvalCode+0x1b)[0x564d1eb082bb] python(+0x2a017d)[0x564d1eb7d17d] python(+0x1a4e6a)[0x564d1ea81e6a] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(_PyEval_EvalFrameDefault+0x3c1)[0x564d1eac86b1] python(+0x140a80)[0x564d1ea1da80] python(_PyFunction_Vectorcall+0x336)[0x564d1ea88976] python(PyObject_Call+0x1c2)[0x564d1ea4a1a2] python(+0x29b52b)[0x564d1eb7852b] python(Py_RunMain+0xc5)[0x564d1eb7ca25] python(Py_BytesMain+0x39)[0x564d1eb7ce39] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f4e2ec8a09b] python(+0x218321)[0x564d1eaf5321]\
Actually, that stack trace is very useful. :)
That code is the old "plugin" setup of upstream. It executes a line on Cling to instantiate a class through the autoloader, then takes the result to cast it. This way, the compiled code does not need to #Include
the headers of these classes and thus not depend on the library implementing them. There's actually no value to this, but I think upstream at some point wanted to have different implementations, just that that never happened.
With the PCH, TStreamerInfo.h
is contained in it. W/o, it must be loaded explicitly. That is either not happening for reasons that I don't understand or it is not available through the Cling include directories.
It's another "this works for me" (with both conda and PyPI) cases ...
That said, when I run:
$ strace python -c "import cppyy" |& grep TStreamerInfo
with export CLING_STANDARD_PCH=none
, there is no output, whereas there is when that envar is unset. I would have expected the opposite (to be sure, the output only shows a stat
, not a read
).
Will do some digging.
Just to clarify, CLING_STANDARD_PCH=none python -c 'import cppyy'
actually works fine. It's a more involved program where things fail.
Unfortunately, there is no strace available on the system where I am seeing this issue. I am not sure whether it can be installed, I could ask.
On my local system, I see the following:
$ python -m flatsurvey.worker ngon -a 2 -a 3 -a 6 # works
$ CLING_STANDARD_PCH=none python -m flatsurvey.worker ngon -a 2 -a 3 -a 6
Traceback (most recent call last):
File "/home/jule/proj/umamba/envs/flatsurf/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/jule/proj/umamba/envs/flatsurf/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/jule/proj/eskin/flatsurvey/flatsurvey/worker/__main__.py", line 78, in <module>
import flatsurvey.jobs
File "/home/jule/proj/eskin/flatsurvey/flatsurvey/jobs/__init__.py", line 35, in <module>
from .undetermined_interval_exchange_transformation import (
File "/home/jule/proj/eskin/flatsurvey/flatsurvey/jobs/undetermined_interval_exchange_transformation.py", line 48, in <module>
import pyeantic
File "/home/jule/proj/umamba/envs/flatsurf/lib/python3.9/site-packages/pyeantic/__init__.py", line 26, in <module>
from .cppyy_eantic import eantic
File "/home/jule/proj/umamba/envs/flatsurf/lib/python3.9/site-packages/pyeantic/cppyy_eantic.py", line 59, in <module>
from cppyythonizations.pickling.cereal import enable_cereal
File "/home/jule/proj/umamba/envs/flatsurf/lib/python3.9/site-packages/cppyythonizations/pickling/cereal.py", line 115, in <module>
cppyy.include(os.path.join(os.path.dirname(__file__), '..', 'include/cereal/details/static_object.hpp'))
File "/home/jule/.local/lib/python3.9/site-packages/cppyy/__init__.py", line 235, in include
raise ImportError('Failed to load header file "%s"%s' % (header, err.err))
ImportError: Failed to load header file "/home/jule/proj/umamba/envs/flatsurf/lib/python3.9/site-packages/cppyythonizations/pickling/../include/cereal/details/static_object.hpp"
In file included from input_line_18:1:
/home/jule/proj/umamba/envs/flatsurf/lib/python3.9/site-packages/cppyythonizations/pickling/../include/cereal/details/static_object.hpp:71:5: error: 'auto' return without trailing return type; deduced return types are a C++14 extension
auto& getStatics() {
^
/home/jule/proj/umamba/envs/flatsurf/lib/python3.9/site-packages/cppyythonizations/pickling/../include/cereal/details/static_object.hpp:73:14: error: no viable conversion from returned value of type 'std::unordered_map<std::type_index, void *>' to function return type 'int'
return statics;
$ conda list |grep cppyy
cppyy 2.3.0 py39hd14de60_0 conda-forge
cppyy-backend 1.14.8 py39hf939315_0 conda-forge
cppyy-cling 6.25.3 py39h3d66fe8_0 conda-forge
cppyythonizations 1.2.1 pyhd8ed1ab_0 conda-forge
cpycppyy 1.12.9 py39hf939315_0 conda-forge
$ export |grep -E 'FLAGS|CLING' # nothing overly suspicious here, note that the CXXFLAGS contain -std=c++17
CFLAGS='-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/jule/proj/umamba/envs/flatsurf/include'
CPPFLAGS='-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/jule/proj/umamba/envs/flatsurf/include'
CXXFLAGS='-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/jule/proj/umamba/envs/flatsurf/include'
DEBUG_CFLAGS='-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /home/jule/proj/umamba/envs/flatsurf/include'
DEBUG_CPPFLAGS='-D_DEBUG -D_FORTIFY_SOURCE=2 -Og -isystem /home/jule/proj/umamba/envs/flatsurf/include'
DEBUG_CXXFLAGS='-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /home/jule/proj/umamba/envs/flatsurf/include'
LDFLAGS='-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,--allow-shlib-undefined -Wl,-rpath,/home/jule/proj/umamba/envs/flatsurf/lib -Wl,-rpath-link,/home/jule/proj/umamba/envs/flatsurf/lib -L/home/jule/proj/umamba/envs/flatsurf/lib'
$ EXTRA_CLING_ARGS=-std=c++17 CLING_STANDARD_PCH=none python -m flatsurvey.worker ngon -a 2 -a 3 -a 6 # works
So, I guess this is unrelated and somehow expected. Anyway, setting EXTRA_CLING_ARGS=-std=c++17
on the system where I see the no type named 'TStreamerInfo' in namespace 'CppyyLegacy'
does not fix that problem.
For building cppyy-cling
, the flag that sets the standard during build (this gets baked in) is STDCXX
. For PyPI (i.e. manylinux builds), it actually defaults to C++11 and that would explain the behavior. However, the default should be C++17 for other builds (see setup.py; the checking for "manylinux" is actually kinda dumb, so maybe the conda build system is incorrectly flagged as a "manylinux" box leading to a C++11 default):
_is_manylinux = None
def is_manylinux():
global _is_manylinux
if _is_manylinux is None:
_is_manylinux = False
try:
for line in open('/etc/redhat-release').readlines():
# mark manylinux1, manylinux2010, or manylinux2014
if 'CentOS release 5.11 (Final)' in line or \
'CentOS release 6.10 (Final)' in line or \
'CentOS Linux release 7.9.2009 (Core)' in line:
_is_manylinux = True
break
except (OSError, IOError):
pass
return _is_manylinux
And yes, the flags get "fixed up" when building the PCH, either from the STDCXX
envar or from the EXTRA_CLING_ARGS
one (if provided). This happens in makepch.py
.
Note that if EXTRA_CLING_ARGS
is set, the defaults (-O2
and -mavx
depending on processor capability) are not added. Otherwise, I would not expect it to make a difference.
For the original error, I'm trying to reproduce the error and am so far not able. I'm thinking that if the header is available (which I expect it to be if there are no other installation issues), then it must mean that somehow in the initialization order, the header is not included before first use of the class. The proper fix is then probably to add its inclusion in the ApplicationStarter
in clingwrapper.cxx
.
For building
cppyy-cling
, the flag that sets the standard during build (this gets baked in) isSTDCXX
. For PyPI (i.e. manylinux builds), it actually defaults to C++11 and that would explain the behavior. However, the default should be C++17 for other builds (see setup.py; the checking for "manylinux" is actually kinda dumb, so maybe the conda build system is incorrectly flagged as a "manylinux" box leading to a C++11 default):
We don't really use the setup.py on conda-forge but invoke cmake directly, see https://github.com/conda-forge/cppyy-cling-feedstock/blob/main/recipe/use-dumb-setup.py.patch. If you think that that's an issue, we could try to go back to using the setup.py but it seemed to be easier to hard-code the flags that we set instead of having setup.py decide things for us.
Anyway, we set -DCMAKE_CXX_STANDARD=17
Looking at the cmake flags that the setup.py is setting, I find the following differences:
-DLLVM_ENABLE_ASSERTIONS=0
, we don't.-Druntime_cxxmodules=OFF
, we don't.-Dbuiltin_zlib=ON
, we set it to OFF
.We also set lots of flag that we found in our build of root, if any of these look wrong to you, please let us know :)
setup.py sets -Druntime_cxxmodules=OFF, we don't.
Modules and the PCH are targeting the same feature: they both contain precompiled bitcode, just that the packaging of modules is, well, modular and can thus be used for non-system code. (They also deserialize less code when Cling is looking for just a single class or function, so reducing memory use.) I'd recommend using only one or the other. Both? I don't know what will happen.
I've not looked in a (too) long time into modules, but last time around they were only fully functional on Linux: I had trouble with debug builds on Mac and even upstream said that support for Windows was minimal. Upstream cares less about Windows, hence they were happy to move. I don't have that luxury.
Solution to issue cannot be found in the documentation.
Issue
The
allDict.cxx.pch
is not created inside the environment but in the user's home.Installed packages
Environment info