Support for clang-16 in cppyy

lukevalenty commented 1 week ago

I was super excited to try out cppyy to enable some property based testing for several open source libraries we have at Intel that are highly dependent on template types and compile-time processing. However, our libraries need at least clang 16. I saw that cling has support for this now, when will cppyy be updated with the latest version of cling with clang 16 support?

lukevalenty commented 1 week ago

Tagging @elbeno and @mjcaisse who will also be interested in this.

wlav commented 1 week ago

I have a big incentive to get that done soon as modern compilers on all 3 major platforms are now failing with LLVM-13 (mainly b/c C++20 is the chosen default, which now brings in code in standard headers using features that makes 13 choke). I started the work, but then upstream is also close to moving to LLVM-18, so waiting for that instead would save me some time, which is in short supply still.

Upstream (https://github.com/compiler-research) also has a fork of cppyy that, once completed, would make future updates to newer versions of LLVM much easier. (Time spent on that is one reason why I haven't kept up here.)

lukevalenty commented 6 days ago

That's great to hear @wlav! Do you have an idea of the time-frame for clang-16 or 18 backend? I'm trying to decide if its worthwhile to spend time on an alternative method or just wait a little bit longer. Any way contributors can help out?

wlav commented 6 days ago

Always consider alternatives. :) cppyy fills a rather unique niche: it either fits or it doesn't.

As for contributions, do you have access to / care about MS Windows? It's always a major holdup for preparing a release (I run it in a VM on Linux, but MS time limits on the images they provide so I have to redo everything for each new release and our IT security folks don't particularly like it), so if you could test LLVM16 there, that would be very helpful.

I've put in the repo what I tested on Linux. Haven't run the CI yet to check Windows, nor updated the version numbers, which makes building from source a bit more complicated. But as-is, it still has six failing tests (on Linux, see below), so I want to get that fixed first before moving on to Mac (and finally Windows).

FAILED test_cpp11features.py::TestCPP11FEATURES::test09_lambda_calls - TypeError: 'cppyy.LowLevelView' object is not callable
FAILED test_datatypes.py::TestDATATYPES::test46_small_int_enums - AssertionError: assert (SmallEnums::Int8Enum::E0) : (char) '\x00' == 0
FAILED test_datatypes.py::TestDATATYPES::test47_hidden_name_enum - NotImplementedError: unsigned int EnumFunctionSameName::bar(EnumFunctionSameName::foo f) =>
FAILED test_fragile.py::TestFRAGILE::test21_failing_cppcode - assert 'expectedunqualified-id' in "FailedtoparsethegivenC++code\x1b[1minput_line_2473:1:2:\x1b[0m\x1b[0;1;31merror:\x1b[0m\x1b[1minvaliddigit'a'indecimalconstant\x1b[0m1aap...
FAILED test_pythonify.py::TestPYTHONIFY::test19_keywords_and_defaults - NameError: name 'MyClass' is not defined
FAILED test_stltypes.py::TestSTLDEQUE::test01_deque_byvalue_regression - TypeError: void f(const unsigned char* buf) =>

Alternatively, given that the above errors are all corner cases, building it from source and putting it through the wringer on Linux is helpful, too. Might catch some more, LLVM16 specific, errors before releasing it.

lukevalenty commented 6 days ago

@wlav, I have a Windows laptop I can test out LLVM16 on. Point me in the right direction (repo/branch and build/test instructions) and I'll give it a go.

ETA: Just saw your commit, going to take a look now.

lukevalenty commented 6 days ago

Actually, I need some help knowing how to build locally for linux, mac, and windows. I have all three. I tried using the circleci recipe but that might not work well for working through errors and rebuilding/testing.

wlav commented 6 days ago

The CI scripts were contributed; I never got the circleci one to work. The reason for its existence was that github emulates ARM, which is slow, and the jobs run out of time. A Linux ARM wheel has been requested frequently, though, so would have been useful. The CI on github also has other limitations: the Linux one uses the old ABI and the Mac one cross compiles for ARM. The latter can't work b/c the build process creates a few executables that are then to be used in the build. That fails when cross compiling. Intel Mac and Windows on github CI was functional, IIRC. I have CI disabled until I get closer to a release; builds take to long to bother with every little commit.

Details on building from source are here: https://cppyy.readthedocs.io/en/latest/repositories.html#building-from-source

An important one is --no-use-pep517 as pip's dependency resolution is different for the build environment than for the runtime environment (always has been, but PEP517 makes it worse). All recent backends are strictly ABI compatible, though, so shouldn't be an issue when building the latest. For development on a local machine, I use cmake to build.

lukevalenty commented 5 days ago

Ok, I'll need to follow these instructions and let you know how it goes. What's the latest commit in which you expect all the unit tests to pass in x86_64 linux? I'm using ubuntu 22.04, do you see any problems with that? Any specific compiler and python versions I should be using?

wlav commented 5 days ago

Just prior to putting in the LLVM16 patches should work for all tests on Linux except 1 (test01_deque_byvalue_regression). That one fails b/c of a naming conflict between 2 tests (ie., it will work when run independently).

Last fully functional commit is c84cc074ff6708587880f1d86c3777675ef9d0c2

Note that there are a handful of tests that fail on Windows (have been for a long time); several esoteric, such as the use of std::tuple. (MSVC implements that through inheritance such that std::tuple<int, int> inherits from std::tuple<int>, with as a consequence that get() on the former is matched by get() on the base, if called earlier.)

I'm down to 4 failing tests on Linux.