Using with llvmlite

[saulshanabrook]

I am looking to call C functions and access C structs using llvmlite, python bindings for the LLVM API (https://github.com/numba/llvmlite/issues/385).

This seems to be similar to the problem this project is trying to solve. Is it possible to use this package to generate LLVM (or even better build it with llvmlite), instead of actually doing the function calls?

Basically, the use case is that I am wrapping C libraries in Numba and need to be able to call functions from them, create structs, and modify their contents.

[aguinet]

Hello!

Looking at your llvmlite issue, it looks like what you need is indeed dragonffi. The API to get what i call the "wrappers" in LLVM IR isn't exposed, but that could be done w/o too much trouble. The wrappers are functions that takes 3 arguments: a pointer to the return value, a pointer to a vector of pointers for the argument akd the number of arguments. These functions are then easy to call from LLVM.

What's your timing on this? I'm in holidays for one week w/o access to a proper computer to develop. If you're willing to try it I can guide you to the changes that could make this work.

Moreover, putting this into llvmlite might also be possible, but I would require including clang within llvmlite, something that would make this package not very light anymore ;)

[saulshanabrook]

The wrappers are functions that takes 3 arguments: a pointer to the return value, a pointer to a vector of pointers for the argument akd the number of arguments. These functions are then easy to call from LLVM.

Ah that makes sense. Most of the C calls I am wrapping I am actually generating myself for getting/setting attributes of structs (https://github.com/Quansight/numba-xnd/blob/47f257d16dbd2d7793bf4d377084427ed8c2915c/xnd_structinfo.c#L193). Just yesterday I decided to pass everything in as pointers as well as the return value, so it looks like I am mini-recreating dragonffi :)

Do you think it would be possible to expose getting/setting struct values somehow as LLVM IR? This is really what I need, and generating all the C functions is just a way to get to that.

What's your timing on this? I'm in holidays for one week w/o access to a proper computer to develop. If you're willing to try it I can guide you to the changes that could make this work.

I am hacking together something short term for numba-xnd, so unless I can't figure out a way I will likely be able to work around this. When you are back, and if you are interested, we could think about ways to expose this logic so it could be helpful in Numba.

Moreover, putting this into llvmlite might also be possible, but I would require including clang within llvmlite, something that would make this package not very light anymore ;)

Ah, I didn't think about this... Well I think if we used dragonffi to produce a standalone LLVM module that has these wrappers, then we could link to that in llvmlite and use them.

[saulshanabrook]

FYI I missed this, but there is already a PR on numba that manually generates c function calls for amd64 ABI: https://github.com/numba/numba/pull/3134/

[aguinet]

Okay interesting! They are basically reimolementing the clang part that handle the x86 abi.

About structures? the wqy i do it in dragonffi is getting the layout from debug informations (generated by clang in llvm ir debug info, it's explained here http://blog.llvm.org/2018/03/dragonffi-ffijit-for-c-language-using.html?m=1). I was thinking about making an offline compiler that would expose this in python (or any other language). It could also be possible to generate llvm wrappers that get/set structure values.

The offline part can be interesting because it would avoid the dragonffi dependency at runtime!

[aguinet]

Thinking about it, a first hack that I can do when coming back from holidays would be t9 generate these getting/setting llvm wrappers for struct fields. These wrappers could (?) be inserted within an llvmlite moduleand used from there. Would that help?

[sklam]

Getting the wrappers in LLVM IR usable w/ llvmlite would be great for many usecases. One potential problem could be mismatching version of the LLVM IR. Llvmlite is using LLVM6 now. DragonFFI is on LLVM5. How dependent is DragonFFI on specific version of LLVM?

[saulshanabrook]

I was thinking about making an offline compiler that would expose this in python (or any other language). It could also be possible to generate llvm wrappers that get/set structure values.

That would be helpful. For my use case, I need to get/set fields and stack allocate C structs. Some of these don't end up being LLVM structs, I guess if they are small the compiler just turns them into pointers.

a first hack that I can do when coming back from holidays would be t9 generate these getting/setting llvm wrappers for struct fields. These wrappers could (?) be inserted within an llvmlite moduleand used from there. Would that help?

Yes, I think that would work... The only things is I think I would have to re-type all the functions in llvmlite/numba that are exposed by that module, which could be some work.

[aguinet]

That would be helpful. For my use case, I need to get/set fields and stack allocate C structs. Some of these don't end up being LLVM structs, I guess if they are small the compiler just turns them into pointers.

The mem2reg optimisation can do that indeed!

Yes, I think that would work... The only things is I think I would have to re-type all the functions in llvmlite/numba that are exposed by that module, which could be some work.

What's the ideal LLVM interface you would like to have?

[aguinet]

Rethinking about it, dragonffi already exposes everything needed to write small wrappers to get/set struct values using llvmlite. I also merged today a cleanup of the python API to get memory views of C objects. I wil ltry and make a POC with all of this, and will keep you in touch!

[aguinet]

Here is an example: https://gist.github.com/aguinet/5d359476d77ff69a7daafa51ad005ea8

[aguinet]

I think the dragonffi dependency could be removed by using an offline generator of Python structs that map to the C one (with the various offset, sizes and types of each fields). Easier said than done thought :)

[aguinet]

@sklam sorry I didn't see your question. So:

Getting the wrappers in LLVM IR usable w/ llvmlite would be great for many usecases. One potential problem could be mismatching version of the LLVM IR. Llvmlite is using LLVM6 now. DragonFFI is on LLVM5. How dependent is DragonFFI on specific version of LLVM?

I am currently using a custom version of Clang/LLVM 5 with two small patches to be able to correctly handle calling convetions. I am planning of migrating dragonffi to LLVM 7 when it will be released (current state is in release candidate IIRC). I "missed" the LLVM 6 release because the aformentionned patches didn't make it to this version.

Will llvmlite plan to migrate to LLVM7?

[saulshanabrook]

@aguinet that gist looks great! I am gonna try it out later today. So cool, thank you :)

[aguinet]

Okay, let me know if that helped you!

FTR, I've just merged an API to get the LLVM IR (in textual form) of function wrappers.

An example is provided here: https://github.com/aguinet/dragonffi/blob/master/bindings/python/tests/wrapper.py .

Even if llvmlite is using LLVM6, it could be used to make a POC to see if it can be "injected" within numba/llvmlite.

I've tagged all of this with version 0.3.0. I'll make a release in pip during the week, so that you guys will be able to play with this w/o compiling dragonffi.

[saulshanabrook]

I am trying to compile locally to test out the examples you posted. I got to this step when compiling LLVM5:

cd llvm-5.0.1.src/tools/clang && patch -p1 </path/to/dragonffi/third-party/cc-clang.patch

but clang seems to be a file and not a directory:

cd llvm-5.0.1.src/tools/clang                                                                                              (numba-xnd)
cd: The directory 'llvm-5.0.1.src/tools/clang' does not exist
ls llvm-5.0.1.src/tools/                                                                                                   (numba-xnd)
CMakeLists.txt                   llvm-config                      llvm-mc-assemble-fuzzer          llvm-strings
LLVMBuild.txt                    llvm-cov                         llvm-mc-disassemble-fuzzer       llvm-symbolizer
bugpoint                         llvm-cvtres                      llvm-mcmarkup                    llvm-vtabledump
bugpoint-passes                  llvm-cxxdump                     llvm-modextract                  llvm-xray
clang                            llvm-cxxfilt                     llvm-mt                          lto
dsymutil                         llvm-diff                        llvm-nm                          msbuild
gold                             llvm-dis                         llvm-objdump                     obj2yaml
llc                              llvm-dwarfdump                   llvm-opt-report                  opt
lli                              llvm-dwp                         llvm-pdbutil                     opt-viewer
llvm-ar                          llvm-extract                     llvm-profdata                    sancov
llvm-as                          llvm-go                          llvm-readobj                     sanstats
llvm-as-fuzzer                   llvm-jitlistener                 llvm-rtdyld                      verify-uselistorder
llvm-as-parasitic-coverage-repro llvm-link                        llvm-shlib                       xcode-toolchain
llvm-bcanalyzer                  llvm-lto                         llvm-size                        yaml2obj
llvm-c-test                      llvm-lto2                        llvm-split
llvm-cat                         llvm-mc                          llvm-stress

Any ideas?

EDIT: Never mind, it is a symlink. Not sure why cd complained about it.

[aguinet]

That's because my instructions doesn't work out of the box... The symbolic link is invalid. You should set llvm/tools/clang as a symlink to the extracted cfe-5.0.1.src directory.

[saulshanabrook]

Yeah that's what the symbolic link was set to, so they did seem to work! I just cded into that directory and am compiling now (45%), so will let you know when it finishes if it all works.

[aguinet]

Okay, you are experimenting the joy of compiling LLVM :) I fixed the README to have something working out-of-the-box.

[aguinet]

@saulshanabrook @sklam : dragonffi 0.3.0 has been uploaded on pip for linux and osx. You can install it using pip install pydffi.

[sklam]

Will llvmlite plan to migrate to LLVM7?

Yes, llvmlite tends to track closely to LLVM releases.

[aguinet]

@saulshanabrook did you manage to do what you wanted to do :) ?

[saulshanabrook]

@aguinet I have been caught up trying to figure out memory management so I haven't had a chance yet to try swapping this in, but it's next on my list :)

[aguinet]

For info @sklam , I rebased dragonffi on llvm7 in the current master branch. A release might appear at the end of the week!

[sklam]

Nice, llvmlite will be releasing at the end of week, too.