experiments by scorpion81 with cffi

[zeffii]

from a convo in #blendercoders

<scorpion81> https://cdn.discordapp.com/attachments/359792703917129729/524307060553416704/cffiTest1.blend thats a test blend and... you need to install cffi inside 2.79/python for it, then also copy over the Python includes to 2.79/python/include...
<scorpion81> another experiment of mine... https://blenderartists.org/t/addon-script-fake-python-based-modifier-stack-experimental/699451/4 a python modifier stack, well ... its a bit of "fake" modifier functionality
<scorpion81> zeffii: btw in blenders python bin dir you can do ./python -m ensurepip to setup a local pip environment and then just ./pip install cffi
<scorpion81> or maybe python3 / pip3, rather

interesting :)

[zeffii]

and this brings up a topic I want to explore in 2.8, writing some routines in a faster language and calling them from python/sverchok. and providing examples for devs to learn from. no ETA.

[scorpion81]

Hey, here a quick and dirty test implementation of cffi based functions via a sverchok scripted node. See the Introduction.txt file inside the zipped blend file for some hints how to use it. Its still very experimental and works with simple stuff only, if at all, lol :)

Looks like this if it works correctly...

sverchok_cffi.zip

[kalwalt]

Very interesting @zeffii @scorpion81 ! Let's do it!

[zeffii]

ok, that uses a version of script node which no longer exists in Sv for b280, i'll see if snlite (its replacement) will do it on win64 :)

[zeffii]

on w64

1. ./python -m ensurepip
2. \2.80\python\lib\site-packages> python pip install cffi

[zeffii]

ok nope. installing packages is the most painful/timeconsuming stuff in python :)

[scorpion81]

./python -m ensurepip should have installed pip and thus there should be a pip.exe or pip3.exe in the \2.80\python\bin dir now (didnt test on windows tho, yet, only linux) and if that exe is there, in \2.80\python\bin just run ./pip install cffi will test in windows vm now

[scorpion81]

C:\BlenderDEV\build_new\bin\Release\2.80\python\bin> .\python.exe -m ensurepip

Collecting setuptools Collecting pip Installing collected packages: setuptools, pip Successfully installed pip-10.0.1 setuptools-39.0.1

C:\BlenderDEV\build_new\bin\Release\2.80\python\Scripts>.\pip3.exe install cffi

Collecting cffi Downloading https://files.pythonhosted.org/packages/ca/f2/e375b7469a2dfe9d1feac81a10df97f18cd771b9a10ac62ca9864b760f7c/cffi-1.11.5-cp37-cp37m-win_amd64.whl (165kB) 100% |████████████████████████████████| 174kB 1.6MB/s Collecting pycparser (from cffi) Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB) 100% |████████████████████████████████| 163kB 3.3MB/s Installing collected packages: pycparser, cffi Running setup.py install for pycparser ... done Successfully installed cffi-1.11.5 pycparser-2.19 You are using pip version 10.0.1, however version 18.1 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Oddly, the pip3.exe is under python\Scripts instead of python\bin... hmm maybe this is different under windows or has changed since python 3.5

Furthermore, you need to copy all of those (under windows) from

https://svn.blender.org/svnroot/bf-blender/trunk/lib/win64_vc14/python/include/Python3.7/

to

2.80\python\include\Python3.7

under windows there is no such directory, create it :)
under linux there is

2.80/python/include/python3.7m

but only one lonely header file is in there, lol :)

The includes are necessary under linux, too in order to have cffi working (especially, Python.h needs to be there)

And for python 3.6 or 3.7 you need MSVC++ 2017, for python 3.5 like with 2.79 you would need MSVC++ 2015. I cannot test here because I only have 2.79 under windows but also MSVC++ 2017 (but cffi wants with python 3.5 MSVC++ 2015.) I also couldnt get sverchok to work with 2.8 (special branch needed ?)

And don't forget, in the compile_wrapper.py script you will need to adapt some hardcoded paths possibly. The path to the "build" directory where cffi puts its intermediate files and dlls as well as the path to the python includes are mandatory. The paths to the blender source and a couple of includes are optional, as long as you dont plan to call C functions from blender. Under windows this might be more problematic too anyway, because afaik not all symbols are exposed thru the blender.exe (iirc under linux this is different)

[zeffii]

i'll follow these instructions as soon as I get back to a decent computer!

[zeffii]

@scorpion81

I also couldnt get sverchok to work with 2.8 (special branch needed ?)

yes, the branch is called "b28_prelease_master"

got as far as filling the content of /include/Python3.7

[scorpion81]

@zeffii how to add a StringProperty to the ScriptLite node (from a script) you have there an example with enum, but i would like to have something like a second script picker for the C Source file (prop_search with a string property over the bpy.data.texts) this would save the extra "loader" node... i did this because it was not possible to add Properties there in script node

[zeffii]

i added only one fileselector "bound" StringProperty in SnLite, the example you might have seen: https://github.com/nortikin/sverchok/issues/942 .. i think it's in there somewhere, i'll have more time later to respond.

I finally downloaded all of that win64 stuff, to get the Python3.7 folder :) took like an hour

Instead of picking a file from the UI, i would edit the text/source loaded by SNLite and manually enter the name (for now) and hit Ctrl+Enter in the TextEditor to trigger an update in the node tree.

[zeffii]

with one fileselector ( i think you might have found this one already ) https://github.com/nortikin/sverchok/pull/1309

i'm quite happy to add multiple 'dummy' StringProperties as standard in an SNLite node. will do ASAP.

[zeffii]

but... as I recently found out, StringProperty<-prop_search(from bpy.data.texts) is going to produce weird results in 2.79 onwards -- to the point where it makes more sense to just manually bind the filenames from within the 'worker' script. (and trigger an update from Ctrl+Enter while in TextEditor)

[scorpion81]

@zeffii

Works great with the fileselector / fh method as you described before :)

sverchok_cffi_28.zip

[zeffii]

note the 3 characters in front of the filename returned by the prop_search. This is a different issue :)

that's cool stuff btw.

[kalwalt]

i tried under Ubuntu 18.04 an blender 2.79 installled pip, cffi, and any other inlcudes (Python.h...) , libffi-dev package as suggested here but i receive this error:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/walter/blender-2.79/2.79/scripts/addons_contrib/sverchok/core/update_system.py", line 325, in do_update_general
    node.process()
  File "/home/walter/blender-2.79/2.79/scripts/addons_contrib/sverchok/nodes/generators_extended/script.py", line 474, in process
    self.set_outputs(node_function, fparams)
  File "/home/walter/blender-2.79/2.79/scripts/addons_contrib/sverchok/nodes/generators_extended/script.py", line 510, in set_outputs
    out = node_function(*fparams)
  File "<string>", line 126, in sv_main
  File "<string>", line 113, in compile
  File "/home/walter/blender-2.79/2.79/python/lib/python3.5/site-packages/cffi/api.py", line 451, in verify
    lib = self.verifier.load_library()
  File "/home/walter/blender-2.79/2.79/python/lib/python3.5/site-packages/cffi/verifier.py", line 104, in load_library
    self._compile_module()
  File "/home/walter/blender-2.79/2.79/python/lib/python3.5/site-packages/cffi/verifier.py", line 201, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "/home/walter/blender-2.79/2.79/python/lib/python3.5/site-packages/cffi/ffiplatform.py", line 22, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
  File "/home/walter/blender-2.79/2.79/python/lib/python3.5/site-packages/cffi/ffiplatform.py", line 58, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: LinkError: command 'gcc' failed with exit status 1

i have also modified the path for the includes and cffi build.

[scorpion81]

hmm i can reproduce this here with official blender 2.79. It is working in my self-compiled 2.79 blender version though... the top of the error message is like:

/usr/bin/ld: Linker-Skriptdatei python.map kann nicht geöffnet werden: Datei oder Verzeichnis nicht gefunden collect2: error: ld returned 1 exit status

the linker cannot find a file called "python.map" My suspicion is that cffi / distutils will pick up the same build configuration for extensions it builds under which python for blender was built. This is located in

/2.79/python/lib/python3.5/_sysconfigdata.py

and that file differs between official builds and self-compiled ones. The offending env variable inside that seems to be:

(official)

'BLDSHARED': 'gcc -pthread -shared -L/opt/lib/bz2/lib ' '-L/opt/lib/lzma/lib -L/opt/lib/ssl/lib ' '-L/opt/lib/zlib/lib -L/opt/lib/libc6/lib ' '-L/opt/lib/ncurses/lib -L/opt/lib/gpm/lib ' '-L/opt/lib/sqlite/lib -L/opt/lib/readline/lib ' '-Wl,--version-script=python.map',

(my build)

'BLDSHARED': 'gcc -pthread -shared',

But... i tried to modify this file, deleting various "pycache" directories under cffi (in site-packages), distutils etc. but it still doesnt work for official builds. I suspect it must be cached somewhere else too, but i need to investigate where exactly. I also tried to add that python.map file to the include dir for example (it is part of the blender sources under

/build_files/build_environment/cmake

but that didnt work as well... I need to investigate further how to fix this.

As workaround you may use a self-compiled blender build if possible, for the time being.

[kalwalt]

Thank you @scorpion81, so it seems like a 2.79 isuue? Do you tried 2.80 under Windows not linux. right? if the 2.80 is less problematic in your opinion i will switch to this version....

[scorpion81]

no, i also have a self-compiled linux 2.80 version here, i think it is more related to official vs self-compiled. I think something may be wrong with the official python build configuration, then. hmmm

[kalwalt]

you compiled from source code because this issue (for linux)? and windows via pip?

[scorpion81]

Nevermind, i think I found the culprit....

/2.79/python/lib/python3.5/config-3.5m/Makefile

Line 86 reads like:

CONFIGURE_LDFLAGS= -L/opt/lib/bz2/lib -L/opt/lib/lzma/lib -L/opt/lib/ssl/lib -L/opt/lib/zlib/lib -L/opt/lib/libc6/lib -L/opt/lib/ncurses/lib -L/opt/lib/gpm/lib -L/opt/lib/sqlite/lib -L/opt/lib/readline/lib -Wl,--version-script=python.map

but should read :

CONFIGURE_LDFLAGS= -L/opt/lib/bz2/lib -L/opt/lib/lzma/lib -L/opt/lib/ssl/lib -L/opt/lib/zlib/lib -L/opt/lib/libc6/lib -L/opt/lib/ncurses/lib -L/opt/lib/gpm/lib -L/opt/lib/sqlite/lib -L/opt/lib/readline/lib

Seems cffi picked this configuration up, also.

I do compile blender on my own because i also develop a bit in the C Code of blender, the Fracture Modifier branch :)

[zeffii]

for win64, a recap.

will i need msvc'17 and '15 if using the Python3.7 + cffi supplied in the builder.blender.org nightly builds, ?

[scorpion81]

No, msvc 15 is only necessary if you want to use 2.79b which still uses python 3.5. For nightly builds of blender you need msvc 17, since those use python 3.7.

Official python 3.5 was compiled by msvc 15 and official python 3.7 by msvc17, i say official here because if you compile blender on your own, that might depend how python was built under windows.

That means it COULD be that python 3.5 for example was built with msvc 17 too. For blender, the windows dependencies are even the same for msvc15 and 17 (win64_vc14).

But it seems also here cffi tries to match with offical python's compiler environment, i guess.

So bottom line, for 2.8 nightlies with python 3.7 you need msvc 17.

[zeffii]

cool. then that's what my mission is today :)

[scorpion81]

Hmm, just tested under win 10 with 2.8 the pathes need to be like on the screenshots (aka python/include vs python/include/Python3.7 and additionally you need python/libs/python37.lib) Weird, but i checked an official python installation and it is like that there, too. Was Python27 in this case, but seems that folder structure still is valid.

[zeffii]

still installing msvc 2017 (community edition i hope is sufficient?.. else i'm shafted)

[scorpion81]

yup community edition is ok, i have that too You need the "Desktop Development with C++" package there, basically... maybe win 10 sdk and python development package too

[zeffii]

those things better be easy to install afterwards, if I didn't tick the right boxes initially

[zeffii]

[zeffii]

the point of this experiment is to see if heavier mesh functions can be made faster by writing the code in C/C++. some candidates are (that can omit dependency on bmesh)

• intersect edges
• inset special
• 3pt arc
• 2pt spline
• smooth lines

While I don't expect this kind of stuff to make it into 2.80 any time soon, it might provide some practice and insight into writing native functions directly for blender. but I am getting ahead of myself here

https://www.cs.rit.edu/~ats/books/ooc.pdf ("objects" in C)

[scorpion81]

This is a more complex experiment with sources of a C++ library and a c interface (voro++, the source files are in the zip file too, might need to adapt the path in the c_voro3d.py file) Furthermore inside the blend there are modified(!) snlite and snlite importhelper files from sverchok. I added an "external_socket" function and an additional parameter to parse_socket, basically. and i changed the update function for the user_filename property (it will update the sockets too now) You need to use the compile_wrapper2 now as script and if its loaded, pick c_voro3d.py as source file for compilation.

The problem is the output sockets dont seem to be populated correctly with data.

sverchok_cffi_29.zip

[scorpion81]

A dedicated CFFI node for sverchok. c_voro3d.py in an example py/C script, and you still need the additional sources from the voro folder inside the zip to compile it.

sverchok_cffi_30.zip

[scorpion81]

Looks like breaking bboxes / cubes works again :)

sverchok_cffi_31.zip

[zeffii]

except for the initial compilation step (about 30 seconds on my slow ass machine) this is really cool :)

[scorpion81]

yup, initial compilation takes a while here too, since the whole voro++ library is built, but with cffi's verify() method the binary will be re-used until the c sources are actually modified. Basically this doesnt happen much because a python method just calling into the c interface methods does the work. I added a margin socket to be able to move the cells apart :)

sverchok_cffi_32.zip

[kalwalt]

This is cool! i have worked with voro++ a bit in the past with Openframeworks was easy to use the library, i think would be a nice addition for sverchok addon. @zeffii I tried with blender 2,80 and the latest of 2.80_prelease_master branch but none of the nodes are displayed maybe i have to switch to a previuos commit?

[scorpion81]

@kalwalt

hmm in the blend there is a cffi1.py (thats the node file). You can put it in sverchoks nodes/generators directory (where the snlite node is) and add the classname "SvCFFINode" into the index.md file in sverchoks root directory (maybe in the "Generators" section too, i put it after the according snlite class name.)

Then, if you reload sverchok (or restart blender) the node should be available. Oh, and dont forget to put the "voro" source folder from the zip file to where that blend is (the path config inside assumes the voro dir is at that location).

If not already happened, load the "c_voro3d.py" file into the CFFI node, then compilation should start.

First compilation of the voro++ lib can take a short while, but then unless you dont remove the resulting binaries from the "build" folder or modify the c sources, it should not be recompiled again.

If not all input sockets are connected yet, some python errors may pop up, but should disappear when all input sockets actually have input / are connected.

[kalwalt]

@scorpion81 thank you for the explanation, sorry i was not clear about what i tried, i didn't tried the voro++ with blender instead i was referring to the first .zip file, but i will try also your voro++ example.

[zeffii]

@kalwalt be sure to use the latest builder.blender.org of 2.80 , i haven't updated mine for a couple of days. (22 dec) . it's possible that more changes made it not load anymore. but if you haven't updated 2.80 for a couple of days, you need to.

[kalwalt]

@kalwalt be sure to use the latest builder.blender.org of 2.80 , i haven't updated mine for a couple of days. (22 dec) . it's possible that more changes made it not load anymore. but if you haven't updated 2.80 for a couple of days, you need to.

Ah good to know! i have in synchro with you commit changes but not with the latest in blender source, because i'm using an "older" downloaded version, i can confirm that nodes not load anymore. I will do that update, thanks!

[zeffii]

yes, with an older 2.80 you might be getting an error like

Context Restricted no such thing as context.preferences

[zeffii]

btw, with that ...31 .blend zip all i had to do was

• load the blend,

• uncomment the single #register() at the end of the bpy.data.texts text file that contains the CFFI node

• and press alt+P to run that file. (which then registers it) .

this should replace the 'undefined' node with the proper CFFI definition.

at this point make sure the console is open / terminal so you can see the compilation process

then click on one of the input sockets, to trigger a node update. Then the compilation step will take some time..eventally it will say compiled.. and you should see results.

[kalwalt]

I am in the CFFI club!!!! ok, it seems that is simpler with blender 2.80, i will continue later to test. Happy that it works! Edit: this is sverchok_cffi_28.zip

[scorpion81]

Ok, carve boolean seems to work (under linux)... For this i pre-compiled the carve lib with CMake / make to get a libcarve.a against which the cffi wrapper is linked. Under Windows, you can also precompile it with CMake gui and MSVC17 Then in c_carve.py, adapt maybe the paths and replace -L with /LIBPATH:. Under windows you will have a "carve.lib" file then to link against.

Edit, to build the carve library, you need the boost library and includes from the blender sources. And you need to tell CMake the path to the boost includes with BOOST_INCLUDE_DIR when building carve. Both under linux and windows.

sverchok_cffi_34.zip

[zeffii]

that's mental @scorpion81 / it really is.

i'm still getting my head around writing C again :) https://gist.github.com/zeffii/f922edf25da663fea6a823377dbf61bc slowly converting that function from python to what i think is C.

[scorpion81]

Voro++ and Carve... together :) this time i built carve and the cffi wrapper in release mode, is a bit faster. (only the linux libcarve.a is included atm)

sverchok_cffi_35.zip

[scorpion81]

Same stuff as above, but slightly reworked (linker paths are set now more generally, not via compiler specific switches, and the carve.lib for windows is now included :) ) Should work on windows now without need to compile carve first :)

sverchok_cffi_36.zip

[zeffii]

that looks so cool!

unfortunately the Carve part seems to exit blender during the compilation step (without errors that I can see easily...)

[scorpion81]

@zeffii did you use the sources (carve and voro folders) from sverchok_cffi_36.zip ? because there are some modifications inside (especially carve) which are important... Did you also check the paths ? in setup() routine of c_carve.py and c_voro3d.py ?

Edit: i also did NOT manage to compile carve "live" as i did with voro++, there is some linker error the msvc compiler chokes with... precompilation with cmake / msvc works for some reason lol... but you have the carve.lib file in place (if you put the carve and voro folders next to the sverchok_cffi_36.blend) so you should not need to compile the entire carve. and the c_carve.py script is adapted to link against the carve.lib now which is expected in the "dist" directory underneath the carve directory.

hmm, maybe you need to run blender under a debugger... under linux that simply worked with gdb ./blender and when it was about to crash inside the wrapper, gdb caught that and showed me the line of the wrapper source even where it crashed (because in this case i passed compiler switches via cffi to make a debug version).

Under windows this unfortunately is rather tricky to set up. You could try to follow this guide here... http://www.lazydodo.com/blender/pydebug/#introduction (additionally make sure to install "Native Python Development tools" in the Python workload section of the MSVC installer). But since I had it in a slow VM and didnt want to wait forever lol.... i stopped experimenting there because I didnt get the debugger to show me the source lines of the wrapper code when it caught the crash. Probably some debug symbols missed or something...