experiments by scorpion81 with cffi

[zeffii]

from a convo in #blendercoders

<scorpion81> https://cdn.discordapp.com/attachments/359792703917129729/524307060553416704/cffiTest1.blend thats a test blend and... you need to install cffi inside 2.79/python for it, then also copy over the Python includes to 2.79/python/include...
<scorpion81> another experiment of mine... https://blenderartists.org/t/addon-script-fake-python-based-modifier-stack-experimental/699451/4 a python modifier stack, well ... its a bit of "fake" modifier functionality
<scorpion81> zeffii: btw in blenders python bin dir you can do ./python -m ensurepip to setup a local pip environment and then just ./pip install cffi
<scorpion81> or maybe python3 / pip3, rather

interesting :)

[zeffii]

carve.lib now which is expected in the "dist" directory underneath the carve directory.

nope, i had not done this. sounds hopeful.

[scorpion81]

@zeffii aaah, you also need to update the cffi1.py (the CFFI Node) file... because i made some changes to it as well

[zeffii]

carve.lib is located in sverchok_cffi_36\carve\dist

[scorpion81]

then the path in c_carve.py must be adapted (lib_dirs) i think, hmm

[zeffii]

ooh! cool. one sec

[scorpion81]

lol its more efficient to chat in IRC :) and less clutter here

[zeffii]

seems to point to the right place.
you're right, irc it is.

[zeffii]

oh btw, this slashes business

def setup():
    import os, bpy
    path = os.path.dirname(bpy.data.filepath)
    py = bpy.app.binary_path + "../2.80/python/include"
    build = path + "/build"
    srcs = path + "/carve/"
    incs = [path + "/carve"] # path + "/carve/include"]
    libs = ['carve']
    lib_dirs = [path+"/carve/dist"]
    print(lib_dirs)

setup()

prints:

['C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_36/carve/dist']

[zeffii]

are you familiar with https://docs.python.org/3/library/pathlib.html?highlight=path#module-pathlib

[scorpion81]

hmm looks still too complicated... aint there some path() function which autoconverts the slashes to backslashes or something if it runs on windows ?

[zeffii]

yep , os.path.join(existing_str, new, new, new)

[scorpion81]

Hopefully the path issues are fixed now (for windows too) ... :) sverchok_cffi_38.zip

[scorpion81]

Fix for a stupid mistake, was accessing a not-yet initialized data structure (export_data->loops). I assumed the access callback would be called after the init callback, but seems it served another purpose in this case, mapping to original index etc. This causes crashes under windows while gcc under linux seemed to zeroize stuff or something there automatically, just a guess.

sverchok_cffi_39.zip

[scorpion81]

Added partial example code for implementation of some math functions.

sverchok_cffi_41.zip

[zeffii]

parking this here for later.. https://github.com/sobotka/blender/blob/c9df453ce70bdf36a4567f8a97b977c1f5e6dfd7/source/blender/python/mathutils/mathutils_Matrix.c#L532

https://github.com/sobotka/blender/blob/02f083cdcecf74a169114850337a99c802c53e34/source/blender/blenlib/intern/math_rotation.c#L1014

[scorpion81]

Hmm, got "something" now with the math algos, but not sure if i got the main algorithm right. I just re-used math functions from the blender source files, so for this you need parts of blenders source code. Just replace the "blenroot" path in the compile script with your own root folder of the blender sources, then it should work.

sverchok_cffi_42.zip

[zeffii]

the main algorithm, is described in python here : https://github.com/nortikin/sverchok/blob/master/nodes/generator/basic_3pt_arc.py

[scorpion81]

Looks like this now, guess this is an arc lol

sverchok_cffi_43.zip

[zeffii]

yep, it should use the 3 points provided to return an arc that starts at point 1, ends at point 3 and touches point 2. will try ASAP :))))

[zeffii]

huh >? hahah

  File "<string>", line 146
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape
2019-01-03 17:12:03,128 [ERROR] sverchok.core.update_system: Node CFFI Node.003 had exception: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape (<string>, line 146)
Traceback (most recent call last):
  File "C:\Users\zeffi\Desktop\scripts\addons_contrib\sverchok\core\update_system.py", line 325, in do_update_general
    node.process()
  File "C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_43\sverchok_cffi_43.blend\cffi1.py", line 177, in process
  File "C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_43\sverchok_cffi_43.blend\cffi1.py", line 155, in execute
  File "C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_43\sverchok_cffi_43.blend\cffi1.py", line 119, in load
  File "<string>", line 146
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape

[scorpion81]

hmm had some odd unicode crap too when running blender / sverchok / cffi node in debug mode... the labels were UUUUUUU something, probably thats some windows bs here... or something with strings hmmmmm... IRC ?

Hmmm https://stackoverflow.com/questions/1347791/unicode-error-unicodeescape-codec-cant-decode-bytes-cannot-open-text-file -> C:\Users... \Us is being treated as Unicode sequence wtf... probably you cannot use that in a string lol... but rather C:\\Users or something ..or raw string like r"my_string"

[scorpion81]

Multi Arc :) but for some reason -2, 0, 0; 0, -1, 0; 2, 0, 0 doesnt work right.. have to use -1.15 else its kinda tilted... hmmm

sverchok_cffi_44.zip

[zeffii]

I made a less elobrate version of that .blend .but it gives errors.

_cffi__xd8fb76x73562afb.obj : error LNK2001: unresolved external symbol interp_v3_v3v3
_cffi__xd8fb76x73562afb.obj : error LNK2001: unresolved external symbol angle_v3v3
math_geom.obj : error LNK2001: unresolved external symbol MEM_callocN
math_geom.obj : error LNK2001: unresolved external symbol project_plane_normalized_v3_v3v3
math_geom.obj : error LNK2001: unresolved external symbol project_v3_v3v3
math_geom.obj : error LNK2001: unresolved external symbol MEM_freeN
math_geom.obj : error LNK2001: unresolved external symbol interp_v3_v3v3v3
math_geom.obj : error LNK2001: unresolved external symbol interp_v2_v2v2
math_geom.obj : error LNK2001: unresolved external symbol minmax_v3v3_v3
math_geom.obj : error LNK2001: unresolved external symbol ortho_basis_v3v3_v3
math_geom.obj : error LNK2001: unresolved external symbol angle_normalized_v3v3
math_geom.obj : error LNK2001: unresolved external symbol project_plane_v3_v3v3
math_geom.obj : error LNK2001: unresolved external symbol rotate_normalized_v3_v3v3fl
math_interp.obj : error LNK2001: unresolved external symbol copy_vn_fl
math_interp.obj : error LNK2001: unresolved external symbol copy_vn_uchar
math_matrix.obj : error LNK2001: unresolved external symbol BLI_svd_m3
math_matrix.obj : error LNK2001: unresolved external symbol EIG_invert_m4_m4
math_rotation.obj : error LNK2001: unresolved external symbol ortho_v3_v3
C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_43\build\_cffi__xd8fb76x73562afb.cp37-win_amd64.pyd : fatal error LNK1120: 18 unresolved externals

with mods to c_math.py

def setup():
    import os, bpy
    from pathlib import Path

    root = Path(bpy.data.filepath).parent
    py = Path(bpy.app.binary_path).parent / "2.80" / "python" / "include"
    build = Path(root) / "build"
    blenroot = Path(r"C:\blender-git\blender") #replace with own src root !
    srcroot = blenroot / "source" / "blender" / "blenlib" / "intern"
    incroot = blenroot / "source" / "blender" / "blenlib"
    incmem = blenroot / "intern" / "guardedalloc"
    inceigen = blenroot / "intern" / "eigen"   # else eigen_capi.h  is not found
    srcs = str(srcroot)
    incs = [str(incroot), str(incmem), str(inceigen)]   #  uhmm...
    libs = []
    lib_dirs = []
    comp = []
    link = []

    return str(py), str(build), srcs, incs, libs, lib_dirs, comp, link

i also removed all but the listinput/num node and the CFFI that loads C_Math and the two output nodes (stethoscope and VD )

[zeffii]

seems like the CFFI c_math.py node is relying on something that is present only if the other CFFI nodes have compiled something?

[zeffii]

assuming this will be worked out eventually, i want to start formalizing (even just for a few beta nodes) how the node's upper section would work. For a straight forward implementation of a function that can return values based on the input params alone.

The input and output signature of that function would be the same whether it's implemented in python or not.

from sverchok.utils.modules.some_dedicated_submodule import arc_from_3pt

maybe sverchok can already add a set of control flow statements to the "some_dedicated_submodule", so that it returns the compiled version if there is one, and the python one until the user has compiled it. (or can't compile is, because CFFI isn't installed)

[zeffii]

or maybe it's simpler to really have unique nodes that could be hot-swapped if the faster version is available.. NodeNameFast vs NodeName

[scorpion81]

Finally... seems i got the algorithm right now... it made some headache with the angle calculation etc.

sverchok_cffi_45.zip

[scorpion81]

@zeffii hmm maybe i should have cleaned up my sverchok_cffi_xxx.blend... it still includes voro and carve nodes... and odd @ window-isms again... grrrrrr... why cant that crap just work out of the box without extra sausage ? lol.... sigh... maybe one has to make a "math.lib" or something again for stupid windoze...

[scorpion81]

So, now i extracted all the relevant function code from blenders sources and put it into one source file, mathlib.c (and mathlib.h). I also updated the cffi1.py once again and all the buildscripts (exposed a "language" = c or c++ option) Furthermore i made the c_arc.py return multiple sets of verts and edges (one per arc). Didnt test under windows, but i hope since the compilation setup is simpler now, it might work. I also had trouble under windows, getting the code directly from the blender sources into the dll. (linker errors)

sverchok_cffi_46.zip

[zeffii]

this rocks :)

[zeffii]

i like separating the source code into it;s own file like this

import bpy
code = bpy.data.texts['arc_code.c'].as_string()

[zeffii]

something like this..

# This file is part of project Sverchok. It's copyrighted by the contributors
# recorded in the version control history of the file, available from
# its original location https://github.com/nortikin/sverchok/commit/master
#  
# SPDX-License-Identifier: GPL3
# License-Filename: LICENSE

import bpy
from bpy.props import StringProperty, IntVectorProperty, FloatVectorProperty, BoolProperty

from sverchok.utils.snlite_importhelper import set_autocolor
from sverchok.node_tree import SverchCustomTreeNode
from sverchok.data_structure import updateNode
from sverchok.sockets import type_map_to

FAIL_COLOR = (0.8, 0.1, 0.1)
READY_COLOR = (0, 0.8, 0.95)

class SvCFFINodeCallBack(bpy.types.Operator):

    bl_idname = "node.cffi_ui_callback"
    bl_label = "CFFI Node callback"
    fn_name: bpy.props.StringProperty(default='')

    def execute(self, context):
        getattr(context.node, self.fn_name)()
        return {'FINISHED'}

class SvCFFINode(bpy.types.Node, SverchCustomTreeNode):
    """
    Triggers: CFFI
    Tooltip: Node exposing C Foreign Function Interface

    """

    bl_idname = 'SvCFFINode'
    bl_label = 'CFFI Node'
    bl_icon = 'SCRIPTPLUGINS'

    script_name: StringProperty()
    script_str: StringProperty()
    n_id: StringProperty(default='')

    def draw_label(self):
        return f"CFFI: {self.script_name or self.bl_label}"

    def sv_init(self, context):
        self.use_custom_color = False

    def clear_sockets(self):
        self.inputs.clear()
        self.outputs.clear()        

    def check_socket(self, direction, name, type):
        socket_type = type_map_to.get(type)
        if not socket_type:
            print('{type} is an unknown shortname for a socket')
            return

        socket_io = getattr(self, f"{direction}puts")
        if name not in socket_io:
            socket_io.new(socket_type, name)

    def make_sockets(self, mdl):
        sock_str = getattr(mdl, "sockets", None)

        if not sock_str:
            print(f'failed to find a sockets comment in {self.script_name}')
            return

        for L in [line.strip() for line in sock_str.split('\n')]:

            if L.startswith(('in ', 'out ')):
                socket_info = L.split(' ')
                if len(socket_info) != 3:
                    continue

                self.check_socket(*socket_info)

    def load(self, name):
        import imp
        if name in bpy.data.texts:
            self.script_str = bpy.data.texts[name].as_string()
            mdl = imp.new_module(name)
            #warning, dangerous !
            exec(self.script_str, mdl.__dict__)
            return mdl
        else:
            return None

    def compile(self, mdl):
        import cffi, os 
        py, build, srcs, incs, libs, lib_dirs, comp_arg, link_arg, lang = mdl.setup()
        ffi = cffi.FFI()
        ffi.cdef(mdl.cdef)

        spaths = []
        for s in mdl.sources:
            spaths.append(os.path.join(srcs, s));

        inc_dirs = [py, srcs]
        inc_dirs.extend(incs)

        #ffi.set_source(lib, code, sources=spaths, include_dirs=[py, src], language='c++')
        #ffi.compile(verbose=True, tmpdir=build)
        #verify automatically caches the binary, so it wont recompile over and over again
        lib = ffi.verify(mdl.code, 
                        tmpdir=build, 
                        sources=spaths, 
                        include_dirs=inc_dirs, 
                        libraries=libs,
                        library_dirs=lib_dirs, 
                        extra_compile_args=comp_arg,
                        extra_link_args=link_arg,
                        language=lang) # to be able to compile c++ too

        return lib, ffi

    def execute(self):

        name = self.script_name
        mdl = self.load(name)
        if mdl is not None:
            lib, ffi = self.compile(mdl)

            #create sockets if not existing
            self.make_sockets(mdl)

            # execute glue / bind code from c script
            # pass current node there, so we can access the socket variables
            mdl.execute(lib, ffi, self)

            #for experimental reason
            #print("SHIT3")
            return lib, ffi, mdl

    def clear(self):
        self.script_str = ''
        self.script_name = ''
        self.clear_sockets()

    def process(self):
        #try:
            self.execute()
            set_autocolor(self, True, READY_COLOR)
        #except:
        #    set_autocolor(self, True, FAIL_COLOR)

    def draw_buttons(self, context, layout):
        sn_callback = 'node.cffi_ui_callback'

        col = layout.column(align=True)
        row = col.row()

        if not self.script_str:
            row.prop_search(self, 'script_name', bpy.data, 'texts', text='', icon='TEXT')
            row.operator(sn_callback, text='', icon='PLUGIN').fn_name = 'process'
        else:
            row.operator(sn_callback, text='Reload').fn_name = 'process'
            row.operator(sn_callback, text='Clear').fn_name = 'clear'

        #self.custom_draw(context, layout)

classes = [
    SvCFFINodeCallBack,
    SvCFFINode
]

register, unregister = bpy.utils.register_classes_factory(classes)

if __name__ == '__main__':
    register()

[zeffii]

except the socket parser should also allow for setting a slider/string/bool/whatever per socket.

[zeffii]

but.. this is all secondary to what I really want to do... write some C :) so maybe you'll update your version with these changes, but don't feel you need to 'just cuz' .

[scorpion81]

Arc stress test lol... used numpy here to reshape the flat output buffer (in C i still used non-flat arrays) This time only with the arc node and the mathlib folders. cffi1.py is not updated yet in this zip.

sverchok_cffi_48.zip

[zeffii]

interesting. https://stackoverflow.com/a/39243394

i'm still tackling the C version of xall, but might migrate to C++ for access to a dynamic array/Vector? i think the max num of interesections of a list of lines is

((n*n)-n)/2
(n*(n-1))/2

or in matrix form

    a | b | c | d | e | f
__|___|___|___|___|___|___
a | -   x   x   x   x   x
b | -   -   x   x   x   x
c | -   -   -   x   x   x
d | -   -   -   -   x   x
e | -   -   -   -   -   x
f | -   -   -   -   -   -

6*6 = 36
      36 - 6 = 30
               30 / 2 = 15

[zeffii]

@scorpion81 sverchok_cffi_46_b_xall.zip

_cffi__x65bfc13cxe7431942.c
C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_46\build\_cffi__x65bfc13cxe7431942.c(240): error C2108: subscript is not of integral type
C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_46\build\_cffi__x65bfc13cxe7431942.c(241): error C2108: subscript is not of integral type
C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_46\build\_cffi__x65bfc13cxe7431942.c(242): error C2108: subscript is not of integral type
C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_46\build\_cffi__x65bfc13cxe7431942.c(243): error C2108: subscript is not of integral type
C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_46\build\_cffi__x65bfc13cxe7431942.c(255): warning C4013: 'closest_to_line_v3' undefined; assuming extern returning int
C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_46\build\_cffi__x65bfc13cxe7431942.c(255): warning C4244: '=': conversion from 'int' to 'float', possible loss of data
C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_46\build\_cffi__x65bfc13cxe7431942.c(266): error C2106: '=': left operand must be l-value
C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_46\build\_cffi__x65bfc13cxe7431942.c(270): error C2106: '=': left operand must be l-value
C:\Users\zeffi\Desktop\test_blenders\sverchok_cffi_46\build\_cffi__x65bfc13cxe7431942.c(271): error C2106: '=': left operand must be l-value

it seems you must subtract 211 from the line number, the errors start in line 29 on the xall.c i think

hwlp :)

[zeffii]

i just want to return all intersections for now

[zeffii]

i forgot to add

float closest_to_line_v3(float r_close[3], const float p[3], const float l1[3], const float l2[3]);

to mathlib.h

[zeffii]

yikes. some serious noob bugs in variable names...

[zeffii]

sverchok_cffi_46_b_xall.zip

[scorpion81]

some hints:

• you need to make sure to use integer-like array indices only (not float)
• you could pre-allocate the new_verts array (shouldnt this be maximal n* something (the expression you used) or can more verts happen ? but in any case, keep track of the count of new verts
• you cannot assign arrays directly, use copy_v3_v3() to copy over a 3 element chunk
• you also need to copy the code for closest_to_line() etc to mathlib.c, just putting the function prototype into the mathlib.h isnt enough here

[zeffii]

yikes.. and lines are of type int[][2]

[zeffii]

(the issue tracker doesn't refresh automatically) i didn't see your response.

[scorpion81]

lol the issue tracker is no chat :)

[zeffii]

yep, i think most of those points have been addressed now. will zip up new one.

[zeffii]

sverchok_cffi_46_b_xall.zip still struggling with passing / changing the value of

• int num_new_verts (number of new intersections)
• float[][3] new verts , where the size .. is (formula result, or.... formula result * 3 ) ?

[zeffii]

sverchok_cffi_46_b_xall.zip

[zeffii]

i am waay too excited about this: sverchok_cffi_46_d_xall.zip

[zeffii]

while that returns many more intersections than desired, it at the very least isn't crashing on me.