The Scripted Nodes

[zeffii]

I suspect we can have multiple flavours of this kind of node, these are the most attractive kinds of nodes for me, and i'd like to see a less clumsy syntax as the default (something like SN2, but we can use very similar syntax as regular abstract nodes.

The ScriptNode shell would be a socketless entity with at first a button to pick a textblock or pick a local file, then similar to the current SN's have a reload / clear.

Draw Ext could have an auto-updater option for edited files in our favourite text editors..

It's been tricky to get started with this. But I do want to get my hands dirty..

[zeffii]

it's not entirely clear to me when it is appropriate to still write longform nodes, and to what extend they need to be different from old sverchok. I could hax it up, but would like to be consistent with your plans.

[ly29]

Scriped nodes would mostly be an ui issue. Such ui might be better to handcraft. I think nice it would be to craft an interface that imports modules from the blender text files.

[ly29]

I needs method to integrate the custom written nodes in the node menu, also the script nodes would automatically get tagged with the script category, then we have a generator function for displaying the menu for the scripted nodes. Also it would be needed to provide mechanism to remove scripted nodes if/when the module where they are defined is reimported.

[zeffii]

something like this?

import bpy
from bpy.props import StringProperty
from bpy_extras.io_utils import ExportHelper

import numpy as np
from ..svtyping import Number
from ..decorators import node_func

# or maybe register this elsewhere, earlier than nodes.
class SvRx_NodeScriptCB_PRE(bpy.types.Operator):
    bl_idname = "svrx.scriptnode_cb"
    bl_label = "Callback used for svrxNodeScript"
    fn_name = StringProperty()

    def execute(self, context):
        n = context.node
        if hasattr(n, self.fn_name):
            getattr(n, self.fn_name)()
            return {'FINISHED'}
        else:
            return {'CANCELLED'}

def load(self, context):
    n = context.node
    n.script_str = bpy.data.texts[n.script_name]
    try:
        rx_compile(n.script_str)
        # hoist ! 
    except:
        n.is_loaded = False

def clear(self, context):
    n = context.node
    n.script_name = ""
    n.script_str = ""
    n.is_loaded = False
    ...  # disconnect?

def re_load(self, context):
    ...  # reload and reconnect if needed

def handle_sockets(self, context):
    ...

def handle_links(self, context):
    ...

def custom_draw(self, context):
    ...  # script nodes should be allowed to include
    ...  # custom operator buttons, or at least faux-operators
    ...  # by adding a function to 'private_functions'
    pass

def default_draw(self, context):
    layout = self.layout
    n = context.node
    CB = "wm.svrx_scriptnode_cb"

    row = layout.row()
    if n.is_loaded:
        row.operator(CB, text='reload').fn_name = 're_load'
        row.operator(CB, text='clear').fn_name = 'clear'
    else:
        row = col.row(align=True)
        row.prop_search(n, 'script_name', bpy.data, 'texts', icon='FILE', text='')
        row = layout.row()
        row.operator(CB, text='load').fn_name = 'load'

    custom_draw(self, context)

@node_func(
    label="Script Node Pre",
    draw=default_draw,
    custom_draw=custom_draw,
    private_props={
        script_name=(StringProperty, {}),
        script_str=(StringProperty, {}),
        is_loaded={BoolProperty, {}}
    },
    private_functions={'load': load, 'clear': clear, 're_load': re_load},
    user_functions={})
def script_node_pre() -> None:
    return custom_internal()         # <-- not sure

[zeffii]

I think the right place for the templates for SN is in TextEditor, and let the draw_ext of SN inherit the same menu. SN doesn't need to know which template dir it loaded a script from

[ly29]

Yeah, that would work, I was thinking more along the lines to have a script node class that gets included as a base when the module is script, and inckudes appropriate custom properties.

The unconnected script node doesn't need a node function since it will be ignored by the back end.

The operators then handle the logic of reconnecting and clean up.

The script for of nodes would be special import handler in python that matches either real files or python text files to create modules there.

Then if one is happy with a script we provide a simple method to move it to a user script directory where it will be available for the user always. Also perhaps a method to submit it as a gist without having to create fork etc.

Just thinking a bit out loud.

[zeffii]

up and download anonymous gists; https://github.com/zeffii/rawr/tree/master/blender/scripts/addons_contrib/text_editor_gists

i'd be quite happy to add that, just fill in a gistid in the UI and it downloads. would be cool, it's been on my mind too.

[ly29]

Regarding imports, leaving it here as a note. http://chimera.labs.oreilly.com/books/1230000000393/ch10.html#_problem_179

[ly29]

I don't intend to allow direct gist to node, but gist to text to node.

A need to read your code.

[zeffii]

my code just imports to the bpy.data.texts :)

[ly29]

That looks close to perfect for our needs.

[zeffii]

i'll make a utils.gist_upload and utils.gist_download, they can be reused in various systems. https://github.com/Sverchok/SverchokRedux/issues/25

[ly29]

Side note. Reading up about imports I realize all imports should be relative.

[ly29]

Okay, working infrastructure for loading, script nodes, from bpy.data.texts. An import from SverchokRedux.nodes.script is redirected and matched to a blender text file.

The menu is reload but and the node is accessible from there but no other ui infrastructure is in place. https://github.com/Sverchok/SverchokRedux/blob/master/core/loadscript.py#L31-L58

Reloading, re opening file is todo.

>>> SverchokRedux.nodes.script.Text2
<module 'SverchokRedux.nodes.script.Text2' from '<bpy.data.texts[Text2.py]>'>

[ly29]

Also the node_script decorator is needed instead of the node_func now.

[ly29]

I should be working on the backend but this has been bugging for so long that I haven't gotten import from the blender texts working. It neatly solves namespace issues and so on. Script nodes are equal to normal nodes in every respect otherwise.

Now I will play with the backend instead, then back to this.

[ly29]

I imagine that with a similar technique, or maybe just copying the files, a user specified directory could hold nodes that user wants loaded outside of the core distribution and still follow master or different branches. Also that an operator could mode script file to that directory when somebody is happy with a node and wants available as a default.

[ly29]

Some more sketches for script node, operators, base ui class etc.

[ly29]

Basically splitting up what is above into parts that fit into the current architecture. ScriptNode is a pure ui class that all scripts are forced to use. It will contain appropriate drawing code reload etc calls. Scripts must be bpy.data.texts and are loaded as modules.
Scripts are real nodes dynamically loaded so they get access to bpy.props etc and there is a easy step from scripts to nodes. There needs to be some additional operator code for replacing a node upon reload with links etc. Also a concrete script node needs to exist even though load module from text editor and then access through menu works fine. It would be nice to not have to reload the menu, at least script and groups should be dynamically loaded via a yield function.

[ly29]

Script nodes are load automatically on loading a file.

@persistent
def load_handler(scene):
    # load scripts
    for text in bpy.data.texts:
        if "@node_script" in text.as_string():
            loadscript.load_script(text.name)

Scripts are added to the menu automatically.

def script_nodes(context):
    #yield NodeItem("SvRxScriptScriptNode", "Script"), doesn't exist yet
    gen = (nd for nd in node_dict.values() if nd.category == "Script")
    for nd in gen:
        yield NodeItem(nd.cls.bl_idname, nd.cls.bl_label)

### in make_categories
       node_categories.append(SvRxNodeCategory("SVRX_Script", "Script", items=script_nodes))

[ly29]

Looking at the code I continue to be impressed with small amount of code needed to load a module from a custom storage space.

class SvRxFinder(importlib.abc.MetaPathFinder):

    def find_spec(self, fullname, path, target=None):
        if fullname.startswith("SverchokRedux.nodes.script."):
            name = fullname.split(".")[-1]
            name_py = "{}.py".format(name)
            if name in bpy.data.texts:
                return importlib.util.spec_from_loader(fullname, SvRxLoader(name))
            elif name_py in bpy.data.texts:
                return importlib.util.spec_from_loader(fullname, SvRxLoader(name_py))

        elif fullname == "SverchokRedux.nodes.script":
            # load Module, right now uses real but empty module, will perhaps change
            pass

        return None

class SvRxLoader(importlib.abc.SourceLoader):

    def __init__(self, text):
        self._text = text

    def get_data(self, path):
        return bpy.data.texts[self._text].as_string()

    def get_filename(self, fullname):
        return "<bpy.data.texts[{}]>".format(self._text)

[zeffii]

i'll need to break this before understanding how this puzzle falls together in mid-air :)

[ly29]

It requires going down into how python import actually work. http://chimera.labs.oreilly.com/books/1230000000393/ch10.html#_problem_179 is a good but longish read, some things have changed since then, also some more for 3.5, oh well.

Also doing this I came to realize F8 reload is mostly useless for deeper development work but very good for node level work.

[zeffii]

the reconnect script for cycles nodes might be useful. (this snippet stores current links, deletes node, recreates node, relinks: a few lines would be the same as what we want)

mostly this:

def store_links_update_reconnect(node):

    node_tree = node.id_data
    props_to_ignore = 'bl_idname name location height width'.split(' ')

    reconnections = []
    mappings = itertools.chain.from_iterable([node.inputs, node.outputs])
    for i in (i for i in mappings if i.is_linked):
        for L in i.links:
            reconnections.append([L.from_socket.path_from_id(), L.to_socket.path_from_id()])

    # also store prop values?
    props = {j: getattr(node, j) for j in dir(node) if not (j in props_to_ignore)}

    '''
         update here / new sockets, new props
    '''

    for prop in props:
        try:
            setattr(new_node, prop, props[prop])
        except:
            print(prop, 'is no longer a node property')

    for str_from, str_to in reconnections:
        try:
            node_tree.links.new(eval(str_from), eval(str_to))
        except:
            print('cannot connect', str_from, str_to)

[ly29]

Something like that is exactly what we need. Recompile a node script would be:

1. reload module
2. check if node classes have been recreated
3. collect properties and links from affected nodes
4. delete nodes
5. unregister old classes, register new
6. recreate nodes with new definitions
7. reload node data and reconnect
8. execute layout, if and how it is needed.

There are some consequences to this and kinks that needs to be worked out, but then node scripts would be first order citizens and not a special case as far as the system is concerned.

[ly29]

>>> n.bl_rna.properties.keys()
['rna_type', 'type', 'location', 'width', 'width_hidden', 'height', 'dimensions', 'name', 'label', 'inputs', 'outputs', 'internal_links', 'parent', 'use_custom_color', 'color', 'select', 'show_options', 'show_preview', 'hide', 'mute', 'show_texture', 'shading_compatibility', 'bl_idname', 'bl_label', 'bl_description', 'bl_icon', 'bl_static_type', 'bl_width_default', 'bl_width_min', 'bl_width_max', 'bl_height_default', 'bl_height_min', 'bl_height_max']

Maybe like this.

ignored_props = ['rna_type', 'type', 'shading_compatibility', 'bl_idname', 'bl_label', 'bl_description', 'bl_icon', 'bl_static_type', 'bl_width_default', 'bl_width_min', 'bl_width_max', 'bl_height_default', 'bl_height_min', 'bl_height_max']

[ly29]

There is a difference between this and the old function where we only allowed one script per file, here there aren't any limitations right now. We could impose them of course but I propose the following. One file can contain many different script nodes, all available under the menu, but the first one will be the default and used for the node it replaces.

[zeffii]

multiple scripts per node could be useful for more elaborate functionality. also perhaps allow importing from other script nodes.. or if these become modules i guess we'd be able to

from ..nodes.script_nodes.named import some_function

I could imagine that would be a neat feature.

[ly29]

Also if somebody puts together a set of nodes with some special functionality, easy to test them out with only distributing one file.