implementing groups

[zeffii]

For my own sanity i felt it was necessary to have a closer look at the mindfuck that is CustomGroup Nodes :) . I was especially interested in reusing as much of the PyNode api as possible, but I think @ly29 already concluded that also this requires us to pretty much roll our own.

I started a branch https://github.com/nortikin/sverchok/tree/group_experiment which hopes to get away from the use of the Frame and instead be more like the ShaderNodeTree with a back button and an overlay to show the parent NodeTree.

---

This is merely an exploration i'm not married to any of this code

here a list of things that need to be taken care of (feel free to interject / edit) in no order of importance.

• [x] socket rename
• [x] deleting existing sockets
• [x] moving existing sockets
• [x] socket type recast
• [x] handle reroutes
• [x] generate group from selected
• [x] generate group from selected + auto connect head and tail to monad IO, and parent node appropriately
• [x] show relevant properties in the sockets of the node tree
• [ ] sort relinks by Y location of periphery (aesthetic - avoid crossing noodles)
• [x] handle node group copy
• [x] handle node group rename
• [x] expand monad to ordinary nodes
• [x] handle IO/json
• [x] handle script nodes
• [x] possible space_data.node_tree (the [ nodetree icon ][ + New ][pin] ) should show the parent node tree name as in ShaderNodeTree ..
• [x] if possible keep old node BETA groups working.

---

Design Reference

lukas_t made his own node group for object_nodes: https://developer.blender.org/diffusion/B/browse/object_nodes/release/scripts/nodes/group_nodes.py

[zeffii]

The blender way isn't always the best. Am tempted to get as much of your node_group socket wrappers understood and implemented.. on the other hand I would like to try my own version of the panel (even if it doesn't make it into a final/accepted version).

[ly29]

As my design teachers used to say, don't be afraid to try! And don't be afraid to change and do things over again.

About the blender way, as much as I like blender sometimes it does drive me mad. But that is true for most advanced software...

[ly29]

The whole issue of having the properties in the nodes comes back to bite us. Of course we could make dynamic nodes with type and copying the custom properties. Or something.

The other option is to expose the relevant properties. But that includes creating an extra node and managing all of that... The best long term option given current blender limits is making a hierarchy of sockets like the standard blender ones...

[ly29]

Been playing around and some observations of issues, not necessarily in order of importance or directly urgent.

• one node function (group) can be present in many places, editing should update all and or provide a way to make unique. this is maybe a little bit tricky
• it does not handle reroutes, to get a corresponding socket there is get_other_socket in data_structure, now extended to handle down and up (this cycles node groups doesn't support) but it is simple and makes sense.
• the sockets are a mess. I don't want to deal with changing them now

[zeffii]

good points.

• The copy / make unique is next up:

  • I think the simplest approach would be an OK preliminary solution, that is to say: consider the act of duplicating a node as synonymous to make unique copy.
  • The more complex approach (and likely going to slow us down) would be to treat a sub group as something that can be compiled and turned into a proper function (composed visually as nodes, but represented as code inside the parent node.: something like:

    # add functions
    group_input = GroupInput()
    group_output = GroupOutput()
    range_float_001 = RangeFloat()
    math_001 = MathNum()
    math_002 = MathNum()
  
    # set function props
    group_input.outputs[0].socket_name = 'count'
    range_float_001.mode = 'COUNT'
    range_float_001.stop = tau
    math_001.mode = 'SIN'
    math_002.mode = 'COS'
  
    # make function connections
    group_input.outputs[0] > range_float_001.inputs[2]
    range_float_001.outputs[0] > math_001.inputs[0]
    range_float_001.outputs[0] > math_002.inputs[0]
  
    # etc...        

  Where GroupInput, RangeFloat and MathNum are purely computational classes..

• The reroute I haven't looked (probably because I don't use it much... easy to overlook the smaller but cool things)

[ly29]

Compiling would slow us down a lot, right now the code in many cases assumes that the process function lives in blender node group etc, now reason to change that right now. for redux it could be a possibility.

I don't agree that copying the node should make the node group unique. Every node group simple has a series of parent nodes and every node group node has one node group.

Something like this in node tree (this case assume a specific node group type for storing the node trees).

    @property
    def parent_nodes(self):
        parents = []
        for ng in filter(lambda ng: ng.bl_idname == "SverchCustomTreeType", bpy.data.node_groups):
            for node in ng.nodes:
                if node.bl_idname == "SvGroupNode" and node.group_name == self.name:
                    parents.append(node)
        return parents

Then we update them all, otherwise the functions/monads aren't reusable

(oops forgot filter syntax)

[zeffii]

I don't agree that copying the node should make the node group unique.

I didn't mean to suggest that, copying ideally should offer to ability to

1. make unique
2. make duplicate ( edit one, edit all )
3. make unique from duplicate

[ly29]

Okay, good, my mistake. 😄

Anyway, a decision that should be taken soon is if we hide away the node groups in an other node tree type that makes them invisible in the selection.

Also, just for fun, you can edit the old SvGroup implementation if you enter the right node group name in Group Node and press edit...

[ly29]

There is a lot of duplication going around, the user interface and your code mostly looks cleaner than mine old, but if we modify my old previously created node groups will still work.

There are some assumptions about editing in my old code and it does not seem to work very well.

[zeffii]

I just read Lukas_t's Interface code, and I think I understand it now after implementing my own.. neat to see much of the same things happening in both implementations.. I kind of like my approach, but might give his approach a run anyway..

[ly29]

If nothing else you learn something about a different style. I think I going to give my old group creation from selection code a test

[ly29]

The sane approach, in retrospect, would be to have a standard hierarchy of sockets with the different default values, even if we would need more than we cycles. having the properties in the node wasn't clever, but at the time we didn't know this problem.

now if we want to have the proper props display we need to replicate all of this, or create generic display mechanism like script did.

for me the most sane approach would be to create node representing the node group interface somwwhere and then expose the property when needed in the node interface.

therefore I conclude that sockets are a mess the way they are done Sverchok today.

[zeffii]

the bpy.props.IntProperty or FloatProperty would come from the Socket, I think only the StringsSocket has any real usefulness as a on-ui-slider.

[ly29]

Yeah, but it would be nice with different types, locked between say 0 to 1 as the original nodes do.

Anyway, defer that until later, I have some solution for that

[zeffii]

i'm just going to call node_groups/sub_groups "monads", you'll know what i'm talking about.

The weird/klunky thing (to me) about using a UiList to display + edit the props is that each monad must necessarily have a unique pair (tied to the group, but not necessarily to groupname) registered at monad instantiation.

I have questions, well one main thing..

• would F8 leave such things untouched, ie don't need to re-register

[zeffii]

My apprehension to going with UILists grows either due to laziness or knowing that in order for UILists to work we might as well copy most of Lukas' object_node node group implementation.

I'm not sure there's a net gain either way, but probably implementing UILists for the Interface should happen in a separate branch, to keep that implementation closer to the example.

[zeffii]

dynamically create properties: http://blender.stackexchange.com/a/1786/47 have not tried yet, but seems interesting...

[ly29]

To get properties to show without having to invent a whole new socket system there are some options I can think of, listed in how "clean" they make me feel.

• Create a dynamic node class with type for each monad containing the relevant properties, that we get the options for by parsing the nodes
• Create a node for each property in the some layout (another node category) and modifying socket to be able to display that property, also some simple modifications to the update system are needed
• A script node like option without the control of the properties

Somehow the second option kind of appeals me right now.

Some notes:

For option one, it is trivial to collect the needed info for the properties for the dynamic class creation.

>>> ng.nodes['Sphere'].inputs['Radius'].prop_name
'rad_'

>>> ng.nodes['Sphere'].rna_type.rad_[0]
<built-in function FloatProperty>

>>> ng.nodes['Sphere'].rna_type.rad_[1]
{'options': {'ANIMATABLE'}, 'description': 'Radius', 'update': <function updateNode at 0x116560268>, 'name': 'Radius', 'attr': 'rad_', 'default': 1.0}

# collect properties
node_properties = default_properties.copy()
for socket in inputs:
     other = get_other_socket(socket)
     if other.prop_name:
            func, options = getattr(other.node.rna_type, other.prop_name) 
            properties[other.prop_name] = (func, options)
group_type = type("SvGroupNode{}".format{group_name}, (Node, SvGroupBase), properties)
#register etc

Ref https://github.com/Sverchok/SverchokRedux/blob/master/core/factory.py

For two the question is where to store them, the answer would be, I guess, in the node monad layout when we save it or another node tree type which shadows the node groups by name and contains relevant properties. this is backward and complicated, but pure brutish hack of it seems kind of fun. This also has the advantage that more complicated things might work better.

The script node method of having a number of fixed properties and using them feel ok there but here it feels out of place since the properties won't be well behaved.

[ly29]

Something like that could work, of course it is also not so pretty. But less bad than some other options...

[zeffii]

I really like the idea that the property used for the slider / bool / whatever is part of the SocketType.

[ly29]

Re where value belongs: It really should be, it makes it clearer where the data comes from and removes the need for bad code. All I have lot more appreciation for the standard hierarchy of node sockets now. I really wish changing to such a format would be none breaking but it isn't. Too much cruft going on.

Also the amount of work is because bad decisions earlier, due lack of understanding of the tradeoffs, if would say we don't care about backwards compatibility it becomes so much easier.

[zeffii]

i think we could tunnel the sliderproperty from the node the socket came from?

parent_node_tree:       parent node.inputs (3) (slider from 1 if not linked)     
--------------------
monad_node_tree:        ng_outputs  (2) <- some_node_input.FloatProperty (1)

[ly29]

unless we have two instance of a monad node, then that would lead to adjusting one of them changes the other.

[ly29]

Imagine something like this for socket.draw()

def draw(self, context, layout, node, text):
    data = bpy.node_groups[node.monad].nodes[self.prop_ref]
    layout.prop(data, self.prop_name)

then changing the exposed property changes all monad nodes interfacing the monad. Get around this lead to option two above, the exact specifics that can be argued. it is fairly straightforward to implement but the lead to lots of things lay around.

[zeffii]

OK.. I'll try with the Frame thing then, but it's not clear to me how that would make monad "instances" easier.

[zeffii]

first I want to see if this is a viable way to tunnel a property.. ;)

[zeffii]

@ly29 do you know if there's a way to do a prop_search on a subset of a collection? like

    layout.prop_search(self, "group_name", bpy.data.node_groups, filter('attr', 'sub_group'))

[ly29]

No, not directly. I guess you can use an enum with a function making the filtering in worst case

[zeffii]

well, it's displaying the 'tunneled' socket now, but no data is transferred when the parent node's sliders are moved. A problem for tomorrow. ..almost tempted to revert because this just feel ridiculous and brittle (even if the code isn't too ugly.. it's still icky...)

[zeffii]

[ly29]

It does feel wrong, doesn't? For the update to work some changes has to be made, which would make it even more brittle...

[zeffii]

am going to revert...

[ly29]

Yeah looking at it now it feels like the sane decision.

[zeffii]

http://stackoverflow.com/a/12049323/1243487

git reset c640aeb >> will say : Unstaged changes after reset: git add -A >> addsthem git reset --hard >> bulldoze git commit -am 'reset to c640aeb' git push origin group_experiment -f

[zeffii]

bpy.ops.node.clipboard_copy()
bpy.ops.node.clipboard_paste()

wtf :)

[ly29]

If it works great, otherwise my old operator to create a group from selection should work with minimal changes (.bl_idname for nodes etc)

[ly29]

https://github.com/nortikin/sverchok/blob/master/utils/group_tools.py

[zeffii]

Yeah I did dig through that first before starting this :) I felt there had to be some way to reuse code that's already tried and tested by millions.

class SvMonadCreateFromSelected(bpy.types.Operator):

    bl_idname = "node.sv_monad_from_selected"
    bl_label = "Create monad from selected nodes (sub graph)"

    group_name = StringProperty(default="Monad")

    def execute(self, context):

        ng = context.space_data.edit_tree
        nodes = [n for n in ng.nodes if n.select]

        if not nodes:
            self.report({"CANCELLED"}, "No nodes selected")
            return {'CANCELLED'}

        parent_node = ng.nodes.new('SvGroupNodeExp')
        parent_node.select = False
        parent_tree = parent_node.id_data
        parent_node.location = average_of_selected(nodes)
        bpy.ops.node.clipboard_copy()

        monad = group_make(parent_node, self.group_name)
        bpy.ops.node.sv_switch_layout(layout_name=monad.name)

        # by switching, space_data is now different
        path = context.space_data.path
        path.clear()
        path.append(parent_tree) # below the green opacity layer
        path.append(monad)  # top level

        bpy.ops.node.clipboard_paste()

        # remove nodes from parent_tree
        for n in reversed(nodes):
            parent_tree.nodes.remove(n)

        return {'FINISHED'}

but this hasn't been heavily tested by me yet.

[zeffii]

it might be sane to have a filter list to unselect certain kinds of nodes if they are in the user's selection... but that's the easy bit.

[ly29]

that code does even spend that much time of copying, just collecting links and such things is quite enough of a chore.

[zeffii]

sv_monad_tools.py holds the Operators and the utils now..

It probably makes sense to place the monad input and output at the extremes/bounding box of the copied nodes, I find myself moving them all the time..

now monad IO nodes are placed at (min_y + max_y) / 2 and min_x and max_x respectively.

[zeffii]

I realize now the one important difference between my simpler make_monad_from_selected vs your original counterpart is that you auto connect the head and tail of the selected nodes to the monad IO nodes, and the parent node.. This means my implementation is only partially successful at the moment

#psuedo
links_to_reestablish = [for selected nodes track which nodes have links with un-selected nodes]
for link in links_to_reestablish:
    # (I think this will automatically propagate like the manual connection)
    if link is from_socket:
         link monad_input_exp 
    if link is to_socket
         link monad_output_exp

or

import bpy

ng = bpy.data.node_groups['NodeTree']
nodes = [n for n in ng.nodes if n.select]

relinks = dict(inputs=[], outputs=[])
for n in nodes:
    for idx, s in enumerate(n.inputs):
        if s.is_linked:
            link = s.links[0]
            if not link.from_node in nodes:
                relinks['inputs'].append(dict(socket_index=idx, linked_node=link.from_node))

    for idx, s in enumerate(n.outputs):
        if s.is_linked:
            link = s.links[0]
            if not link.to_node in nodes:
                relinks['outputs'].append(dict(socket_index=idx, linked_node=link.to_node))

print(relinks)

current implementation of mine is a bit broken

[zeffii]

also..

• perhaps monad IO should auto recast their Socket type, and also auto rename in that case, but not auto rename when the socket type doesn't change
• maybe the parent node should also get a dummysocket?

get_relinks() and relink() are a little broken.. but committed anyway.

[zeffii]

let's add more naming convention to make it easier to talk about.

• pre monad nodes: all selected nodes prior to turning them into a monad.
• periphery nodes : are those nodes which are not part of the pre_monad set but that do have inputs or outputs attached to one of more of the pre_monad set.

after monad creation

• links should be re-established between the monad's parent node and the periphery nodes,
• internally the monad links should connect those links to the monad's input and outputs.

[zeffii]

I may copy liberally from group_tools.py/SvNodeGroupCreator :)

[ly29]

Feel free to do it. Also an important point is that one socket may have many to sockets. Or also why socket.links is an array. (well it is a property function actually but that is a different matter)

[zeffii]

ah yes, outputs[index].links[:] nodes can output to various other nodes from one socket, but can only receive (or we have limited it to do so) into inputs[index].links[0].

..this way take a while to figure out a nice way to do it, no ETA.

[ly29]

The ugly way is in SvNodeGroup

[zeffii]

I'm under no illusion that i'll come up with anything nicer than SvNodeGroup, but i still want to have a think about it

[zeffii]

Possible the code I just pushed will suffice as 'make monad from selection' - else i'll need to really just rip your code.. because I have little patience for this.