groups in IO/json per #782

[zeffii]

per https://github.com/nortikin/sverchok/issues/803

this is a topic in it's own right..

I feel like doing something experimental with the serialization. Failing that then I guestimate that the current IOjson code only needs minimal massaging to hold groups.

[zeffii]

i'm thinking perhaps less experimental..

the layout_json (dict) will take an additional key called 'monads`, into which each monad is serialized.

{
"_all other keys_": [],
"monads": [
    {"name": "monad_name", "nodes": [ ]}
    {"name": "monad_name2", "nodes": [ ]}
]
}

[zeffii]

each monad_instance_node can be asked to produce the cls information needed to recreate at least the IO interface (outer and inner).

[ly29]

I agree, only small things are needed. Something like on import first import groups. Also if a group in group is found defer and make a second pass after all groups are created.

[ly29]

The monad class is created from the real monad, it is (right now) the only way to generate the actual class. Monad can uniquely identified from the monad.cls_bl_idname.

Create monad:

1. Create monad
2. Set .cls_id_name
3. Import all nodes into the monad
4. Call monad.update_cls()
5. Create the monad instance node with .cls_bl_idname as node type.

[ly29]

Sometimes I think a generic group node without special class/properties support that you can use during these sort of situations would be a good idea.

[zeffii]

A dummynode just for the purposes of being able to first make / connect sockets, then fill appropriately with the monad it represents? A generic hotswap node function is needed anyway :)

[ly29]

Yeah, exactly.

[ly29]

Look here: It has 3 values to set after creation and then it behaves as the monad node except it does not have properties.

https://github.com/nortikin/sverchok/tree/dymmy_monad_node

[ly29]

Reading the old code I noticed we referred to sockets by name.

Also feel the need to impose some interface for nodes and shift responsibility to the nodes, would clean up the export code a lot and move responsibility to where it belongs. Which given OO and all makes sense.

[ly29]

Dummy/generic monad node updated to handle be functional but without properties...

[zeffii]

not sure my brain is able for this now (or ever).

but I came to the same conclusion during the early stages of IOjson, nodes should have a function which the IOjson can call to get all the needed info. Many nodes can probably inherit a generic function, of the 180 nodes maybe 20 require individual instructions, but the others (iirc) follow a predictable pattern.

Also there's the def __getattr__(self, name) which we don't use anywhere in Sverchok, but has quite magical possibilities. see https://github.com/zeffii/pyBLOK/blob/master/core/blok_functions.py#L202-L236

pyBLOK is a scripting interface to produce XML files for a nodebased softwareSynth.

[ly29]

I think it is topic worth revisiting. the __getattr__ is fantastic but is hasn't quite worked itself into my vocabulary so to speak

[ly29]

Ended up reading about __getattribute__ as well. Fun stuff. Python is fantastically flexible in so many ways.

[zeffii]

cool, i have never used __getattribute__ nice writeup. looks like fun

[ly29]

The dummy/generic node is in master now. It also helps with linking & append scenarios.

[ly29]

Was thinking if we remake this we drop the old export code but keep the old importer for older versions of the json files. To be less bothered with compability considerations => cleaner code.

[zeffii]

best way to proceed. imho.. and new code would be written with redux in mind i hope.. :)

[ly29]

As a side note a shocking number of template fail to import or contain old nodes.

[ly29]

@zeffii That topic is for #818

[zeffii]

and the least fun of all tasks... is making the exporter .json write out lists on one line rather than drop a new line for each value.

[ly29]

Some interface like this for the nodes.

def get_json_dict(self, external=False):
     return dict(self.items()) # + some default things

def set_json(self, items):
     for key, value in items.items():
         setattr(self, key, value)

Then nodes with special requirements could simply give a dict that they get back set themselves up with.

[ly29]

Some nodes could even share the code for example text resources etc...

[ly29]

Well, we (but mostly you) got a change to test the templates today. It is a good way to get to know the problem domain a bit better. :)

[zeffii]

well, the importer could have done it in an automated way, there was only a few things

{ 
  "CentersPolsNode": {"replace": "CentersPolsNodeMK2"},
  "SvTextInNode": {"text_lines": ["add 'stored_as_json' with content of text_lines" ]},
  "LineConnectNode": {"replace": "LineConnectNodeMK2", "polygons": {0: "Edges", 1: "Polys"}},
  "BMeshViewer": {"replace": "SvBMeshViewerMK2"}
ListSum
ListShift
}

[ly29]

Such logic would be sane to have, especially if could apply the same filters to opened files...

[zeffii]

i missed one in 5.Sverchok_Spread.json

[zeffii]

as a first step i'm contemplating turning all the bodies of the if statements to function calls within the export / import code, then working these functions into the nodes..

                elif node.bl_idname == 'SvTextInNode':
                    params = node_ref.get('params')
                    current_text = params['current_text']
                    node.textmode = params['textmode']

                    if not current_text:
                        print(node.name, "doesn't store a current_text in params")

                    elif not (current_text in texts):
                        new_text = texts.new(current_text)
                        if node.textmode == 'JSON':
                            json_str = json.dumps(node_ref['text_lines']['stored_as_json'])
                            new_text.from_string(json_str)
                        else:
                            new_text.from_string(node_ref['text_lines'])

                    else:
                        texts[current_text].from_string(node_ref['text_lines'])

would become

def import_text_from_IO(node, node_ref):
    texts = bpy.data.texts
    params = node_ref.get('params')
    current_text = params['current_text']
    node.textmode = params['textmode']

    if not current_text:
        print(node.name, "doesn't store a current_text in params")

    elif not (current_text in texts):
        new_text = texts.new(current_text)
        if node.textmode == 'JSON':
            json_str = json.dumps(node_ref['text_lines']['stored_as_json'])
            new_text.from_string(json_str)
        else:
            new_text.from_string(node_ref['text_lines'])
     else:
        texts[current_text].from_string(node_ref['text_lines'])

and later called by

              elif node.bl_idname == 'SvTextInNode':
                       import_text_from_IO(node, node_ref)

As an interim step, then in a next phase add the functions to the nodes, and replace node. with self.

[ly29]

Sounds like a very sane approach, did something similar with monad class creation.

[ly29]

Just for reference: how I did it in redux

https://github.com/Sverchok/SverchokRedux/blob/master/ui/node.py

[zeffii]

        for n in sorted(nodes_to_import):
            node_ref = nodes_to_import[n]
            bl_idname = node_ref['bl_idname']

            try:
                if old_nodes.is_old(bl_idname):
                    old_nodes.register_old(bl_idname)
                node = nodes.new(bl_idname)
            except Exception as err:
                print(traceback.format_exc())
                print(bl_idname, 'not currently registered, skipping')
                continue

            if create_texts:
                add_texts(node, node_ref)

            gather_remapped_names(node, n, name_remap)
            apply_core_props(node, node_ref)
            apply_superficial_props(node, node_ref)
            apply_post_processing(node, node_ref)

[zeffii]

The Redux implementation seems neat, but I think that has more to do with it being devoid of nodes that do complicated things. It's definitely something we can hope to achieve (iojson was big learning experience w/respect to pynodes, for me anyway.)

https://github.com/nortikin/sverchok/blob/iojson_codeshuffle/utils/sv_IO_panel_tools.py#L426-L459

[ly29]

It is mostly useful as reference for a default implementation of nodes. the 80-90%. The rest is a lot of work getting right of course... 10% of nodes give 90% of the work.

Groups are a slightly subtle. Have to think about a neat solution there.

[zeffii]

10% of nodes give 90% of the work.

so true.

[zeffii]

now.. the first part is what it should have looked like from the beginning..

    def generate_layout(fullpath, nodes_json):
        '''
        first create all nodes.
        '''

        print('#' * 12, nodes_json['export_version'])

        update_lists = nodes_json['update_lists']
        nodes_to_import = nodes_json['nodes']
        groups_to_import = nodes_json.get('groups', {})

        group_name_remap = add_groups(groups_to_import)  # this return is not used yet
        name_remap = add_nodes(nodes_to_import, nodes)

        print_update_lists(update_lists)

        '''
        now connect them
        '''
        ...
        ...

[zeffii]

lots of deferment.. possibly easier to edit without feeling like your going to break something accidentally.. i don't know.

[ly29]

group name remap isn't actually needed, a monad instance can find it's monad

[zeffii]

part deux

adding support for monads. first make it work, then probably drop references to IsGroupNode

[zeffii]

it imports monads now, but I didn't have time to switch the dummynode with a monad_instance_node. There's a bunch of things still to do,

[zeffii]

maybe i'm over-complicating this, but am I crazy for thinking the SvMonadGenericNode needs some kind of class function to convert itself to a monad_instance_node -- when passed the imput templates / cls_dict?

[ly29]

Actually if you set the .cls_bl_idname it should just work, without any properties but work. But such replace function would be sane. The node expand operator is the method I would use as a template.

[zeffii]

i thought also the dynamically generated MonadNode instance should have some kind of cls_dict generator function?

[ly29]

The monad generates the instance class. The instance is dumb in that regard. https://github.com/nortikin/sverchok/blob/master/core/monad.py#L108-L150

[zeffii]

some progress... and showing signs of what needs to be offloaded to the node i think.

[ly29]

Yeah, you know that the best after laboring with the I/O code.

[zeffii]

For the most part it seems to be working, except the minor (lol) issue of the inner sockets not connecting to the group_input / out_put. This feels brittle..

[zeffii]

seems I need to repopulate the IO nodes in the add group part, prior to setting the links.

[zeffii]

the update_list has one weakness, it doen't distinguish between multiple same-named sockets on an monad's IO nodes - I think we'll need to force the socket names on those two SvGroup*putNode to be unique.

[zeffii]

or include the index in the update_list..

[ly29]

The simple solution is to sort the links based on update list and socket index. But I feel a cleaner solution should be avilable... The update system could also benefit from a review but that is code I would prefer not to mess with right now.

[zeffii]

I motion to force name uniqueness in the SocketAcquisition class, a sane approach that has no real impact on the user.