Plans for Sverchok

[zeffii]

We wrote a lot of code in a year, and have gone through various changes and we now have a pretty fast execution model which for all intents and purposes works.

Maybe it's time we started abstracting again, rather than gradually modify the existing code -- set the existing code aside and start fresh. We have all the working code to reflect on, and can make more brutal decisions about alternative approaches.

If we always have to maintain backwards compatibility things are so much slower to progress. @portnov submitted a patch that would vastly simply bmesh based nodes, and @kosvor submitted his own bmesh nodes, but really the framework we provide could be better.

After extensive exploration of python over the past year I think we should attempt a SverchokRedux, which intends to make a more serious effort at separating the Node/UI code from the code that Operates on input/output. Reading through antimony was an indication that there's a far less verbose alternative (but caveat, Antimony is not vectorized...this means we have an extra level of complexity).

The benefit of further separation of UI and execution is that we could eventually replace The UI and the Operation code.

I think Sverchok in it's current state has reached the limit of what is possible, next step is to take a step back and think about how to rewrite it better.

Please visit https://github.com/Sverchok/SverchokRedux for the latest information and discussions.

[zeffii]

This is just how I feel about it, you guys of course may have different ideas.

[ly29]

I agree. The current code while working, isn't optimal by any means.

What I would like is to do

• change the execution model
• optimize memory consumption.
• implement proper groups as code reusable code blocks
• redo data model
• interaction model
• data model
• make nodes serialize

Execution model Instead of each node parsing the complete list view the nodes as source code and compile them to a program and execute it for each element of the list, simplifying most nodes a lot. Some, a few might need to manipulate the data structure itself but for many cases it isn't needed. Benefit, we can lazily eval the data which can also, potentially, allow for parallelization of the execution.

Optimize memory consumption. This ties together with the execution model, now every state of the every program is stored which leads to a lot of memory being wasted.

Implement groups as reusable blocks This ties together with some other ideas, of how we view the node layouts. Each layout is essentially a source code file.

Redo data model. Create sockets that makes sense, like mesh, color. Match those sockets to efficient representations that makes sense in blender context, like numpy arrays, bmesh etc using the most efficient means for accessing and setting data, foreach_get etc.

Interaction model. First of all, take a look at where in the blender geometry nodes makes sense. Edit mode and object mode. (Perhaps also sculpting, weight paint and vertex paint.) Example, in edit mode you could have number of Sverchok programs available as operators manipulating, say the the selection state of a mesh, perhaps being called from an auto generated/custom call menu. Of course creation of parametric objects like we can do now is also possible.

Make nodes abstract from the blender node code. This makes is possible to create json representations in a more sensible way, to compile and use custom classes also becomes easier. The idea being, on startup dynamiclly create the blender node classes and load the code.

[ly29]

In the open gl or viewer node a lot of settings cannot be controlled by the layout, I think these settings could be abstracted out into a separate controller node so that two viewers could be controlled from one setting node which only does this. This would work in a standard way for such settings so that also can they can be controlled by default from another node.

[ly29]

This isn't to say this is a lot of work to bootstrap and get working before we start porting a more complete set of nodes and not to mention working out problem as we bump into them. But I think it is less work than try to redo what we have now into some better and well thought out.

[ly29]

A node would consist of two parts, a settings part and a execution part. The settings should be used as a argument for the constructor. The settings for each and connection betweens them would be all that is needed to serialize a layout.

• Okay this might be to detailed for now

[ccamara]

Hello, although I have never contributed seriously to this repo, I've been following it for quite some time, unfortunately I have never been able to contribute since A) My python/programming skills are quite fair (I'm an architect myself and never had a formal training in programming), B) I could not understand how could I contribute or help you C) I am not an experienced user on parametric modelling (I just know the concept, although I have never had the change to use it).

I think that this refactoring may be the perfect situation to try to help a bit on this great project, and this is good news for me.

However, I think that this can be also good for keeping things a little bit clear in the repo. I would suggest that, in case the refactoring you are talking about is to take place (sounds good to me, although I have to admit it goes beyond my python knowledge) it would be a good idea to create a new branch called dev or v.2.0 or something like that and use it as a base to fork, develop and merge. In this way we could keep current sverchock in master branch (or legacy, or even better, tag it with version 1.x or whatever), which will be very useful for example in the first moments of refactoring, when the new code will likely break the old one. From what I've seen so far (correct me if I'm wrong) the development workflow is not very clear and due to the fast development pace you guys have (thanks again for your great job) sometimes, things get messy or even break other ones, and having a better development workflow could solve these issues.

Regards and thanks for your hard and great job so far.

[zeffii]

realistically we already have our nodes now as operators which take input, extend the input if needed, process the input as a function of parameters, and produce an output.

it seems you have much more elaborate and considered vision @ly29 , the ability to serialize the nodes is definitely high up on the wishlist, being able to confidently export a tree to a pain-text json brings a massive amount of flexibility.

[zeffii]

@ccamara yes, it is not my intention to break current sverchok. Using a GIT repository and version control allows us to have pretty wild experiments and keep the original code unchanged. If that was a concern, it need not be.

Keep in mind, these are only preliminary discussions -- planting seeds, watching ideas grow and become more mature. I don't know how soon it will materialize, it depends on how much we can agree.

From my perspective development had stagnated for a while, because i'd pretty much done all the nodes I wanted, and making changes to existing nodes brings a bit more work than I want to spend time on. Maintaining backward compatibility is for me really a weakness during a development cycle. Old blends will always work with old sverchok.

[ccamara]

@zeffii Thanks for clarifying. If, by any chance, I can help you, I will do it gladly.

[zeffii]

developing an addon that has persistent state (ie a node tree) is very different from writing operator based addons. With regular addons you don't have to worry that your new code / code changes will break existing blends, because addons get invoked when you need them, and then have done their job.

but a node tree is different, you need to be careful about added sockets/parameters and internal functions, and tiptoeing around that is like a minefield, and frankly not very pleasant and it has prevented me from adding code as freely as I would like to. I love writing code, but what I don't like is to have to take all that other stuff into account. I'm sure @ly29 , @nortikin understand that..

[zeffii]

@ccamara the best way you can help us without writing code is to show us cool things you've done or share .blends / .jsons with that. I find seeing rendered results by other people very motivational, because it reaffirms that this isn't some solipsist's dream. Keep asking questions when you find something confusing or lacking.

[nortikin]

I agree all of Linus and Dealga said. Data model for me is main thing to user. IFC implementation (BIM) in output of bmesh viewer to write to RNA blender parameters of object and than export to IFC (there was importer of IFC). We need global IFc specifications realized in viewers.

About core coding, it is my weak part, Not see who can help except @portnov with C++ wrting. Than Sverchok became some more than addon, we will need compile C++, will it be integrated to blender somehow and become part of blender? not sure.

No doubts numpy in data model will be, but not consistent arrays can break things, as i understand, or can we awoid regulat arrays in numpy (different length of polygons etc.)?

Also threading needed to be implementing, i see one core loaded always from sverchok, even python has threading librery.

Wish we could make work preparations for version 1.0.0 to start coding.

I have 'organization' in github, propose to start new repository there https://github.com/Sverchok

[zeffii]

BIM and IFC : http://www.visualarq.com/info/ifc/#IFC

[ly29]

The sverchok redux would be completely separare from current and both addons could work at the same time.

[zeffii]

yeah, their operations should not interfere with eachother.

[nortikin]

agree, it will let us concentrate on basic things.

[zeffii]

I'm not sure if we need C/C++ , JIT or just smarter py / numpy.

In numpy you can have object arrays, but you could also store a flat array of indices and a second array to store start and end for each polygon.

# both data-streams would be fast, can all be built by generators
flat_array = [0,1,2,0,2,3,0,4,5,3,4,5,2,3,5,6,4,2]
poly_indices = [[0,2],[3,5],[6,10]]   # 3, 3, 4 verts

but I would test the speed of an object array with a more human readable data structure too, because if the speedup isn't enormous, i'd use the slower one for readability.

[zeffii]

how-to-make-a-multidimension-numpy-array-with-a-varying-row-size (explains why a variable length array isn't so useful in numpy)

[ly29]

Polygon data is a problem, but a small one which we can solve in some different ways.

[ghost]

I just wanted to chime in and voice my interest and thanks. The plans for a new SverchokRedux sound well thought out to me and I agree that sometimes it makes more sense and can even be faster and more motivating to start over at least partly and solve what has been learned in the process at the root.

Thanks to all involved!

Tom

[zeffii]

import numpy as np

v = [
    [-1.0, -1.0, -1.0],
    [-1.0, -1.0, 1.0],
    [-1.0, 1.0, -1.0],
    [-1.0, 1.0, 1.0],
    [1.0, -1.0, -1.0],
    [1.0, -1.0, 1.0],
    [1.0, 1.0, -1.0],
    [1.0, 1.0, 1.0]
]
verts = np.array(v, np.float32)

# face keys, all quads in this example.
# [[1, 3, 2, 0], [3, 7, 6, 2], [7, 5, 4, 6], [5, 1, 0, 4], [0, 2, 6, 4], [5, 7, 3, 1]]
idx_flat = np.array([1, 3, 2, 0, 3, 7, 6, 2, 7, 5, 4, 6, 5, 1, 0, 4, 0, 2, 6, 4, 5, 7, 3, 1], np.int32)
idx_lookup = np.array([[0,3], [4,7], [8,11], [12,15], [16,19], [20,23]], np.int32)

# float64 would be possible too

here the lookup is a begin and end slice into the flat array, this allows a homogeneous 2d array to act as a lookup table to a flat array of [*tri, *quad, *ngon, *tri, *quad, *quad, *qad, *tri, *ngon]

[unity2k]

Seeing I won't be adding code, I can contribute financially as a show of support if further monies need to be raised to take this next step.

[nortikin]

@zeffii , so, you propose sent two values for polygons... i like such idea, but... nested objects, also grouped objects (data tree) will increase this slices to enormous value, lets count 1 data 2 slice objects 3 slice groups of polygons if something 4 slice polygons hm. it need to be thinked.

[nortikin]

unity2k, we have paypal account http://nikitron.cc.ua/sverchok_en.html here feed sverchok. current state - accumulating money.

[zeffii]

@nortikin the data type needs to be well-engineered, for nestedness we use objects with layers.

[zeffii]

we should be able to ask an object, "how many layers do you have", I don't think we will loose flexibility, sverchokRedux should not be a regression in any aspect, but a rethink, yes.

[nortikin]

ok, it is all shaped in class-object of python.

[ly29]

Definitely not a regression, however in the beginning some functionality might me lost waiting for correct implementation, some behavior will be standardized and therefore some things might work differently. In the beginning we should be strict with exceptional behavior to produce the best standard methods for doing things resulting in the a simple, clear and powerful solution. Extendability and flexibility should be priorities.

[ly29]

That said we should really focus on simplicity in the node function and keep complexity in the core.

[zeffii]

i'm not interested in writing C code, but http://julialang.org/ looks like a nice JIT language with execution speeds which are not too far off low-level languages. Not saying this is the way to go for sverchok, but it's worth looking at before going C/C++...if we step away from pure python execution .

[ly29]

Cython might be good. But for now python is enough.

[zeffii]

Yes! Cython should have enough support for the things we're doing, none of our python does anything really obscure I think.

[ly29]

Some concrete steps. Bootstrap

• sockets
• node tree
• nodes
• compiler
• execution engine
• Development documentation

Here the number of nodes should be kept very small to keep changes very simple to implement.

Development

• Node types - as needed, mostly to support the compiler
• Execution modes
• Dynamic node creation
• Data structure - data types - finalize and develop
• Make the structures serialize

And then, create full node set and documentation.

[zeffii]

Early implementations of Nodes I think should include

• number ( float / int switchable )
• range (int / float switchable)
• math node (limited subset first)
• vector math node (limited subset first)
• basic list joining, length, split, reverse
• UV EdgeSurf ( uv connection but with more standard name)
• a scripted node (mk3)
• initial View nodes.
• lightweight index viewer
• objects in
• matrix in / matrix transforms

It might be an idea to be able to draw the output of any node even without connections, if the combined outputs of that node can produce a drawable mesh. So a server-client relationship between the nodes and the master viewer draw mechanism.

[zeffii]

@ly29 I know you like the idea of doing away with the separation math - vector math nodes, how does @nortikin feel about that?

personally, i'm OK with them not being merged, unless the resulting code is a massive improvement

[ly29]

@zeffii I feel bad about it. perhaps even create a separate trigonometry etc, node since the amount of methods is to big in the dropdown. code sharing behind the scenes etc I think is fine.

[zeffii]

great. I'm happier about math - vector math - trig math separation too.

[zeffii]

and yeah, as I see it the math node could be a parent class. and vector math and trig math nodes would inherit from it, allowing all the convenience methods to be defined in one place.

[ly29]

How do you feel about Svr prefix instead of Sv for blender classes? To similar?

[zeffii]

doesn't bother me, SvX perhaps? more prominent

[ly29]

pep8 says CamelCase for classes, would prefer SvXr in that case -> SvXrNodeMath. Very small thing but nice to settle before starting

[zeffii]

cool, or SvRx ?

[ly29]

ui factory test, run from text editor. of course should have an init method that parses the class and creates sockets etc. I think this is only possible using exec etc but would be happy to be proved wrong.

import bpy

node_template = """
class SvRxNode{name}(bpy.types.Node):
    bl_idname = 'SvRxNode{name}'
    bl_label = '{name}'
    bl_icon = 'OUTLINER_OB_EMPTY'

    @classmethod
    def poll(cls, ntree):
        return True
"""

class SimpleTest:
    pass

def ui_factory(cls):
   test = node_template.format(name=cls.__name__)
   exec(test, globals())

if __name__ == '__main__':
    ui_factory(SimpleTest)
    bpy.utils.register_class(SvRxNodeSimpleTest)

[zeffii]

well I don' t really mind if we go fully dynamic, similar to things we already do in the ctrl+space menu https://github.com/nortikin/sverchok/blob/master/ui/nodeview_space_menu.py#L76-L87 too

It does mean a higher threshold for anyone new to the project if they want to debug their node, i think... we'd have to be very certain that's a good idea.

[ly29]

that way looks more sane... yes, that is good remark. I think the advantages outweigh the negatives in this case. it might be made optional. the main advantage is that all properties and sockets are inspectable in a standard way.

[ly29]

sometimes if even think about an osl like syntax...

[ly29]

Now is the time discuss basic ideas anyway.

[zeffii]

using 'type()` there was an eye opener for me, without it that file would have been maybe 100 lines longer for no good reason other than boilerplate ( already it is bloated by the choice to show/not show icons... ) but the end users will never encounter that as a problem...

[ly29]

It is very cool indeed, also works with properties which I was worried about (for some reason)

import bpy

class SvRxNodeBase():

    @classmethod
    def poll(cls, ntree):
        return True

class SimpleTest():
    pass

def ui_factory(cls):
    # parse class to create dict
    name = "SvRxNode{}".format(cls.__name__)
    args = {"my_prop":bpy.props.FloatProperty(), "my_class":cls, "bl_idname":name, "bl_label" : cls.__name__}
    bases = (SvRxNodeBase, bpy.types.Node)
    res = type(name, bases, args)
    bpy.utils.register_class(res)

if __name__ == '__main__':
    ui_factory(SimpleTest)

[zeffii]

for instance for FLOW I was getting nodes down to a couple of lines,, but they were basic

At that level I was having problem registering nodes using __name__ , tho SvRxCore could have utility functions to take a list of Class References

from SvRx.Core import multi_class_register

register():
    # it would auto unpack, not expect a list.
    multi_class_register(class1, class2, class3)

but discourage more than a sane amount of node / class definitions per file anyway