Proposed refactoring

[wclyffe]

Hi guys,

First, great work on this addon as parametric modeling was sorely missing in Blender.

As I became intrigued and tried to wrap my head around what's possible, I decided to have a look at the code, which is growing at a frantic pace.

Anyway, in the process of trying to understand for myself how all that is working, I ended up forking and refactoring a bit: https://github.com/wclyffe/sverchok/tree/refactor The details of what have been done are in the commit log: https://github.com/wclyffe/sverchok/commit/a5d1b16f21c319cb555439534c383c6280f738ca

Essentially moving nodes in their own directories and changing how the nodes are imported in main init file. Apart from the imports the files contents should be untouched

On my end, all seem to be working as before, but I must say I haven't tried all the nodes...

Before I continue and begin to eventually refactor further, I would like to know if you are interrested, and if that is the case, some input would be greatly welcomed.

Again, you have done a great job and I certainlly hope that Sverchok will continue to extend what's possible with Blender

[ly29]

It is a long standing issue that we meant to do something about #104 I like it, some work probably needs to be done regarding the Viewer Draw code anyway, there are some bad practices in the node_s.py where import from some files for a clean up handler etc. Issues that are better resolved than not.

[ly29]

Looked quickly, saw one problem, in interpolation.py the node imports the python bisect module, not the bisect node, easy fix. Might have missed other things some subtle, some not.

[wclyffe]

About Viewer Draw, I haven't looked closely at it yet. For the first try at a refactor, i wanted to change as little as possible to the code itself. Just get things working again with the updated directory structure and since I had modified the imports anyway I took a stab at a saner way to do the reload. I think there is a possibility to do away with the make_categories function if there's only one node per file. Currently there's voronoi and delaunay living together that I've noticed.

About bisect. I will have a look at it. While trying to do away with the 'from foo import *', I migth have missed some stuffs. And I have not tested with all the nodes

For further refactoring I have to investigate deeper to become more familiar with the codebase. While doing that, I was thinking of doing some general pep8 cleaning

[ly29]

Text In/Out also. There is some double nodes due to legacy also, List Item and Circle, we discussed that when it happens we should put into separate files and upgrade them automatically, which is something the file upgrade.py is working towards case for case... The name data structure is slightly misleading, but it is also a file that should be re-factored, for example whole update section should be split out from there.

[zeffii]

pep8, but not strict on line lengths.

[wclyffe]

Concerning the file naming, I just had a cursory look at the content and made a guess. That is something where you are more qualified than me, but I though util was too general

pep8: on line lengths I agree

[zeffii]

grouping by directory is certainly a nice development, I expect this (and what it involves) will make its way into master soon

[wclyffe]

While I'm doing pep8, do I put license comment block in each file? And if so, do you have a specimen from some file where there's already one?

[ly29]

Not voronoi.py which isn't gpl, but okay to use. but otherwise from VectorMath I think is good standard gpl block

[wclyffe]

I had noticed about voronoi.py Also, is there any particular reason for the presence of if __name__ == "__main__": register() at the end of each node file?

[zeffii]

it's not in each file :)

[zeffii]

those lines are used for loading the node in text editor and testing, but I prefer to avoid them.

[wclyffe]

Ok. I was confused as I didn't see a case where it would be run directly from the interpreter So what do I do with those lines?

[zeffii]

Ditching those lines gets my vote.

[nortikin]

init.py has GPL description

#  ***** BEGIN GPL LICENSE BLOCK *****
 #
 #  This program is free software; you can redistribute it and/or
 #  modify it under the terms of the GNU General Public License
 #  as published by the Free Software Foundation; either version 2
 #  of the License, or (at your option) any later version.
 #
 #  This program is distributed in the hope that it will be useful,
 #  but WITHOUT ANY WARRANTY; without even the implied warranty of
 #  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 #  GNU General Public License for more details.
 #
 #  You should have received a copy of the GNU General Public License
 #  along with this program; if not, see <http://www.gnu.org/licenses/>
 #  and write to the Free Software Foundation, Inc., 51 Franklin Street, 
 #  Fifth Floor, Boston, MA  02110-1301, USA..
 #
 #  The Original Code is Copyright (C) 2013-2014 by Gorodetskiy Nikita  ###
 #  All rights reserved.
 #
 #  Contact:      sverchok-b3d@yandex.ru    ###
 #  Information:  http://nikitron.cc.ua/sverchok.html   ###
 #
 #  The Original Code is: all of this file.
 #
 #  Contributor(s): Nedovizin Alexander, Gorodetskiy Nikita, Linus Yng, Agustin Gimenez.
 #
 #  ***** END GPL LICENSE BLOCK *****
 #
 # -*- coding: utf-8 -*-

[nortikin]

With lines in name mein register do nothing, because i use editor in blender, it is cousy for my to use that way. Have question. init.py in subfolders may be empty and it works? realy?

[zeffii]

yeah, empty __init__.py is how it works

[zeffii]

I don't think the collaborators line needs to be in all node files

[wclyffe]

About __init__.py : basically it tells python that the folder is a package. And what's in the init file is executed on import

More about this python magic:

• Python Module/Package docs
• Python project structure

[nortikin]

@wclyffe thanks, dude :+1:

[nortikin]

you forked this from some commit, that is changed from that time. how we could commit your changes? did you deleted files or renamed in git with direct commands? will it break all that we need manually change files? than we should weld fork as soon as possible to prevent many job in it after

[nortikin]

is it realy works?

[wclyffe]

It should work. I took great care to rename and move the files with the git mv command. So git knows which file has been renamed to what.

I just checked and I'm currently five or six commit behind your master branch. So not so far behind. I think I will pull your commits in my refactor branch to stay up to date. As to the excact procedure to merge back, I will have to investigate a bit. But it should work.

I have almost finished the cleanup on all the nodes files, and I will make a commit once that part is done. I found a dozen or so 'imports' that I had missed on the first pass.

[nortikin]

so, i propose to wait for your pull request and not change much in code. @wclyffe @ly29 @zeffii @Cfyzzz it concern everybody, do less this days and maybe concentrate on particular nodes. i.e. vector math node for zeffii. i will just stop commit to make life easy to @wclyffe

[zeffii]

@nortikin I understand what @wclyffe is doing and how, this is a great (inevitable) development.

will hold back from doing commits until merged.

[ly29]

I will hold off for now, I don't have time anyway :)

[wclyffe]

I have almost finished. But I just need to check some references to StringsSocket. In some nodes there's some typecheck: if type(..) == StringsSocket Is this the StringsSocket from node_s , or the one from bpy.types once it has been registered ?

[nortikin]

@wclyffe please, make that all in sverchok folder will be in root folder. ok? it is all about upgrade of sverchok. we, of course can make moving folders, but if you could.

  File "/home/nikitron/.config/blender/2.70/scripts/addons/sverchok-master/__init__.py", line 66, in <module>
    import data_structure
ImportError: No module named 'data_structure'

[nortikin]

@wclyffe StringsProperty is from node_s. bad name indeed. it is not bpy.types.StringProperty. liter s

[wclyffe]

So in util.py:

def get_socket_type(node, inputsocketname):
    if type(node.inputs[inputsocketname].links[0].from_socket) == bpy.types.VerticesSocket:
        return 'v'
    if type(node.inputs[inputsocketname].links[0].from_socket) == bpy.types.StringsSocket:
        return 's'
    if type(node.inputs[inputsocketname].links[0].from_socket) == bpy.types.MatrixSocket:
        return 'm'

the type check is against the already registered class, but not in the nodes?

Concerning the folder structure: The files have been moved so that the sverchock subfolder is now the addon folder, so as to not bring all the auxilliary files (docs, scripts templates, readme in the blender addon folder). And I have not updated the readme to inform on that change...

For the update mechanism I noticed you check against a version number in a file Couldn't you make tags on the repository and check that instead? Using the master branch for stable releases, tagging those releases, and then develop in a dev branch?

[zeffii]

@wclyffe, it might be the other way around. If we had branched development, then naturally the update mechanism would be able to facilitate more options. like update to dev if adventurous or update to stable if cautious. Then also collect the commit log and write them to a Blender Text file.

Regarding util.py that's @nortikin and @ly29 territory.

[nortikin]

@wclyffe , get_socket_type() is part of adaptive sockets. in util.py ther are sections of file, that separated by magic comments-titles. you just can see what part is what. no need to separate util, mostly util is for deal inside nodes, but update section is separate by the way cache magic with temp_handle is of object in node first update magic is unused, old case of magic bmesh magic is something, i think, old too, but we can preserve in util

all magic subscribed

[nortikin]

ofcourse we can beat all to separate files, if needed, but much of workk have to be done in nodes themselves

[wclyffe]

It seems to me that everyone developping their own feature in parallel and then commiting simultaniously to the master branch could lead to trouble. Personnally, I branch to develop something, be fairly confident that it works before merging it from master. But I don't know what kind of workflow you have established around here. For example merging my refactor branch straight from master... I'm not 100% certain that I didn't introduced some bugs or unintended behaviours

[wclyffe]

@nortikin So you are positive that the StringsSocket in the node files is from node_s? I was not sure : having looked in util.py I saw it came from bpy.types and with the from node_s import * and other from foo import * in the node files I was not 100% positive about it

[nortikin]

yes, import * is wrong way, but till now works ok. Last time we try add sv_ as prefix and we telling each other what we are doing. brenching is good and i wish to go that way, but we ok with master. maybe yes, development brench will be better.

Strings is not String. I positive Strings from node_s and String from bpy.types

[zeffii]

So far we've been working on our own interests and we kind of stay out of each-other's current area unless it's a joint effort. No formal enterprise strategies, it's a small - aware - team.

[nortikin]

yes, situations when needed own brenches not obviouse when we doing locally at home computer and test there

[wclyffe]

I know there's no need for formal enterprise procedures. I just wondered on how you worked. Once I read the upgrade code, it became clearer.

[zeffii]

It seems we all agree about having the separate branches. The fact that it has been put off until (possibly) now, suggests that we are more motivated to make new nodes and fix/update existing ones than the task you have set yourself.

[zeffii]

I may switch over to your branch in a separate Blender install, smell the roses.

[wclyffe]

@zeffii I certainly could see for myself that the motivation to make new things was there, given the pace of development. Before making any new node, the first thing I had to do was understanding what's what in the code. And here I was with a hundred plus files in the root folder. Seemed to me that the project had come to a point where a bit of structure was needed. So yeah, it is not the most glamorous task, but in the process of doing that, things are becoming clearer to me. I hope, when all is said and done, that it will be an easier code base to apprehend.

@nortikin I had a look at the upgrade code, but I won't fix that now, as extracting to nested subfolders doesn't seem to be straightforward. I have make some tests before.

[wclyffe]

@zeffii I would wait till I commit what I have done since, as I had missed several imports. Before I pep8ed the files, my editor was like a christmass tree and it was difficult to pick out the undefined names

[zeffii]

@wclyffe yeah, the project was arriving at a tipping point -- that's a correct assessment. Can you imagine that it was bootstrapped from Blender Text Editor? ... no real autocomplete, intellisense, pep8, convenience editor shortcuts.

[nortikin]

@wclyffe collaborator now, can comit your brench to nortikin/sverchok here

[wclyffe]

@zeffii Coding that sort of complex addon from within blender is some achievement. But nowadays, I personally wouldn't code anything from within blender except for extremelly trivial things. It is great for trying stuff, especially the console. Usually I just make a link to my dev folder in the blender addon folder and edit externally

[wclyffe]

@nortikin many thanks I still have to quash some bugs before i can commit. Still, I will make a pull request from my fork. Since I work locally on a fork clone, I not sure I can commit directly to upstream from it, but I may be wrong

[wclyffe]

@nortikin @ly29 @zeffii @Cfyzzz There's now a new refactor branch in which I merged what I have done.

• all your commits have been merged, plus one from me for a stupid mistake...
• the addon folder is not the root folder anymore, it is the sverchok subfolder
• upgrading from within blender currently doesn't work
• chances are pretty high that some bugs are crawling in there

You're welcomed to test and report any problem, suggestions, requests that you might have.

[ly29]

@wclyffe Thank for your effort of organizing the mess we created.

Regarding the Stringssocket is the same, unless someone else registers the class within blender which open for world of problems since all class inside of blender lives inside of flat namespace in bpy.types ..., it has affect us in the past once or twice. The amount of classes inside of bpy.types is scary. In my current blender setup.

>>> len(dir(bpy.types))
3462

Which is why all new nodes/classes should have the Sv prefix...

Notes: I think the script templates/examples should be distributed with the add-on, perhaps they shouldn't be inside the script folder but keeping them there makes sense. There is some functionality that isn't quite general or solved enough yet that is in there, it is the add on to the add on so to speak.

I am not software developer and Sverchok is a optimistically growing hack. There are many issues that should be resolved. But the modularity of the node approach makes it surprisingly powerful.

[ly29]

I get the following error.

  File "~/blender-2.70a-linux-glibc211-x86_64/2.70/scripts/modules/addon_utils.py", line 299, in enable
    mod = __import__(module_name)
  File "~/.config/blender/2.70/scripts/addons/sverchok/__init__.py", line 75, in <module>
    'nodes.{}'.format(category))
  File "~/apps/blender-2.70a-linux-glibc211-x86_64/2.70/python/lib/python3.3/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "~/.config/blender/2.70/scripts/addons/sverchok/../sverchok/nodes/modifier/solidify.py", line 27, in <module>
    from sv_bmesh_utils import bmesh_from_pydata
ImportError: No module named 'sv_bmesh_utils'

Fix of couse:

from utils.sv_bmesh_utils import bmesh_from_pydata