Proposed refactoring

[wclyffe]

Hi guys,

First, great work on this addon as parametric modeling was sorely missing in Blender.

As I became intrigued and tried to wrap my head around what's possible, I decided to have a look at the code, which is growing at a frantic pace.

Anyway, in the process of trying to understand for myself how all that is working, I ended up forking and refactoring a bit: https://github.com/wclyffe/sverchok/tree/refactor The details of what have been done are in the commit log: https://github.com/wclyffe/sverchok/commit/a5d1b16f21c319cb555439534c383c6280f738ca

Essentially moving nodes in their own directories and changing how the nodes are imported in main init file. Apart from the imports the files contents should be untouched

On my end, all seem to be working as before, but I must say I haven't tried all the nodes...

Before I continue and begin to eventually refactor further, I would like to know if you are interrested, and if that is the case, some input would be greatly welcomed.

Again, you have done a great job and I certainlly hope that Sverchok will continue to extend what's possible with Blender

[ly29]

The bpy.utils.register_class puts the class in bpy.types. But the bpy.types path is specified as an ugly way to work around the problems you have been solving with proper import structure instead of from module import *

Also get_socket_type should be updated to use bl_idname instead s m v to make more it future proof and to simplify that code, anyway that change is needed for more socket types, but since I didn't write that code I havn't spent time to change since it has been working for now.

It should be like this in the future.

def get_socket_type(node, inputsocketname):
    return node.inputs[inputsocketname].links[0].from_socket.bl_idname

[ly29]

Other problems are with Formula2 node that also abused the import system by dumping the whole math module into local namespace to make sin etc available.

[zeffii]

node_script_templates folder definitely needs to move into the sverchok folder, it is relied on in the Text Editor menu plugin and the node_Script node.

https://github.com/nortikin/sverchok/blob/refactor/sverchok/utils/text_editor_submenu.py#L25-L28

https://github.com/nortikin/sverchok/blob/refactor/sverchok/nodes/generator/script.py#L87-L94 and in https://github.com/nortikin/sverchok/blob/refactor/sverchok/nodes/generator/script.py#L160-L167 too

[zeffii]

i propose to expose a path string through sv_tools.py or whatever , sv_path to avoid the hassle of path finding like this

def dirup(path, n):
    for i in range(n):
        path = os.path.dirname(path)
    return path

sv_path = dirup(os.path.realpath(__file__), 3)

sv_path would be constant for that session.

[zeffii]

Traceback (most recent call last):
  File "C:\blender_trunk\2.71\scripts\addons\sverchok\utils\sv_tools.py", line 52, in execute
    speedUpdate()
  File "C:\blender_trunk\2.71\scripts\addons\sverchok\..\sverchok\data_structure.py", line 1001, in sp
eedUpdate
    do_update(itertools.chain.from_iterable(update_list), ng.nodes)
  File "C:\blender_trunk\2.71\scripts\addons\sverchok\..\sverchok\data_structure.py", line 958, in do_
update
    nods[nod_name].update()
  File "C:\blender_trunk\2.71\scripts\addons\sverchok\..\sverchok\nodes\vector\interpolation.py", line
 147, in update
    out = eval_spline(spl, t, t_corr)
  File "C:\blender_trunk\2.71\scripts\addons\sverchok\..\sverchok\nodes\vector\interpolation.py", line
 85, in eval_spline
    n = bisect.bisect(tknots, t, lo=0, hi=len(tknots))-1
AttributeError: 'function' object has no attribute 'bisect'

[ly29]

Check that it imports bisect from python and not the bisect node.

[zeffii]

yeah it's importing from modifiers

[zeffii]

https://github.com/nortikin/sverchok/commit/dfa371e276238dde0fab7b3abadf4db5e26e6843

[wclyffe]

@ly29

• sv_bmesh_utils : I tought I had fixed that one with a quick commit immediatly after creating the branch. Are you using the refactor branch from this repository? If not you should be using this one and not my fork
• get_socket_type : I will have a look
• formula2 : doesn't seem to import math anymore. On the other hand, math_scalar still from math import * as it was somewhat legitimate, but could be changed

@ly29 @zeffii

• bisect : I clearly misundertood, thanks for the fix
• templates : I thought about moving it, but wasn't sure where. How about a text_editor subfolder in utils, with texteditor(plugins/submenus).py and a template subfolder?

[ly29]

@wclyffe You are right, I looked at the wrong fork, my fault.

• Scalar Math node should be made to import explicitly even though it is somewhat legit since uses almost the whole math module. But it isn't critical.
• get_socket_type this should be restructured, could be a lot simpler if it used bl_idname directly...

https://github.com/nortikin/sverchok/blob/refactor/sverchok/data_structure.py#L1028-L1085

[wclyffe]

@ly29 about bl_idname : if i'm not mistaken, that's what goes into bpy.types flat namespace. If so, bl_idname should follow some sort of name convention, or at least be prefixed to avoid some future potential headaches. By the same token, the nodes class names should follow suit. what about, with Foo being the category, Bar the node name :

• SvBarNode or
• SvNodeBar or
• SvFooBarNode or
• SvNodeFooBar

[ly29]

@wclyffe It most definitely should be like that. We started with Sv prefix once we learned about the problem but havn't been strict enough about it... Doing a big rename offering an inplace upgrade system is the right solution but before that is really workable we should split the actual processing from update function since that is a function meant for ui update events. That is the next big task I think, it is becoming urgent.

[wclyffe]

@ly29 And the code concerning ui udpate would be: https://github.com/nortikin/sverchok/blob/refactor/sverchok/data_structure.py#L682-L1010

[zeffii]

node_script_templates is used in 2 different places, I prefer a folder for ../node_scripts/templates which refers to what they are used for rather than where they are used, not a text_editor subfolder.

[ly29]

That is code for ensuring that nodes update gets called in right order. The order those functions generate can be used in the future. Blender calls the nodes in order they where created by default so we have impose our own system to ensure valid execution.

[zeffii]

what about

sverchok
sverchok/node_scripts/
sverchok/node_scripts/templates/ <-- this is intended to get newcomers up to speed
sverchok/node_scripts/user_names/  <- we keep our own prototypes here

fits logically with

sverchok/..
sverchok/nodes/..
sverchok/node_scripts/..
sverchok/utils/..
..etc..

or sverchok/nodes/scripts/node_scripts/..

[nortikin]

PLEASE!!!!! make working init.py lay on root folder. i want user to download Sverchok from github and install as usual addon. PLEASE

[zeffii]

the way to go might be an Installer addon. to place the sverchok folder inside /addons one time, then the addon can be updated similar to how it worked before the refactor but with option to update to dev version or only to new stable versions.

[wclyffe]

@nortikin I can and will do that if you insist, you are the owner of the addon after all. But first, let me explain why I did package the addon like that:

• it doesn't change much for the user
  • a user still has to download the zip from github
  • a user still has to first unzip the archive
  • the only change is that instead of copying the entire folder, only the sverchok subfolder has to be copied/moved in the blender addon subfolder
  • if a user is capable of using blender/sverchok, that shouldn't be hard, if explained on the readme on the frontpage.
• now, for the advantages:
  • it allow to have others things that have nothing to do in the blender addon folder to live and be tracked in the same repository.
  • for example, the documentation. As of now there's not much, but I think a future documentation effort could be included in the docs folder.
  • another folder for presets/nodegroups could be in the root too
  • another folder could be for example .blend files
• with this refactor, I was trying to think ahead, to not have to refactor soon in the future.
• most python modules are packaged this way. In fact, the Luxrender blender addon, luxblend, is packaged this way, with a preset folder.

I understand your concern, and will do as you wish.

But you should think about it. It might be marginaly more convenient for the user, but a deal more painfull to develop and maintain in the future. An alternative, might be to have a 'distribute' folder in the root, with pre-zipped files. Then those could be hyper-linked directly against.

[zeffii]

fetching a zip and only unpacking certain folders is straight forward iirc.

[wclyffe]

@zeffii I thought a bit about the updates : stable or dev Say the master branch is tagged for the stable releases, then dev is just the tip/head of the branch. I had a look at the github API and it allows to query for tags and return a direct link to a zip/tarball of the chosen tag

[ly29]

@wclyffe A common workflow is to install the zip directly from github. Just download and install directly from blender add on settings, it isn't what the zip is meant to be but it is very convenient... As long as there is simple way to download Sverchok as a install ready zip file it is all good I think.

[zeffii]

An alternative, might be to have a 'distribute' folder in the root, with pre-zipped files. Then those could be hyper-linked directly against.

we'd need to automate the zip procedure when current branch has been tagged as stable, a cron job for a server bot maybe.

[wclyffe]

@ly29 The possibility to install directly from blender settings by pointing to the zip did escape me.

My point was more about what the repository is meant to be : it is primarly an environment in which to develop. Distributing the addon with all the added cruft developper might need, in my mind, is not optimal.

A solution to this dilemma, might be to have a distribute folder and a package script to produce the zipped, installation ready file and point to that as a save as link in the readme (which now appear on the front page without scrolling)

[wclyffe]

@zeffii Travis-ci.org offers that for open-source project I think

[zeffii]

@nortikin I think @wclyffe has very good points, good opportunity to do this the correct way now, more convenient for development. Our users will be bright people..

[ly29]

@wclyffe I agree about what the repository should be, the way to produce the install ready zip sounds good to me. :+1:

[zeffii]

once we document the installation process changes, and implement the update mechanism again all people have to do is press update anyway.

I could make a simple Sverchok installer addon, as a bootstrap device to help simpliiy the first install

[wclyffe]

I didn't anticipated the changes to happen that quickly... This refactoring is still clearly unfinished. Some things are currently broken and that was to be expected. Changes are needed and we migth as well take our time to discuss and find a solution that works for everyone, users and developpers

[wclyffe]

I investigated a little bit on how to produce the addon.zip. Github has a release feature, including changelogs and binary assets (i.e. addon.zip), as well as an API to query for those assets. By hooking the repository with Travis-ci, a build service integrating with gihub, we could produce the assets in question automagically when taging a release. In the future we could even compile docs.md files to a nice pdf. Some info on all that:

• Github - Release your software
• Github - Release API
• Travis - GitHub Release Uploading

[zeffii]

I've thought about this for a bit now, and have made small changes locally to fix nodes/generator/script.py i need node_script_templates (or some variation of this name) to be inside sverchok folder somewhere. My preference goes to sverchok/node_scripts/templates/, any objections?

[wclyffe]

@zeffii Not from me. Do you want me to move it there? Or will you move it?

[zeffii]

If you could please! node_script_templates to sverchok/node_scripts/ as subfolder templates so sverchok/node_scripts/templates/ preserving for now the subdir structure of username-folders and .py files.

Maybe it's time to setup my local Git correctly for this project :)

[zeffii]

btw, the addon.zip way is going to be awesome :)

[wclyffe]

@zeffii

• The node_script_templates folder has been moved.
• the addon.zip way : Once setup, it should work nicely!

[nortikin]

@wclyffe

• zip file can be installed from blender user prefs directly. and we not need make release each time to provide new verion for user.
• the way user think is not always as you mentioned. As i remember, i started blender from simple solutions and addons installing was not obviouse. More over, if you make your own way to generate installer - it may confuze people
• propose to make development brench (now is refactoring brench) and to make merges to master every day or something. But when merging it should be merged folder sverchok as root, not as in development brench. or realy to make new versions every day...
• we have requests often and if releases allows us to make different folder-structures in zip, than it is ok. The name of released zip is critical in this case.

[nortikin]

so, development brench ==> refactoring/sverchok/nodes master brench ==> master/nodes

this way i guess or not do it too complex and put init file to root, because if we change something, we eny time could put it back. I afraid, releases haedake will be long-lasting or take too much time. Now it works well for users and .gitignore file there

[zeffii]

• yeah, we only need one addon.zip called 'sverchok.zip'
• sverchok.zip would be located in the root of the sverchok project.
• sverchok.zip would be built automatically by a bot whenever we make a new stable tag on the repository
• sverchok.zip can be dragged by first time users to desktop and in blender the user can point at that file and install from there.
• Once installed user doesn't need to worry about any of that any-more. Upgrade will be reimplemented and offer option to upgrade to head (current snapshot, "dev build") of repository or last stable (tagged stable, "stable build")

Good documentation will explain this, and I will make sure of that.

[nortikin]

@zeffii how you will handle one name everytime? with overwriting of old zip file?

[nortikin]

@zeffii i tryed now, it warning me Duplicate tag name so, no way to make one-name release

[nortikin]

@zeffii or we can delete release every time... maybe and create new one manually... isn't ti headake? when it was zip archive on site, that i manually updated, i found that i am too lazy to create it often. with master git it more often. or let it be. ok. first time user will download, make folder copy-paste and so on. Ok, i dont mind. in same time i don't see real benefits of init file in subfolder.

Or, could we make init file as soft link to subfolder of real sverchok? that way we could make user install with zip and in the ither way install subfolder itself... am i right?

[zeffii]

@nortikin - it seems like a headache, but it won't last long. @wclyffe seems to know what he's doing.

As far as I understand, Travis cl.org "archiver bot " can name the archive anything we need. We force it to name the archive sverchok.zip. it overwrites the existing archive whenever we tag a snapshot as stable. This is an automated process, because yes we are lazy to update .zip ourselves, and there is a framework exactly for this situation.

@wclyffe can explain better -- sorry if i'm making things confusing. The end result is simple, but the process can be difficult to explain.

[nortikin]

@zeffii yes, maybe i didn't understand it right.

Sverchok not loading and not reporting anithing in terminal... how to find what is what there? manually searching?

[zeffii]

@nortikin look in the enabled list, it will say something like 'sverchok-master' not found. untick it. download sverchok-master, drag subfolder 'sverchok' into addons. then enable. works here. (this is what we do until we have an sverchok.zip archive in the sverchok root.

[nortikin]

sorry, i was changed stupidly sv_tools.py file that is why it didn't registered

[zeffii]

Okay! cool.

[wclyffe]

@nortikin I just pushed a packaging script (run from the command line) which produce a sverchok.zip (in the distribute folder) installable from within blender. The way it could work:

• Tags - version numbers - (i.e v1.2.3) are used on the master branch to signal a release -- instead of incrementing it in the version file
• Github create a release for this tag. Just like you have done : https://github.com/nortikin/sverchok/releases. If you click on the release link, you can see the details, with links to download the source code. If you then click the edit button, there is a zone where you can drop any file to be associated with the release.
• You can associate the shervok.zip file a file with the release and it will appear on the release page, just like the link for the source code.
• For now, it is in the distribute folder.
• But the script I pushed can be ran automagically when you make a tag on the master branch and associate the file to the release.

So all in all:

• you wouldn't have to worry about making an addon.zip file each time you make a tag
• you can use the github webAPI to check for a new version (tag)
• you can use the webAPI to get the link to the asset associated with the new release
• and then update like before

I know this is confusing. And it is a work in progress, that's why I made a refactor branch, to be merged when ready with the master.

[nortikin]

why it making screenshots? or is it only with me?

[nortikin]

ok, thanks

[nortikin]

where are new range int/float nodes? in what folder?