Proposed refactoring

[wclyffe]

Hi guys,

First, great work on this addon as parametric modeling was sorely missing in Blender.

As I became intrigued and tried to wrap my head around what's possible, I decided to have a look at the code, which is growing at a frantic pace.

Anyway, in the process of trying to understand for myself how all that is working, I ended up forking and refactoring a bit: https://github.com/wclyffe/sverchok/tree/refactor The details of what have been done are in the commit log: https://github.com/wclyffe/sverchok/commit/a5d1b16f21c319cb555439534c383c6280f738ca

Essentially moving nodes in their own directories and changing how the nodes are imported in main init file. Apart from the imports the files contents should be untouched

On my end, all seem to be working as before, but I must say I haven't tried all the nodes...

Before I continue and begin to eventually refactor further, I would like to know if you are interrested, and if that is the case, some input would be greatly welcomed.

Again, you have done a great job and I certainlly hope that Sverchok will continue to extend what's possible with Blender

[wclyffe]

Some more explanations:

• release =~ tag =~ version, used instead of version file
• you already used versioning of releases as you use git
• sverchok.zip is a pure add on, for the users, not a git repository anymore
• to develop, you need to use the repository itself

[zeffii]

@nortikin Range Int and Range Float in number

[wclyffe]

@nortikin In nodes/list/

[nortikin]

@zeffii

class GenRangeNode(bpy.types.Node, SverchCustomTreeNode):
    ''' Generator: Range '''
    bl_idname = 'GenRangeNode'
    bl_label = 'List Range'

it is old range nodes.

@wclyffe i have to move file, than i have to change init in nodes folder and that is it?

[zeffii]

it is old range nodes.

ooh... I didn't realize..

[nortikin]

moved range nodes, forgot change init

[zeffii]

we will get comfortable with this , then full steam ahead :)

[wclyffe]

@nortikin You want to move nodes/list/range_int ? git mv oldname newname so when you do git status you should see remaned instead of deleted and add or new and yes, you should update the nodes_dict in the init file

[nortikin]

@wclyffe done, it works and it is main

[nortikin]

@wclyffe just count steps to install for old and new folred structure: old 1.download 2.install

new 1.download 2.unzip 3.install

maybe not so much steps... but locally version should be writed somewere - it is essential for upgrade checking. don't understand how it is locally presented. file version should stay to be able to check for new version. Or you propose other solution?

[ly29]

@nortikin It should be: 1) download 2) install, all the pices for making that work aren't ready yet however...

[nortikin]

@ly29 what about upgrade? how sverchok will find local version number and compare? that is question. or we should change version in two places (file version + tag of release) every release? or simply to move init.py to upper folder.

[zeffii]

we don't change version file anymore

[wclyffe]

@nortikin I know and I agree. I'm working on it. The first step was the package.py script which is producing the file that you can install directly without unziping it first. Next todos:

• automate the running of package.py, called when tagging
• change the update function:
  • the function will use the github API
  • check what is the last release/tag on github
  • grab the addon.zip associated with the tag
  • update in place

[nortikin]

@wclyffe one thing is not mentioned, i not sure you understand.

to see the last version on github with API is ok, i understand. But if the version in github is the same already installed? what you gonna do? we have to compare local version with github. how to define local version? autogenerate version file?

[zeffii]

the builder bot could place a commit tag file in the root of the zip, so locally you know what commit was the last in this zip, then that can be checked against online latest tag

[nortikin]

that what i wanted to understand. @zeffii

[zeffii]

should be doable, compared to a refactor is simple :)

[wclyffe]

@zeffii @nortikin yes, that could be done like that, but couldn't we use blender_info["version"] in the main __init__.py file ? I had not much time to spare today, but i have set up a test repo at https://github.com/wclyffe/travis-test Currently, it doesn't upload the zip file, but package.py is called correctly by travis-ci

[zeffii]

blender_info["version"]

sounds good, or some other key

[wclyffe]

@zeffii why the need for some other key? It already exist. The blender_info dict is already there and required by blender as some sort of addon manifest

[nortikin]

@wclyffe this is double job. every time we post new version we should change it... if it will be in one place it cool. maybe creating of new release will take version automatically from blender_info['version']?

[zeffii]

@wclyffe when I think of version, i dont immediately think of commit number. To developers, maybe that makes most sense, but to the users perhaps it makes sense to still preserve a version key in bl_info to state the general version (only to change on big or important additions/changes) and a secondary minor key commit which states the commit tag.

We don't need a new key, but if we want to preserve version and last_commit as separate things - it probably makes sense. Just like Blender has a version, and commit revision.

or am I missing something vital ? what would the 'version' value look like?

[wclyffe]

@nortikin @zeffii We could had a new key, but is it of any use for a user? The version a user will use is still a released/tagged one (and git associate a commit cheksum with it). Nevertheless, it makes sense in the case of a dev version. Anyway, it is nothing fundamental, just a detail.

In the mean time, I had a better look at travis-ci. If you are interrested, here's some more explanations:

• basically, it is a virtual machine (vm) on which the repository is checked out.
• it is ubuntu/debian, with, in our case, python in a virtual_env
• you can use pip to install python packages (pep8, flakes,...)
• you can also install packages from the ubuntu repos (perhaps blender itself?) or even compile stuff
• after this "setup" part, it runs what you instructed it to do
• as it is normally used for testing, it is called on every commit to perform tasks/tests and return a failed/passed status
• after all that is done, the vm is destroyed

The catch is that travis has read-only access to your repo. So it can't modify things directly. But it can be configured so that when you tag a release it perform a "deploy" action. There you have access to the building/testing "artifacts" before the vm is destroyed. One of those artifacts is the .zip produced by package.py. And you can authorize travis to upload this zip as an asset to the release.

[nortikin]

@wclyffe checksum? this is something new... than we have to implement password to python code to have access to ours repository from VM? any difficulties? because more i know from you about github, it becomes more scary and complicated for me XD :8ball:

[zeffii]

Agreed with @nortikin, it's probably easy once you've done it a few times -- but from the description alone it's not the most trivial thing. But for a big project, which Sverchok has the potential to become (or is already) the extra level of automation starts to become important.

Though I don't understand the exact details, this seems like the way to go. Friends in gaming industry swear Jenkins and Travis has improved their quality of life :)

[wclyffe]

@nortikin A commit checksum is for example a5d1b16. This is just a way for git to identify a commit in your repository. You can see it as the commit "name" A tag is a way to mark a particular commit as special. It is some sort of "bookmark" in the history of the code repository. Say you make a tag v1.1 on the commit a5d1b16. Now, you can tell others to download the v1.1 version. And later, if you want to go back to v1.1 it is easy, you have a bookmark.

@zeffii Yeah, it migth be a pain to setup. But once it is done, you can forget about it, it is working for you in the background to do the repetitive tasks. And then you can just concentrate on developping

[wclyffe]

I'm working to set it up on my test-repo. Once it is done, I will give you commit acces so you can test for yourself how it is working.

[wclyffe]

@nortikin about password: travis is made to work with github. It is a bit like an application on android, where for example, you give a calendar app access to your gmail calendar, contacts, ... The app can't modify your contacts but it can add a new event on your calendar. Just like in a smartphone app, github can give authorization to travis to upload an asset. The fact that it can't modify your repository directly is a good thing. There's no risks to the integrity of the repository.

[nortikin]

@wclyffe how you about integrating to authors stuff? i can write your name in authors, but you need to breafly introduce yourself. This is depending on you.

[nortikin]

After refactoring version should be at list 0.4.0, because we allready made much of stuff. maybe next: 0.5.0 for OpenGL in viewer draw improvement and text viewer improvements, 0.6.0 for panel with sliders in 3d for parameters, float and int and list in will support it with boolean, 0.7.0 for nodes colors, 0.8.0 for functions separate from nodes - interface and mechanism 0.9.0 for data structure (we should consider what it should be, it is important) or to leave as is for 1.0.0 1.0.0 for icons in OpenGL ind interface

[nortikin]

OK, we have troubles with users to upgrade. i propose next. temporerly use old upgrade function and in future use twice upgrade - stable releases @wclyffe proposed and unstable releases from master branch, that we have. for now i moving all files in root. just for commiting master branch. If you not agree, say now.

[zeffii]

so just this will be master branch?

[nortikin]

all in one. it is good when user can install from zip just downloaded other stable releases will be as usual. i commit and push now, for master. When @wclyffe will give his solution, we will change to his needs, but i propose keep twice upgrade - for stable releases and for current master. now it is for current unstable

[nortikin]

i used command git mv * ../

[zeffii]

add back the version file?

[nortikin]

for now yes

[nortikin]

looking for future, if we change git server, than we need to change upgrade. better to have two versions - current and for tight close to github API. Not see any conflict here. but only thing - am i correctly moved files? if not - i will deny changes and make other way. maybe file by file. how is it? did you puplled from github?

[zeffii]

nope, I have a simlink in scripts/addons/sverchok to refactor branch/sverchok I don't like working with multiple sverchok folders in my addons directory.

[zeffii]

on windows that looks like:

C:\blender_trunk\2.71\scripts\addons>mklink /j sverchok C:\Users\dealga\Document
s\GitHub\sverchok
Junction created for sverchok <<===>> C:\Users\dealga\Documents\GitHub\sverchok

This creates the sverchok folder in /addons, and links it to the folder (path shown above) in my Github directory.

[nortikin]

so. is it ok?

[zeffii]

yeah, works fine! we can stay like this - i don't mind. Every now and then merge refactor to master for stable releases.

[zeffii]

Realistically the most important part of the refactor is definitely done already.

[ly29]

I think one or two fixes might be needed, but they shouldn't affect it working, just remembering and applying prefs, preferences, but this was bug waiting to happen, it is an easy fix, but don't have time to test today.

What need to happen, from root __init__.py__

data_structure.__package__ = __file__

Or something that defines the name of the package. And in data_structure.py

Replace the hardcoded "sverchok" while getting preferences in some places. I was thinking about this earlier today.

[zeffii]

@ly29 replace hardcoded sverchok with the sverchok folder name?

[ly29]

yes, but it has to bet set from __init__.py or to get __package__ working.

[ly29]

__package__ doesn't work for some reason, but can be set from __init__.py where __file__ gives the right info.

[zeffii]

noticed @wclyffe is currently working on getting the last bits of the backend set up. https://github.com/wclyffe/travis-test

[wclyffe]

@zeffii @nortikin Sorry, I didn't had time to finish that sooner... I added you two as collaborators on https://github.com/wclyffe/travis-test It seems to be working now.

• on commit, travis does currently nothing, appart from running package.py
• on release/tag : travis add a foo-package.zip on the latest release (follow the link in the readme, it is the green button)
• to try that, you can commit something, and then make a new release
• it may take a few minutes before travis has done its job. You can check the build status at https://travis-ci.org/wclyffe/travis-test/builds
• i also added a update mechanism. If you download the zip and extract it, you can update in place by running update.py on the command line (if there's has been a new release since)
• to quickly try the update.py, you can edit the release_info file (which is added to zip while packaging) and change the date to 2013 for example, delete a few files (not update.py or release_info obviously) and then run update

[nortikin]

it seems like easy to update now. will look tomorrow and try