Stream Objects

[zeffii]

I propose a testing Stream Object, we got to start somewhere and I have a suspicion that a composite object of numpy structures could be interesting.

import numpy as np

class np_Mesh():

    def __init__(self, py_data):

        v, e, f = py_data

        self.v = None
        self.e = None
        self.f = None
        self.f_lookup = None

        if v:
            self.v = np.array(v, np.float32)

        if e:
            self.e = np.array(e, np.int32)

        if f:
            flat, idx_lookup = self.processed(f)
            self.f = np.array(flat, np.int32)
            self.f_lookup = np.array(idx_lookup, np.int32)

    def processed(self, f):
        flat = []
        flat_extend = flat.extend
        idx_lookup = []
        idxs_append = idx_lookup.append

        idx = 0
        for poly in f:
            flat_extend(poly)
            begin = idx
            idx += len(poly)
            idxs_append([begin, idx])

        return flat, idx_lookup

    @property
    def verts(self):
         return self.v  

    @property
    def edges(self):
         return self.e

    @property
    def faces(self):
         return self.f, self.f_lookup

    # def face_walker(self, generator_func=True) :
    #      # recomposed non homogenous index lists.
    #      if generator_func:
    #          return a lazy
    #      else:
    #          return a proper list.

fv = [[0,1,2,3],[2,3,4,5],[6,3,2,3,4],[12,23,14,13]]

M = np_Mesh([None, None, None])

k = M.processed(fv)
print(k)

vflat, vlookup = k
for j, k in vlookup:
    print(vflat[j:k])

outputs:

([0, 1, 2, 3, 2, 3, 4, 5, 6, 3, 2, 3, 4, 12, 23, 14, 13], [[0, 4], [4, 8], [8, 13], [13, 17]])
[0, 1, 2, 3]
[2, 3, 4, 5]
[6, 3, 2, 3, 4]
[12, 23, 14, 13]

[zeffii]

And not build the nested-ness into the Vertex structure, instead add a control structure like index_lookup if there is special information about loops of vertices. No need to complicate an otherwise flat structure.

[ly29]

This is not how I formulated it my mind but more or less how I envisioned it to work.

[zeffii]

we need to get away from the complication of having something like this (not just for polygons.. but levels of vertices as output by some nodes)

  nested = [[v1, v2, v3],[v4, v5, v6],[v7, v8, v9]]

it should be (and is faster to store)

  flat_fast = [v1, v2, v3, v4, v5, v6, v7, v8, v9]
  vertex_groups = [[0,2], [3, 5], [6, 8]]

[zeffii]

storing the Matrix per np_Mesh would also be easy, and we could ask for the mesh coordinates to be premultiplied by it...

[nortikin]

recomposing of nested, how it looks?

flat_fast = [v1, v2, v3, v4, v5, v6, v7, v8, v9]
vertex_groups = [[0,2], [3, 5], [6, 8]]
objects = [[0],[1,2]] #from vertex_groups

for every stage we need additionally nested indexes, so it is recursion and it in plain list [flat_fast, float_x3_vertices, secondary_vertex_groups, objects] for case of additional level i.e. in plane generator (Separate function)

[nortikin]

or for generators it is by default to have additional level for UV separation? Lets call Separate function UV in redux to be clear. Now not every node deal with levels correctly.

[zeffii]

The only way for Numpy to be fast is using the simplest forms of data storage, we must devise a scheme which lets us describe complex relationships using perhaps several translation structures..

Best to talk about this using practical examples..

[nortikin]

class np_Mesh():
    def __init__(self, py_data):

        v, e, f = py_data

        self.v = None
        self.v_lookup = []
        self.e = None
        self.e_lookup = []
        self.f = None
        self.f_lookup = []

        if v:
            v_flat, v_idx_lookup = self.pydata_to_np(v)
            self.v = np.array(v_flat, np.int32)
            for v_look in v_idx_lookup:
                self.v_lookup.append(np.array(v_look, np.int32))

        if e:
            e_flat, e_idx_lookup = self.pydata_to_np(e)
            self.e = np.array(e_flat, np.int32)
            for e_look in e_idx_lookup:
                self.e_lookup.append(np.array(e_look, np.int32))

        if f:
            f_flat, f_idx_lookup = self.pydata_to_np(f)
            self.f = np.array(f_flat, np.int32)
            for f_look in f_idx_lookup:
                self.f_lookup.append(np.array(f_look, np.int32))

    def pydata_to_np(self, any):
        flat = [] # flat list of polygon indices (unfolded)
        idx_lookup = []

        k = 0
        for i in any:
            flat.exend(i)
            j = k
            k += len(i)
            idx_lookup.append([j,k])
        # allowing ngons to be represented by homogenous lists
        return flat, idx_lookup

    def np_to_pydata(self, flat, idx_lookup):
        depth = len(idx_lookup) - 1
        if depth == 0:
            pass
        else:
            any = []
            idx = idx_lookup.pop(-1):
            flat = flat[idx]
            any = self.np_to_pydata(flat, idx_lookup)
        return any

    @property
    def verts(self):
         return self.v  

    @property
    def edges(self):
         return self.e

    @property
    def faces(self):
         return self.f, self.f_lookup

    def face_walker(self, generator_func=True) :
         # recomposed non homogenous index lists.
         if generator_func:
             return a lazy
         else:
             return a proper list.

[zeffii]

interesting. warning , don't use any as a variable name... and store vertices as float32 not int32

[zeffii]

I like the idea that we can also include a way to easily serialize a np_Mesh object, for export or import and have functions like

def from_mesh_data(self, data, modifiers=False):
    ... populate all structures using obj.data 

def from_bmesh(self, bm):
    ... populate all structures using bm

[zeffii]

@nortikin I think I understand what you mean with the two data structures for v,e,f, and even am glad it is expressed verbosely at the moment. The potential problem is that by default py_data carries no information about vertex groups, edge groups. I think it is important that we pick a name for those concepts , but not layers..or groups.. maybe rings? - i think it always involves rings...

[zeffii]

import numpy as np

class np_Mesh():

    def __init__(self, py_data, vertex_rings=None, edge_rings=None):

        if py_data and len(py_data) == 3:
            v, e, f = py_data
        else:
            return

        self.v = None
        self.v_rings = None
        self.v_ring_lookup = None

        self.e = None
        self.e_rings = None
        self.e_ring_lookup = None

        self.f = None
        self.f_lookup = None

        if v:
            self.v = np.array(v, np.float32)
            if vertex_rings:
                self.make_index_tables(vertex_rings, self.v_rings, self.v_rings_lookup)

        if e:
            self.e = np.array(e, np.int32)
            if edge_rings:
                self.make_index_tables(edge_rings, self.e_rings, self.e_rings_lookup)

        if f:
            self.make_index_tables(f, self.f, self.f_lookup)

    def processed(self, nested_structure):
        flat = []
        flat_extend = flat.extend
        idx_lookup = []
        idxs_append = idx_lookup.append

        idx = 0
        for element in nested_structure:
            flat_extend(element)
            begin = idx
            idx += len(element)
            idxs_append([begin, idx])

        return flat, idx_lookup

    def make_index_tables(self, nested_structure, s1, s2):
        flat, idx_lookup = self.processed(nested_structure)
        s1 = np.array(flat, np.int32)
        s2 = np.array(idx_lookup, np.int32)

    @property
    def verts(self):
        return self.v

    @property
    def edges(self):
        return self.e

    @property
    def faces(self):
        return self.f, self.f_lookup

    @property
    def full_simple_mesh(self):
        ''' will be fine for most preliminary stuff anyway... '''
        return self.v, self.e, self.f, self.f_lookup

    def face_walker(self, generator_func=True):
        '''
        recomposed non homogenous index lists.
        returns stuff like [[0, 1, 2, 3], [2, 3, 4, 5], [6, 3, 2, 3, 4]]
        or the generator function equivalent.
        '''
        if not (self.f and self.f_lookup):
            return

        if generator_func:
            return (self.f[j:k] for j, k in self.f_lookup)
        else:
            return [self.f[j:k] for j, k in self.f_lookup]

fv = [[0, 1, 2, 3], [2, 3, 4, 5], [6, 3, 2, 3, 4], [12, 23, 14, 13]]

M = np_Mesh([None, None, None])

k = M.processed(fv)
print(k)

vflat, vlookup = k
for j, k in vlookup:
    print(vflat[j:k])

[nortikin]

def processed(self, nested_structure): not nested as i see, it is simple structure.

[zeffii]

fv = [[1,2],[3,4]]

this is nested

[nortikin]

ok, but fixed nestyness, without recursion rings. it will be good to define levels of nestyness (rings) automatically every time and write somewhere in sverchok data... if it is not too expensive for processing.

[nortikin]

than we can get class with

rings:[v,e,f], # integers of number
verts:np.array,
verts_rings1:np.array,
edges:np.array,
edges_ring1:np.array,
faces:np.array,
faces_ring1:np.array,
materials:{'Material.001':np.array,'Material.002':np.array}

materials will assign to vertices or polygons? i donno how it works now... hm.

[zeffii]

regarding materials.. it's not something I thought about much, I don't think one face can have multiple materials (else how would one define the mix ratio?)

therefore, perhaps the np_Mesh can have a material_bank, just like material_slots

self.material_bank = {-1: None, 0: 'material.001', 1: 'material.002'}
self.face_materials = np.array([0,0,0,0,0,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0], int32)

[zeffii]

for vector_colors, we have the opportunity to use several methods, side by side as they can all be useful

1 color per vertex
1 color per vertex per face
1 color per face

[nortikin]

yeas, color per vertex and per face needed.

[zeffii]

good point... can define per face!

[ly29]

We should introduce loops perhaps. Vertex color is each vertex of a face, not for each vertex. A vertex can have one color for each face that uses it.

[zeffii]

we can do it three ways, in the end Blender only does it one way (every vertex shared over multiple faces gets a unique colour per vertex reference per face)

There's no confusion here how that works

[ly29]

It is even a strength to have the different end results.

[nortikin]

simple additional np.array as R,G,B,R,G,B, and slide for polygons. what next? we defined object of data, sockets, maybe to describe definitions and classes of core? it is handy here mind map i guess

[zeffii]

Perhaps materials and vertex color maps can be considered at a later stage, i would like to not include them in the 'constructor' or __init__ a simple set of additional functions can be added vcols_type, vcols_add, vcols_update, vcols_clear, get_vcols materials and vertex colours. -- I think this is all too specific for an experimental model?

[zeffii]

something like

# the interface
def set_vcols(self, mode='face', flat_colors=None):
    if not flat_colors:
        return

    valid_modes = {'vertex', 'face_vertex',  'face'}

    self.vcol_mode = mode if (mode in valid_modes) else None
    if self.vcol_mode:
        self.vcols = self.make_vcols(flat_colors)

# the worker
def make_vcols(self, flat_colors):
    m = self.vcol_mode
    if not m:
        return

    if m == "vertex":
        '''each vertex gets a color'''
        ...

    elif m == "face_vertex":
        '''each vertex incidence on each face gets a color'''
        ...

    elif m == "face":
        '''each face gets a color'''
        ...

[nortikin]

@zeffii no need to complicate. vertex of face and nothing else, defined how to assign by special node - colorize. it will define what to colorize - polygon or vertex or vertex on exact polygon, but data is vertex of polygon. And I do not say materials by default is filled. By default it can be None, that. Class must have material and color, or additionally weight paint as vertex coeffitient, by default at first steps it really can not be used, but possibility must to be. Other is extendable.

[zeffii]

I think this is enough theoretical talk about this form of stream object, it's what i've had in mind for a while. But without trying it i'll never know if it's useful or not.

[nortikin]

hm... if we create object every time, than when we try same processes as in subrings we need to combine created objects in one? or lets do one object npmesh for scene, than we can from any layout work with one object and real objects will be as subrings... so when i for example use uvconnect (my lovely node) i have not to do many operations. For now this my node working only with two levels - objects and groups of vertices. Imagining how much work have to be done with this nodes i try make subject maximum clear to my mind. one object for scene?

[zeffii]

The idea is to do the following

• if an ' object ' / ' product of a generator ' has extra information about UV loops, then that information is passed along with the ' stream ', but it is not strictly part of the Vertices data structure.

This doesn't change much for the user.

But when will it happen? sadly I don't know. I'm preoccupied :(

[nortikin]

@zeffii if we operate with several objects, than np1 + np2 will create new object np3 in result, not modify old? or to choose between two?

[zeffii]

yeah, it doesn't have to modify old, or even create new!. For example common operations such as the reshape function doesn't create new values, but creates a new object with the references arranged differently (A different 'view' of the object). (and references have a smaller footprint than the objects / structures they reference).

Nodes that take several such np streams I think should leave the original data unchanged, just like is the case with current Sverchok, because the stream can be used as an input for other nodes expecting the original np stream.