The Rewrite: BB2 Redux

[zeffii]

Over the next couple of days (unless something unforeseen happens) I have time to make a start with building the add-on from scratch. Getting it to match the working features of BB2 will take a while. It's difficult to put a time frame on it until the process gets rolling.

The decision to start from scratch is not done in haste. I think some of the responders on this repository have expressed similar feelings about how to proceed. There's lots that can be taken from the current code-base, but a lot that I really want to sanitize and arrange into separate files. The core of this was written when Blender 2.5 was still young and the API not very well defined, and prone to convoluted solutions.

If I can reach feature parity and structure the code better then I will consider it a success, speeding up the import (and avoiding recursive updates) is not my first priority, making it easier for others to contribute is far more important.

Ticks indicates functionality seems to be restored for that panel, certainly report any issues with those sectons. The unticked sections have had no major checks done other than suppressing import errors. The section labelled next is either being worked on or that's where the most recent code changes have taken place.

• [x] gui_panel_ep.py
• [x] gui_panel_mlp.py
• [ ] gui_panel_nma_viz.py
• [ ] gui_panel_output_animation.py
• [x] gui_panel_pdb_import.py
• [ ] gui_panel_pdb_output.py <---next
• [ ] gui_panel_physics_sim.py
• [x] gui_panel_view.py

[zeffii]

The plan. (might change as it progresses, and is not chronologic)

• [x] All new code must be pep8 compliant, with the exception of the following error types --ignore=E402,E702,E501,E731,E241 (Use a pep8 linter!, else thyne code shalt not pass!). If you use tabs instead of 4 spaces for indentation you will be slapped with a fish.
  • pep-20 / Zen of Python helps you decide how to do things.
  • http://docs.quantifiedcode.com/python-anti-patterns/index.html
• [ ] Generate the default scene and objects using a script, unless it really can't be done any other way
• [ ] Generate new documenation folder with layout something like

BioBlender/Manual/images/01_01_Intro.png
BioBlender/Manual/images/01_02_Intro_p2.png
BioBlender/Manual/images/02_01_Install_Guide_step1.png
BioBlender/Manual/images/02_01_Install_Guide_step2.png
BioBlender/Manual/images/02_01_Install_Guide_step3.png
BioBlender/Manual/Introduction.md
BioBlender/Manual/Installation.md
BioBlender/Manual/Feature_01.md  <-- replace number with shortname for feature
BioBlender/Manual/Manual.md  <-- would link all .md files together.

• [x] Separate operators into own files
• [x] Separate panels (ui) into own files
• [x] Separate Tables
• [x] Separate Utility code
• [ ] Employ OS agnostic python tools to deal with the double quote mayhem ( there's a dedicated core python module for it shlex
• [ ] Create disposable non rigged import for fast viewing (already coded..not merged)
• [ ] Parent the EP Curves to mainting outliner sanity. Make sure it works with animated EP

[MonZop]

Great plan! I am certainly willing to take care of the documentation folder (help always welcome, of course, but if you are a coder it might be more useful in other places), which is the only part that I understand, almost. uh

[zeffii]

I like a fresh start, unfortunately there won't be anything to test for a while.. it's like a fetus now

[zeffii]

I made a new branch and split the panels into their own source files. This is no-where near working, but it's necessary until gradually things start to work again. It is my hope that I will have useful progress before the replacement dependency graph (aprox two weeks from now according to the developers meeting).

It has been difficult to find motivation to do this lately, but i'd like to think I can finish what I started.

[zeffii]

panels are now separated, and load in the correct sequence. <still non functional>

now to start linking up the global storage dictionaries again. This is the part I'm not terribly sure what kind of time it will take. But already in the last 2 hours this has gone faster than I anticipated. Back to coding.

[MonZop]

Good to hear things are going! Just let me know whan (and if) I can do something, like testing

[zeffii]

it would help me a lot if you found time to learn how to use GIT from the command-line, or find someone to teach you. Even if you aren't coding anything it helps if i'm confident that the files you are testing are indeed the branch i'm working on.

There are only a few commands that anyone needs to know

git pull --all
git push --all
git add --all
git commit -a -m "commit message here"
git checkout -b <branch_name>  # new_branch_name_based_on_current_branch
git branch -D <branch_name>
git branch

[MonZop]

OK, I'll make an effort, (find someone to help me tomorrow, if available). I already know that once learned, it will be easy, but i've been lazy, and had also much to do. This Sunday (10 of May) I'm going to Munich to the Premiere of the film The Dark Gene, which contains several minutes of an animation made by us using BioBlender! Munich DOK.fest I'm looking forward to see how the poublic will react at those images! (I mean, besides Blender people, and biologists)

[zeffii]

so far so good, but won't know how easy the rest will be until the next push.

Cool, a premiere! Must be great to finally see your work in context, enjoy!

[zeffii]

This is much easier to navigate now. wow.

I think the import panel has reached feature parity with older versions, but it has not been tested extensively yet. The import panel is 99% existing code, and small changes to make things Blender version agnostic. I've replaced boolean and integer global variables with scene specific properties which are by definition global. Added aliasing to avoid 'dot lookups' where possible.

Time to continue to the next panel. MLP. because I know it a little bit.

[zeffii]

mlp surface works now:

still too rough for testing..

[zeffii]

render mlp to surface now too:

[MonZop]

Beautiful! Congratulations! I have also tried (using the redux_s3_FIRE) , and got the MLP on the surface, but not the render. I also have a question: is this MLP the value calculated by MLP.py, or is it the color of pyMOL, simply transformed to grey scale, and rendered?

[zeffii]

I have also tried (using the redux_s3_FIRE) , and got the MLP on the surface, but not the render.

i'll look at it. The code feels a little like this now : screenshot from 2015-05-08 14 46 23

there's a conversion function that strips out the colour, will look carefully at what it does.

[MonZop]

WOW, it makes me feel dizzy with vertigo!

[zeffii]

At present:

I haven't yet traced manually what they do, technically.

[zeffii]

(wrong button pressed closed the thread)

here's the abstract of those two functions, ps: this might take a while to fully write and will edit this post over the course of hours/days. I have made no functional changes to any of this code, except minor formatting (may 2015).

bb2_operator_mlp()

In theory:

• expects user to select an atom of a model, any atom.
• if multiple atoms are selected, its looks to see if the atoms are from one .pdb or multiple
• for every pdb model identified by the selection it creates a tID variable
  • perform the mlp(tID) function
  • todoAndViewpoints()

bb2_operator_mlp_render()

• requires the surface, (or surfaces if multiple) to be selected..
• it finds the selected PDB ids and gives them an ID, tID (the ID is used for internal reference, it doesn't reflect the pdb file name..)
• for every found tID,
  • call mlpRender(tID)
  • todoAndViewpoints()

mlp()

..todo

mlpRender()

..todo

[zeffii]

While reading the operator code, I'm wondering if it wouldn't be tidier if there was a UI list, which shows the currently imported models, and lets the user select them from there, rather than the atoms. I'll read up on how to do a UI list, it's something i've wanted to know for a while anyway.

[MonZop]

Several issues here:

• A list of objects (elements from one or more pdb files) would be great. I thought that this could be done (more) easily in the Outliner, so that every object would be associated with its atoms, its surface, its UV map (from MPL calculation) and its set of lines (from EP).
• BB2_operator_mlp() is supposed to:
  • get the molecular surface
  • run pyMLP.py using data from the pdb file, and fi values stored
  • use the .dx grid generated by pyMLP.py to attribute the value to every vertex of the surface
  • convert those values to grey scale
  • build a .obj file that lists the vertices as x,y,z and mlp
• BB2_operator_mlp_render(), will use the surface, that now has a UV map, and apply the render procedure that makes it rough/dark/dull or smooth/white/shiny according to the mlp value.

Hope this hleps!

[zeffii]

on reflection yes, using the outliner alone is fine for atom selection

i'm not sure about this question:

I also have a question: is this MLP the value calculated by MLP.py, or is it the color of pyMOL, simply transformed to grey scale, and rendered?

My code changes shouldn't have changed any operational features, are you asking me about the internals or has something changed due to my edits?

[zeffii]

mlp()

Quoted text by monzop as response to my (zeffii) code observations.

1. first determines the force formula (dubost,...buckingham)

   Yes, default formula is Testa

2. if posix based (linux, osx), then chmods pyMLP.py (gives it permission to be run)

3. then launches this nightmare of a string concatenation (this is one line) . Which I think means "get from tmp.pdb a tmp.dx"

   command = "%s %s -i %s -m %s -s %f -o %s -v" % (quotedPath(pyPath), quotedPath(homePath + "bin" + os.sep + "pyMLP-1.0" + os.sep + "pyMLP.py"), quotedPath(homePath + "tmp" + os.sep + "tmp.pdb"), method, spacing, quotedPath(homePath + "tmp" + os.sep + "tmp.dx"))

   _I cannot read (language unknown to me), but the tmp.dx is a file which is generated by pyMLP. One pssiblity is that pyMLP uses a file already prepared, and fills in witht he values. Pls. note that pyMLP needs all the fi values, which are stored as valuesfi, by aminoacid and by atom, in the very big BioBlender2.0.py code,(Lines 186 to 224).

4. then it calls surface()

   1. opens tmp.pdb to correct 1+ 1- errors (whatever those are)
   2. generates tmpPath0 a surface.pml which seems to be a script that pyMOL can read
   3. launched pyMOL with python command = "%s -c -u %s" % (quotedPath(pyMolPath), quotedPath(homePath + "tmp" + os.sep + "surface.pml"))
   4. imports a tmp.wrl file as an object and names it "SURFACE"
   5. removes doubles, shades smooth, translates and rotates the surface to match the rotation of the Empty.

   if the wrl file also contains color infomration, this should be removed at this step if the user already generated a surface, it is not necessary to repeat the process here.

5. waits for the asynchronous process (3) to complete
6. then the .dx is opened and read and updates a number of variables dxData, dimension, origin, delta

7. if there was dxData , vertex_paint_mode is entered and again it removes doubles and shades smooth (this might be redundant)

   _indeed, it seems redundant. it also seems odd that the removal of doubles and smoothing should be associated with vertex_paintmode

8. then it reads the vertex colour map (which was generated during the wrl import (4. IV), for each vertex of each polygon in the color map it performs a modification to the vertex colours as imported.

   tmp = (color_map.data[i].color[0] + color_map.data[i].color[1] + color_map.data[i].color[2]) / 3
   color_map.data[i].color = [tmp, tmp, tmp]

   definitely forced greyscale

   • this is where we say (in italian) that the donkey falls. In english: error revealed. In step 4.IV all colors must be removed. The new colors are attributed to the verteces by trilinear interpolation of the values in the .dx file, to the corresponding vertex. this step never worked after BB0.6 because (the coder said) vertex identities were not accessible. *

9. then does a vertex_color_smooth on the whole map
10. sets viewport to TEXTURED
11. lastly it hides the atoms in the scene

These last steps do work. I am almost sure that if you do remove the colors from the sutface generated by pyMOL, the part of running pyMLP and attributing colors to verteces has never worked. The way it used to work in early versions was by calling a script called OBJcreator, which was capable of putting together the info from the surface (.wrl) and the MLP (.dx) and created a new file with colors (.obj)

[zeffii]

mlpRender()

todo

[zeffii]

@MonZop i've formatted that last post so it is more clear to me (and readers) who is commenting what.

[zeffii]

OK, i'll check out OBJCreator script

[zeffii]

the .dx file is a massive list of 3 Floats per line (I doubt these are colours?) the .wtl is imported using a import_x3d provided by blender, it contains verts+faces+colours per vertex

In the end where do you want the colour to come from?

[zeffii]

wait... the dx file is a grid ? a representation of values in a 3d vector space? So is the idea to make the polygons on the 'SURFACE' object mesh to the nearest value in the dx file?

[zeffii]

like this?

[zeffii]

object 1 class gridpositions counts 43 40 34
origin -2.213350e+01 -1.778850e+01 -1.658250e+01
delta 1.000000e+00 0.000000e+00 0.000000e+00
delta 0.000000e+00 1.000000e+00 0.000000e+00
delta 0.000000e+00 0.000000e+00 1.000000e+00
object 2 class gridconnections counts 43 40 34
object 3 class array type double rank 0 items 58480 data follows

suddenly makes more sense: 43 * 40 * 34 == 58480

[MonZop]

Great job! Sprry I was offline for some days, or I might have explained better before.

The value to attribute to the vertex in the original plan was obtained by trilinear interpolation. The option of usign the value of the nearest grid point is good, especially if it makes the calculation much faster, but only when the grid spacing is fine (up to 2 A). For coarser grids it is much better to interpolate, in order to have a smooth painting of MLP on the surface.

[zeffii]

That image of the grid was only to communicate the concept, that code is not written yet.

OK. if I can match up the vector space of the dx file with the 3d coordinates of the triangulated surface mesh, mapping to closest or (some function of the nearest n grid points and falloff gradient) is possible using a KDTree.

[zeffii]

numpy has a very convenient way to reshape a flat list of vectors to a 3d matrix, numpy comes with all blender installation these days. it's worth adding as a dependency.

[MonZop]

Whatever it means, if it makes things easier/faster/smoother, then it's a good idea

[zeffii]

a few variables worth understanding.

dxCache is a dict, mapping surface vertex index ---> value in the .dx file a that coordinate.

There is a function (which is unused by any of the current code..) called def getVar(rawID): which generates the dxCache and queries it progressively.

[zeffii]

obviously this is not what you want.. but it is what happened when I plugged getVar in without functional modification..(only passing extra variables to the function because I put it somewhere else)

[zeffii]

the code

# these are mere references, no copying is taking place.
local_vars = dimension, delta, origin, dxData, dxCache, ob

color_map_collection = ob.data.vertex_colors
if len(color_map_collection) == 0:
    color_map_collection.new()
color_map = color_map_collection['Col']
i = 0
mesh = ob.data
for poly in mesh.polygons:
    for idx in poly.loop_indices:
        # tmp = ((0.21 * color_map.data[i].color[0]) + (0.71 * color_map.data[i].color[1]) + (0.07 * color_map.data[i].color[2]))
        # tmp = (color_map.data[i].color[0] + color_map.data[i].color[1] + color_map.data[i].color[2]) / 3
        loop = mesh.loops[idx]
        rawID = loop.vertex_index
        val = getVar(rawID, local_vars)
        color_map.data[i].color = val
        i += 1

[zeffii]

this looks almost right I think... the banding looks like the bug... but conceptually this seems like it got close.. what do you think?

[MonZop]

Interesting... It seems that there is a problem with just one of the three coordinates, that lead to the 'layered' appearence. Which protein are you using?

[zeffii]

this one : 06_1L2Y_4GE.pdb it's synthetic?

[mcallieri]

Hi there.. I am Marco Callieri, I wrote the original C++ code for the DX mapping and line generation, the code was then partly ported to Python. The original code is not my cleanest no most optimized code (especially the surface sampling & line generation), sorry :) Anyway, feel free to ask clarifications on the procedure and data... a lot of time has passed, and I may not remember everything accurately. For the DX data...

object 1 class gridpositions counts 43 40 34 origin -2.213350e+01 -1.778850e+01 -1.658250e+01 delta 1.000000e+00 0.000000e+00 0.000000e+00 delta 0.000000e+00 1.000000e+00 0.000000e+00 delta 0.000000e+00 0.000000e+00 1.000000e+00

it meas that the grid is 43 x 40 x 34, with the origin (the corner with lowest X,Y,Z number) in [-22.13350, -17.78850, -16.58250]. The next three tells you that the X axis of the grid has an increment of [1, 0, 0] with respect to the origin, the Y axis an increment of [0, 1, 0] and the Z [0, 0, 1]. It is basically the reference system of the grid that, in this case, is aligned to the axis of the scene and a unitary spacing (pretty much what you will expect. I've never encountered a non axis-aligned grid, but have found grids with a non-uniform spacing (e.g. 1 unit on X, 2 on Y, 1 on Z).

If everything is done correctly, the grid will be superimposed to the 3D coordinates of the model. If this is not so, try checking if you are applying some transformations (from the blender side). Another possible source of errors is the fact that every program uses a different reference space (right-left, Y up, Z up).

Since I had to keep in memory all the grid anyway, and the grid is full, to map the value I just determined the cell (integer division), and interpolated the values of the corners of the cell. Sicne the grid is regular, this is as quick as any other method :)

[MonZop]

YEs it is a synthetic peptide, but this makes no difference for bioBlender, or for the MLP calculations.

[zeffii]

Welcome mcallieri! . don't worry about this code, it is at least readable! My focus is to get it working, and once we confirm it does what it should, then proceed with optimizing and alternatives.

[MonZop]

Getting there, i would say! The peptide does have a hydrophobic (white) patch on the center-top, and several darker areas on the sides.

[mcallieri]

one thing about using only the closer vertices in the grid: the calculation of the MLP and EP values for points inside the molecule can produce incorrect and unstable results... points inside the surface may have values 1000 times higher or lower than the "surface" ones, and with random sign... If by chance the closest grid vertex is inside the cell, the mapping will result in strange patterns.

[MonZop]

I think that doing the mean of the 8 points of the grid surrounding the vertex point or doing trilinear interpolation should give the same result. (I am almost sure)

[MonZop]

Render works! Does the operation of mapping MLP also move the coordinates of the vertex? It seem that there is a 'square pattern' hidden in the surface, or is it an impression?

[zeffii]

Technically before the vertex colours are assigned, the original mesh as imported from the wrl (SURFACE) has a remove doubles applied.. which might introduce some numeric noise but it should be below the threshold of making any difference graphically.. Maybe you can show a screenshot of what impression you have there?

[mcallieri]

The remove double is because the VRML file exported by the tool that creates the mesh has each face separate from the others (so each vertex is duplicated among all incident faces), the coordinates are, however, exactly the same... the checkerboard pattern is due to the fact that the surface has been creating using a tessellation which is locally based on a 3d grid AND we are mapping a value coming from another grid with a different step and position.... and voila'.... Moiré Patterns :)

[MonZop]

Using the latest BioBlender-BB2_Redux_s3_FIRE_MLP the MLP on Calmodulin is like this:

[MonZop]

Thanks for the explanation Marco! And how do we fix this now? It did work well with the prgogram we used before: what's the trick?

[MonZop]

It does appear that vector spaces in which the various meshes/files occur is properly aligned. Else there would be no correlation between features and colour, it would look random or stretched. So efforts will continue under the assumption that in-fact, the alignment is OK. : Yay / Nay?

Not sure: in general yes, but the moirè effect would be produced by a very small disalignement, something that could refer to a very minute difference, not sufficient to brake the correspondence of the mesh with the UV map.

Variations

After long thining I understand that rawID is the coordinate of a vertex of the mesh (right?). I think that sampling inside the molecule should not be a problem in the case of MLP, and suggest that the 8 points of the grid are used to get the mean value and attribute it to the vertex. Very high values of the grid are more likely in the EP grid, where strong charges can be found.