The Rewrite: BB2 Redux

[zeffii]

Over the next couple of days (unless something unforeseen happens) I have time to make a start with building the add-on from scratch. Getting it to match the working features of BB2 will take a while. It's difficult to put a time frame on it until the process gets rolling.

The decision to start from scratch is not done in haste. I think some of the responders on this repository have expressed similar feelings about how to proceed. There's lots that can be taken from the current code-base, but a lot that I really want to sanitize and arrange into separate files. The core of this was written when Blender 2.5 was still young and the API not very well defined, and prone to convoluted solutions.

If I can reach feature parity and structure the code better then I will consider it a success, speeding up the import (and avoiding recursive updates) is not my first priority, making it easier for others to contribute is far more important.

Ticks indicates functionality seems to be restored for that panel, certainly report any issues with those sectons. The unticked sections have had no major checks done other than suppressing import errors. The section labelled next is either being worked on or that's where the most recent code changes have taken place.

• [x] gui_panel_ep.py
• [x] gui_panel_mlp.py
• [ ] gui_panel_nma_viz.py
• [ ] gui_panel_output_animation.py
• [x] gui_panel_pdb_import.py
• [ ] gui_panel_pdb_output.py <---next
• [ ] gui_panel_physics_sim.py
• [x] gui_panel_view.py

[MonZop]

You are right, as usual.. Import of lipids did not work in the Master . I was confused because it used to work before, when building of bonds was not automatic, but was done only if the user requested so.

[zeffii]

hey, that's not to say it can't be modified to work :) you will need to explain in detail what you expect (use images!).. and maybe assign me something to read to help put the terminology in context.

[MonZop]

Ok! ever so kind! The issue here is that whle most PDB files contain proteins and nucelic acids, some of them might well have other molecules. These will be described by their atoms and positions, but it will not be possible to build a skeleton of bonds between them, because the BB library will not have information on how to do so. In this case, I think that the best solution would be to have all atoms placed in the 3D scene according to the xyz coords, and a warning to user saying that these parts might not behave in the right way during game engine simulation. I'll prepare some images to show the details, but I think you are right that at this time it is most important to have the basic features of BB up and running, with changes coming afterwards.

[zeffii]

Will start on the EP panel soon, have spotted the parts that I missed in the separation work, and will implement the finer radius fixes from one of the older branches (fixes particles not following the described curves).

Expect I'll do it this evening or tomorrow.

[zeffii]

well, i found time anyway. But there's a problem, on linux I get this error:

run-detectors: 
unable to find an interpreter for <.....>addons/BioBlender/tmp/apbs.exe

it shouldn't even be looking for a .exe. I don't recall changing any OS specific code that might have made that broken.. -- will investigate, maybe I did.

[MonZop]

APBS is available from Sourceforge. http://sourceforge.net/projects/apbs/files/apbs/ also linux version. Inour version we had version 1.2.1, I am not sure if the newer will be beter, or will introduce problems..

[zeffii]

https://github.com/Electrostatics/apbs-pdb2pqr/tree/1.4.1-binary-release

[zeffii]

i'll compile from source if possible.. else will check out the Sourceforce

Sourceforge is coming under fire from a lot of opensource people, may be interesting to know.

[zeffii]

ok cool. this is the offending bit of code:

          if opSystem == "linux":
                shutil.copy(quotedPath(homePath + "bin" + os.sep + "apbs-1.2.1" + os.sep + "apbs.exe"), quotedPath(homePath + "tmp" + os.sep + "apbs.exe"))

linux doesn't have the .exe convention, so I suspect this was never tested on a Linux machine directly..I have a binary now (thanks for the link @MonZop ..i have little patience for compiling atm).

[zeffii]

Not sure exactly how to proceed, but will first drop the binary of apbs-1.4.1 into that 1.2.1 directory - to at least test locally. I'm under the assumption that you have permission to include these binaries in the current Master already? would be great to include the latest (at least for linux...as that's what i'm testing on)

Using 1.4.1 i get error while loading shared libraries: libamd.so.1: and while I know what that error means i'm not sure It's trivial to fix for the average user. I have a much newer version of this library (libamd2.3.1, which comes standard on ubuntu 14.04 LTS)

[MonZop]

If I understand the question, the answer is that APBS is released under a license that permits the inclusion of the code in any other program.

[zeffii]

OK, cool then, i've always noticed requests to please register before downloading etc..

[MonZop]

Yes, we are registred users. I think they like to know who's usign their program, but then are also happy to make itr as widely distributed as possible

[MonZop]

I have APBS in my pc, but BioBlender-bb2_REDUX_s5_EP cannot calculate EP: here's the error

[zeffii]

When did you update that Branch of s5_EP ?

Unfortunately, i'll need to get it working here first before I can proceed - it could be any number of things.. i'm not worried yet :)

[MonZop]

last updated 16.17 (less than 30 minutes ago) Glad to know that you're not worried!

[zeffii]

turns out.. apbs is easily installed on ubuntu.. let's try linking into the tmp dir. ln -s /usr/bin/apbs ~/Desktop/GITHUB/BioBlender/tmp/apbs

nope still not... it's not finding the libraries.. this will have to wait till tomorrow Ok this kind of works but I need to comment out the copy apbs and chmod apbs lines for the time being.

[zeffii]

This was bugging me..

Looking deeper at this, quite a few of the linux and darwin (osx) paths are hard coded for EP stuff. Slowly tracking. lines like:

                    homeutente = os.path.expanduser("~")
                    shutil.move(quotedPath(homeutente + "/.config/blender/2.67/scripts/addons/BioBlender1/bin/pdb2pqr-1.6/pot.dx"), quotedPath(homeutente + "/.config/blender/2.67/scripts/addons/BioBlender1/tmp/pot.dx"))
                    shutil.move(quotedPath(homeutente + "/.config/blender/2.67/scripts/addons/BioBlender1/bin/pdb2pqr-1.6/io.mc"), quotedPath(homeutente + "/.config/blender/2.67/scripts/addons/BioBlender1/tmp/io.mc"))
                except Exception as E:

At one point in time, i'm sure it made sense to do it that way... but now we can totally use relative paths to the current BioBlender folder -- This will change, until then I can't get anywhere.

In short: it's still very broken, but the reasons are becoming more obvious

[zeffii]

almost..

[zeffii]

it's less broken now :) -- you may want to try again at your earliest convenience and report any error messages. I had failed to import a needed function from the new utils.py module, therefore it wouldn't have worked on any OS.

The curves should be parented really right? At the moment it just throws them all into the scene...it fills up the out-liner really fast.

[zeffii]

glad to see I already suggested that in a previous roadmap thread: https://github.com/MonZop/BioBlender/issues/16

will add (from older branch fixes):

ob.field.guide_minimum = 0.0008
ob.field.distance_max = 0.0008   # originally 0.05

and think about what impact making a curves Parent object will have.

[MonZop]

Something strange happens: If I import a protein and request EP, I get the error above (No such file...). However, if I request MLP, and then EP, everything proceeds and I do get lines and particles. Next I will check how it works with moving protein

[zeffii]

I've always done MLP then EP (i think), so.. little wonder then that EP directly doesn't perform as expected. Should be easy to resolve. (I was under no delusion that the EP panel had reached parity)

[MonZop]

I also notice that the number of lines is really high. But very many of them are not visible because they are completely inside the protein surface (for this reason they are also 'not interesting' from the point of view of how the protein relates to the rest of the world). In theory there was a step which checked which lines are useful, and discards all those that are not. I have to check how this was done

[zeffii]

wouldn't surprise me if apbs (or pdb2pqr?) has a switch to exclude generating lines inside the surface. Looking at the relevant code in BioBlender doesn't suggest this was ever implemented as a post-processing step.

[zeffii]

oh.. it's the scivis_tool that outputs the tmp.txt used for the construction of the eplines.

[zeffii]

from the scivis_tool source code

  //---- parameters parsing ----
  if(argc < 9)
  {
    printf("\ntoo few parameters\n");
    printf("USAGE: scivis 3dmeshfilename DXfilenamne linesfilename linesdensity haltgradient haltangle reduce reduceangle\n\n");
    printf("3dmeshfilename     3d mesh to load, obj format\n");
    printf("DXfilenamne        potential grid, DX format\n");
    printf("linesfilename      output file name for lines, txt format\n");
    printf("linesdensity       lines density, as # of lines per Ev/squareAngstrom, float\n");
    printf("haltgradient       stop line if a gradient lower than this is found, float\n");
    printf("haltangle          stop line if an angle larger than this is encountered, float\n");
    printf("reduce             simplify lines before exporting, 1=yes 0=no\n");
    printf("reduceangle        maxim angular change in simplification, float\n");
    printf("\nEXAMPLE: scivis calmodulin.obj calmodulin_EP.DX output_lines.txt 0.01 0.0 45 1 3.0\n\n");
    return -1;  
  }

[MonZop]

Sprry I'm going back: Import is supposed to either import a molecule from alocal repository , or fetch it from pdb.org, if the input is a 4 letter code (for example 2ylb). However fetching does not seem to work.

[zeffii]

cool. I have never used fetch, it's probably missing a few imports.

[zeffii]

I've committed fixes for this pdbdotorg function, seems to work.. That 2ylb seems to be pretty big, do you know of smaller molecules I can test? (slow computer here)

[MonZop]

2ylb is about 7000 atoms 1HP7 is about 3000 atoms 1l2y is about 300, several confomrations

[zeffii]

fetch should work, works here..

continuing on the EP view asap.

[MonZop]

Yes! Fetch works here as well

[zeffii]

Is functionality of EP panel restored now?

[MonZop]

I would say YES :-) the outliner is still clogged with all the curves: If possible, I think thatthey should be all parts of a single object; this becomes important when we do animations of moving molecules: in this case as the proteins move, their EP will change, and thelines will need to be reclaculated every 5 frames or so. You can easily see that even a 10 second animation wil become very difficult to handle, unless we put a little order in the outliner. https://vimeo.com/132071264 here a short animatied EP

[zeffii]

coool. OK, I will step through the code used for animation to see if I've missed anything or can implement some kind of grouping (parenting to an Empty is imo a great way to do this, "grouping" in the Blender sense is useless for this... parenting will keep the outliner clean too)

[MonZop]

Sure! Parenting to an Empty is good idea. I suggest that the Empty is created with a meaningful name, possibly starting with the name of the molecule, and including the frame N of creation. I will write a description of how the EP works in animation, to be included in the BB manual, possibly today.

[MonZop]

Once again, I am going back to a panel that was already 'closed'. One of the possible ways to look at a moleucle, is by using the Van der Waals spheres. This was encoded as a choice in the import panel, but it woudl better fit in the BB view panel. The result should be that, when Vad der Waals is selcted, each atom sphere, instead of having its 'atomic' or 'covalent' radius, should be represented by a sphere with tis VdW radius (typically much larger that the covalent). Spheres will naturally compenetrate, like this

The VdW sphere is the one used for calculating the surface by the rolling probe algorithm.

[zeffii]

This was encoded as a choice in the import panel, but it would better fit in the BB view panel.

I will look into it, and it's definitely noted.. not sure if it's tricky to implement or not. Ideally the way the molecule is viewed has nothing to do with the import step, and the molecule should be viewed any sort of way dynamically at a later stage, right now I do think the import stage sort of has a finalizing effect on how the objects are viewed and what sphere radius is used to view and for game-engine dynamics.

recap: I think I understand the request. but am certain that I'm not intimate enough with code yet to say how easy it will be to facilitate it.

[zeffii]

A note to self here about registering scene properties in a nicer way, right now each Scene property is registered and unregistered explicity, but there's a nicer way using a propertyGroup and might worth doing it that way.. it may look something like:

class BioBlenderProperties(bpy.types.PropertyGroup):
    @classmethod
    def register(cls):
        Scn = bpy.types.Scene
        opSystem = utils.detect_os()
        homePath = utils.get_homepath()
        pyPath, pyMolPath = utils.get_pyPath_pyMolPath(opSystem)

        Scn.bioblender_props = PointerProperty(
            name="BioBlender internal global properties",
            description="BioBlender uses these properties for shared properties between operators",
            type=cls,
        )

        cls.bb25_importReady = BoolProperty(default=False)
        cls.bb25_bootstrap = IntProperty(default=-1)
        cls.bb25_opSystem = StringProperty(default=opSystem)
        cls.bb25_homepath = StringProperty(attr='bb25_homepath', default=homePath)
        cls.bb25_blenderPath = StringProperty(default=str(sys.executable))
        cls.bb25_pyPath = StringProperty(default=pyPath)
        cls.bb25_pyMolPath = StringProperty(default=pyMolPath)
        cls.bb25_pdbID = IntProperty(default=0)
        cls.bb25_currentActiveObj = StringProperty(default="")
        cls.bb25_oldActiveObj = StringProperty(default="")
        cls.bb25_activeModelRemark = StringProperty(default="")
        cls.bb25_viewFilterOld = StringProperty(default="")

    @classmethod
    def unregister(cls):
        del bpy.types.Scene.bioblender_props

def register():
    ...
    bpy.utils.register_class(BioBlenderProperties)

def unregister():
    ...
    bpy.utils.unregister_class(BioBlenderProperties)    

[zeffii]

regarding the sphere radii, it might be simpler than I thought!

obj.scale = [scale_cov[index][0], scale_cov[index][0], scale_cov[index][0]]

each atom has that line performed, the view panel could easily enough have an additional mode to show vdw, and back to cov. It's a reasonably fast operation linear time (unlike object creation which becomes progressively slower due to name collision checking)

s = scale_vdw[index][0]
obj.scale = [s, s, s]

perhaps an operator for a function like

lookup = scale_vdw if show_type == 'vdw' else scale_cov
for obj in objects_of_interest:
    atom = obj.BBInfo[76:78].strip()
    s = lookup[atom][0]
    obj.scale = [s, s, s]

[MonZop]

The theory sounds good. I also noticed a version s6_radii, and I checked it, but i cannot find how to visualize vdw

[zeffii]

yes, some code is committed but nothing is hooked up yet, it might take me a few days to get around to it.

[zeffii]

added this to the View panel, consider the UI only temporary, it should work on any number of selected PDB ids. Will have to wait for heatwave to die down.

[zeffii]

[MonZop]

The heat is too much, even for italians.. but luckily I have air conditioning in the office. The button works, and the radius does change, but it deos not apply the true VdW riadius: Covalent radius:

'VdW radius in new BB2 (slightly bigger atoms)

VdW raidus in old BB (0.6) real VdW radii

[zeffii]

@MonZop I observe this too, i'm sure it will be a simple fix.

[zeffii]

I don't have much time to do this at the moment, other matters unfortunately take precedence. I hope to tackle this soon but can not suggest any ETA.

[MonZop]

I'm afraid it's not a simple scaling of all atoms: some atoms scale by 1.5 while others multiply the radius by 4 or more, relative to covalent. The VdW radius is a value empirically measured. To complicate things, not all experiments give the same measures for the same atoms, but there is a sorto of consensus, and for those who really care about a minute difference, they can use QM to solve them. I copy here a table of the values typically used.

[zeffii]

I used the two tables in tables.py called scale_vdw and scale_cov , so each atom is scaled according to that lookup. If you look in the Object properties tab, you'll see the scale used by current BioBlender (same as master)..

reproduced here

# Define atom scales [visual Van der Waals scale, collision radius scale]
scale_vdw = {
    'C': [1.35, 0.6], 'CA': [1.59, 0.6], 'N': [1.23, 0.6], 'O': [1.2, 0.6], 'S': [1.43, 0.6],
    'P': [1.43, 0.6], 'FE': [1.59, 0.6], 'MG': [1.37, 0.6], 'ZN': [1.1, 0.6], 'CU': [1.1, 0.6],
    'NA': [1.8, 0.6], 'K': [2.18, 0.6], 'CL': [1.37, 0.6], 'MN': [1.59, 0.6], 'H': [0.95, 0.3], 'F': [1.16, 0.6]
}

# Define atom scales [visual covalent scale, collision radius scale]
scale_cov = {
    'C': [1.1, 0.6], 'CA': [0.99, 0.6], 'N': [1.07, 0.6], 'O': [1.04, 0.6], 'S': [1.46, 0.6],
    'P': [1.51, 0.6], 'FE': [0.64, 0.6], 'MG': [0.65, 0.6], 'ZN': [0.74, 0.6], 'CU': [0.72, 0.6],
    'NA': [0.95, 0.6], 'K': [1.33, 0.6], 'CL': [1.81, 0.6], 'MN': [0.46, 0.6], 'H': [0.53, 0.3], 'F': [1.36, 0.6]
}

Should this be adjusted?