Import and use with Blender 3d

[rstewar2]

Hello,

I am trying to adapt this library for use with Blender 3d. I am currently working on a project to extract skeletons from a 3D model dataset. However, this dataset I'm using has models with mesh errors that produce ZeroDivision errors when I try to use Py_BL_MeshSkeletonization. Since this is currently the only other public 3D mesh skeletonization library (that I'm aware), I was hoping to use this to extract the remaining skeletons.

I was able to install navis and skeletor inside Blender and the scripts are capable of running. However, my results are less than desired. Is there a way to replicate the TreeNeuron (for the skeleton) in Blender? I understand that navis has a blender3d interface and I've been using the tutorial found here. Also, is there a way to scale the extracted skeleton back to the original trimesh object?

Thank you.

Below is my sample script (insider Blender Python Environment):

import bpy
import trimesh
import skeletor as sk
import os
import numpy as np
import scipy
import ncollpyde
import navis
import navis.interfaces.blender as b3d

# Remove the default cube
bpy.ops.object.select_all(action='DESELECT')

# If default cube is present in scene, remove it
if bpy.data.objects.get("Cube") is not None:
    bpy.data.objects['Cube'].select_set(True)
    bpy.ops.object.delete()

# Path to model resources obj folder on local machine
obj_parent_dir = "/home/rstewar2/Documents/RPE/AnimSkeleton/ModelResource_Dataset_preprocessed_3DV19/model_resource_data"
obj_dir = os.path.join(obj_parent_dir, "obj")

obj_file_path = os.path.join(obj_dir, "8556.obj")
"""
obj_file = open(obj_file_path)
mesh = trimesh.exchange.obj.load_obj(obj_file, maintain_order=True)
obj_file.close()
"""

#t_mesh = trimesh.Trimesh(vertices=mesh['vertices'],faces=mesh['faces'], vertex_normals=mesh['vertex_normals'])
t_mesh = trimesh.load(obj_file_path)

cont = sk.contract(t_mesh, epsilon=1e-07, iter_lim=10, time_lim=None, precision=1e-8,
             SL=2, WH0=1, WL0="auto", operator='umbrella', progress=True,
             validate=True)
swc = sk.skeletonize(cont, method='edge_collapse', shape_weight=1, sample_weight=.1, output='both', validate=True)

swc_graph = swc[0]
swc_df = swc[1]
#swc_df = sk.clean(swc_df, mesh)

swc_df['radius'] = sk.radii(swc_df, t_mesh, method='ray', aggregate='mean')

skeleton = navis.TreeNeuron(swc_df.copy(), units='8nm', soma=None)
meshneuron = navis.MeshNeuron(t_mesh, units='8nm')

# Initialize Blender handler
h = b3d.Handler()

# Load neurons into scene
h.add(skeleton, connections=True, neurites=True, soma=False)

This is a picture of the original trimesh model:

This is a picture with the extracted mesh (scaled to 25 in x,y,z directions) next to the original model:

[schlegelp]

Hi.

Is there a way to replicate the TreeNeuron (for the skeleton) in Blender? I understand that navis has a blender3d interface and I've been using the tutorial found here.

Not sure I fully understand your question here. From your code it looks like you have managed to turn the SWC table into a navis.TreeNeuron and import it into Blender (although I don't see it in the screenshots)?

Also, is there a way to scale the extracted skeleton back to the original trimesh object?

In theory, the skeletonization does not change the coordinates. There is a chance that the mesh contraction - in particular with the umbrella operator - could shrink the mesh. Maybe this is the cause: by default, b3d.Handler.add scales objects on import by a factor of 1/10000 - if you want to have no scaling you can initialise it like so:

h = b3d.Handler(scaling=1)

Re your results: what am I looking at in the second screenshot? Is that the contracted mesh? If so, the mesh contraction step was not sufficient. Could you check what the final epsilon (i.e. contraction rate) is (should be shown in the progress bar)? If it's more than 0.1 (i.e. 10% of original volume) then the skeleton will likely look crappy.

Hope this helps.

Best, Philipp

[rstewar2]

Hi Mr. Schlegel,

Thank you for responding. I guess my first question wasn't really a question as I had already done it in Blender. What I did was use Blender's pip module to install navis and skeletor (with their respective dependencies). From there, I could construct a navis.TreeNeuron within Blender itself and load it using b3d.Handler.

Yes, in the second screenshot, the object on the left was the contracted mesh. The mesh on the right is the original model. When I tested the script in Jupyter Notebook, the FloatProgress value was 0.0. However, after I set the scaling to one i.e. h = b3d.Handler(scaling = 1), it fixed the issue. Now, the contracted mesh looks more representative of the one shown in navis.plot3d

This was the resulting contracted mesh shown in navis.plot3d

This is the same contracted mesh (highlighted in orange) overlaid with the original model inside Blender 2.90.

One thing you may notice is the feet and hands are disconnected with their respective limbs. What parameters would you recommend changing to better this result?

For reference, here is my script:

# Path to model resources obj folder on local machine
obj_parent_dir = "/home/rstewar2/Documents/RPE/AnimSkeleton/ModelResource_Dataset_preprocessed_3DV19/model_resource_data"
obj_dir = os.path.join(obj_parent_dir, "obj")
obj_file_path = os.path.join(obj_dir, "8556.obj")

# Load model using Trimesh
t_mesh = trimesh.load(obj_file_path)

# Perform mesh contraction step
cont = sk.contract(t_mesh, epsilon=1e-07, iter_lim=10, time_lim=None, precision=1e-8,
             SL=10, WH0=15, WL0="auto", operator='umbrella', progress=True,
             validate=True)
swc = sk.skeletonize(cont, method='edge_collapse', shape_weight=1, sample_weight=1, output='both', validate=True)

swc_graph = swc[0]
swc_df = swc[1]
#swc_df = sk.clean(swc_df, mesh)

swc_df['radius'] = sk.radii(swc_df, t_mesh, method='ray', aggregate='mean')

skeleton = navis.TreeNeuron(swc_df.copy(), units='8nm', soma=None)
meshneuron = navis.MeshNeuron(t_mesh, units='8nm')

# Initialize Blender handler
h = b3d.Handler(scaling=1)

# Load neurons into scene
h.add(skeleton, connections=True, neurites=True, soma=False)

Thank you for your time and have a pleasant day.

Sincerely and respectfully, Robert Stewart

[schlegelp]

Am I correct in that your mesh is not fully connected? I.e. the feet, legs and hands are actually separate from the body?

[rstewar2]

Yes sir; that is correct.

[schlegelp]

skeletor does not connect parts that are disconnected in the mesh. You could try to merge the skeleton fragments using a minimum spanning tree like so (I recommend using the most recent navis version from Github for this):

healed = navis.heal_fragmented_neuron(skeleton)

When it comes to preserving details you try this:

• use the default Laplacian: operator='cotangent'
• use vertex clustering with an appropriate sampling size parameter (depends on the dimensions of your mesh) for the skeletonization