Human Segmentation (Boundary Edges)

[obisker]

I attempted to use the pre-trained network on personally generated data and the segmentation results were quite poor. The mesh is decimated down to ~1500 verts and 3000 faces, manifold, and watertight. I attached the OBJ (inside zip) I used. mesh.zip

[obisker]

Looks like the network expects 2250 edges (1500 faces) [or fewer?]. Segmentation works great once I decimate the mesh to the target density.

[ranahanocka]

Awesome! Thanks for letting me know. Yeah, I trained that network on 2250 edges, so deviating from that too much probably won't work well.

It is possible to retrain the network with more edges, but you should probably modify it a bit (the pooling resolution). Also it is possible to segment the original mesh from the lower resolution version. I have some plans to add code to do this :)

[ranahanocka]

@obisker I am curious -- where is this mesh from? It looks like it may have been scanned or something, which is pretty cool.

[obisker]

That's right, the mesh is from a total body capture system.

An additional observation about the accuracy is that the currently trained network does not like mesh boundary edges. Presumably, because it was trained with watertight meshes it does not seem to generalize to edges which fail to collapse as an impl limitation as opposed to a learned collapse.

def __pool_edge(self, mesh, edge_id, mask, edge_groups):
        if self.has_boundaries(mesh, edge_id):
            return False

[ranahanocka]

Cool.

The mesh that you sent is indeed watertight manifold. Are you referring to a different mesh you did not send?

So the network pooling can technically handle meshes with boundaries (although, I don't think human seg dataset has boundaries). If the edge is a boundary, we do not collapse it (a "boundary-preserving edge-collapse"). So it will just move to the next edge in the queue.

It is possible to synthetically add boundaries in the humans training data (like "break" the mesh) and re-train, to see if that helps with generalizing better with the boundaries on your mesh (didn't try it!).

I think the best option is to try to close the hole in your mesh / meshes. Have you tried using this: https://github.com/hjwdzh/Manifold ? Out of curiosity, if these are human meshes, why are there boundaries ?

[obisker]

In a typical 3D capture system, achieving complete coverage of every visible body area can be prohibitive either in number of cameras or time (if using sweeping scanner). Small holes in armpits, ears, bottoms of feet, and around hair are quite common. Automatically filling these holes is the typical solution (what I did on the mesh I linked), but requires additional processing time. The other scenario with edges is partial body capture - suppose you’ve only captured the front of the body.

This type of dataset: http://www.dais.unive.it/~shrec2016/

I believe that your model should be able to cope with holes or missing portions if you can figure out how to do pooling on the boundaries and padded edges. For example, if I pass in a half body with 1125 edges, the code will pad an additional 1125 edges. Now correct me if I am wrong, during pooling it will not collapse any of the 1125 padded edges and will also not be able to collapse any of the more than 100 boundary edges (in the valid 1125 half body set). Instead, it will skip the padded and boundary edges and collapse edges out of the remaining pool of 1000 edges. In my tests, things bomb out trying to meet the 1800-1350-600 edge pooling reductions... basically the queue doesn't have enough valid edges to collapse in order to meet the target edge counts.

You can see in the attached image how the boundary edges persist through the pooling stages (green marks boundary edges).

At the very least, boundary edges are displacing more informative (higher weighted) edges from surviving to the bottom of the U in the network.

[ranahanocka]

Nice, it sounds like an interesting application.

So about the padded edges, they are place holders which are removed in the pooling layers. Take your example, a mesh with 1125 edges which is padded to 2250 edges - and pooled to: 1800, 1350, 600 target edges. In the first and second pooling layers (1800 and 1350) the pooling will remove the padded edges. Only in the last pooling layer (to 600) will real edges be removed. But you are correct, if there are too many boundary edges, it is possible that there will be no more valid edges to collapse, and the target edge count (of 600) will not be met.

One option is to retrain the network on different pooling resolutions: download the human seg data, and just modify the flag which is currently --pool_res 1800 1350 600 to something like: --pool_res 1800 1350 800. For example, you can check what the smallest number of edges the network can collapse your data to (if it is 800, so you can make that the last pooling layer target number of edges).

About pooling boundary edges, it requires a bit of extra checks, since we don't want to pool boundary edges which would make the mesh non-manifold (for example, have a dangling edge). I will see what I can do about adding it to the mesh pooling (we will do it anyway at some point), but it might take some time

Let me know if the re-train works :blush: