Categoryless Training/Testing

[iKessho]

Hi,

1. Would like to ask if this could work with a data set that has no defined category (example vases could be mixed in with mugs in the same dataset)
2. Are the number of segmented parts fixed? Or can there be a unlimited number of segments?
3. Can it be trained to segment finer details? Example i have a wooden plank, it will be segmented into 6 different surfaces. And if the wooden plank had holes drilled, the segments would increase by 1 for each hole. (i think this term is called surface segmentation? if it exists)

Regards, Kessho

[ranahanocka]

1. Would like to ask if this could work with a data set that has no defined category (example vases could be mixed in with mugs in the same dataset)

This code provides a means for doing supervised semantic segmentation. So basically, you can use any dataset which has meshes and per edge segmentation labels for training the network. I am not sure if this is what you are getting at: but say you take the set of "aliens" (4 possible segments) and "chairs" (3 possible segments) from this repo (coseg segmentation). I provided code to train each one as a separate network. But you could train one network which could recieve chairs or aliens and predict a label from 1-7. The first 4 being aliens , and the last 3 being chairs.

2. Are the number of segmented parts fixed? Or can there be a unlimited number of segments?

You can train any number of segments you like (assume you have enough data , network capacity and GPU power). The number of segments is defined by the number of classes in the 2d-cross entropy loss.

3. Can it be trained to segment finer details? Example i have a wooden plank, it will be segmented into 6 different surfaces. And if the wooden plank had holes drilled, the segments would increase by 1 for each hole. (i think this term is called surface segmentation? if it exists)

Not sure I completely understood, but same thing applies as above. With enough data labels you should be able to learn to do a fine grained segmentation.

Also, make sure to check out the segmentation wiki.

[iKessho]

Hi,

allow me to rephrase and clarify some of my questions: Yes i can combine your set of "aliens" and "chairs" but would it still work if i do not explicitly categorize them as "aliens" and "chairs" when training/testing?. So for example when predicting an unknown (but is actually an alien) object with legs, would the legs be then be segmented as a chair's legs? what about not having a fixed set of parts? https://www.youtube.com/watch?v=6JQdNzsw7jA perhaps the first 20seconds of the video helps to bring across what i'm trying to say

in addition, for your 'aliens' data set you have labeled them 1-4 corresponding to their different parts, e.g. head, body, tail etc. am i able to train the model in such a way that it doesnt have this constraint? i can just say that this section of the alien is a part and that section is another part, and the model is able to segment the object into its different parts

[ranahanocka]

Yes i can combine your set of "aliens" and "chairs" but would it still work if i do not explicitly categorize them as "aliens" and "chairs" when training/testing?.So for example when predicting an unknown (but is actually an alien) object with legs, would the legs be then be segmented as a chair's legs? what about not having a fixed set of parts? https://www.youtube.com/watch?v=6JQdNzsw7jA perhaps the first 20seconds of the video helps to bring across what i'm trying to say

So as for explicitly knowing whether it's a leg that came from an alien vs. chair, it depends how the ground truth-label is built. For example, if we said that 1-3 are the segments for the chair and then 4-7 are the alien segments, so the category of the object is implied by the segmentation labels. Since predicting segment 1-3 must mean that the object is from the class chair. However, this constraint is of course not required in test time, since the network will just predict the segment.

About grouping them together. You would have to train the network to possibly predict a max number of segments (say 10). but that doesn't mean that all the segments must be present in each mesh model. The network will predict a probability the edge belongs to a particular segmentation ID, so that can be any value (for example, for one particular mesh it could predict most of the edges segment 9 and the rest 8). There is no constraint that all segments should be present in each mesh, only that each edge should predict a probability for all segments. For segments that are not present, this probability should be close to 0.

in addition, for your 'aliens' data set you have labeled them 1-4 corresponding to their different parts, e.g. head, body, tail etc. am i able to train the model in such a way that it doesn't have this constraint? i can just say that this section of the alien is a part and that section is another part, and the model is able to segment the object into its different parts

It is technically possible, but I am not sure the meaning of such a thing. I mean, what exactly determines which part corresponds across different shapes? In the case of very similar shapes which have similar semantic parts, I think this could make sense (e.g., different legged animals, humans, etc.). Otherwise I am not sure.

All in all it seems you are looking for something unsupervised, or at least, weakly supervised. Which this code as is doesn't support. But you are welcome to take the layers we provided and build an unsupervised segmentation approach :)

[iKessho]

I see, i kind of understand what you are trying to say. So it would be close to impossible to label the alien model parts with 1-3 and then the chair parts with 1-4 and then throw the model a object and say hey, segment it into different parts. The part labeled '1' would have to at least have similarities across the training sets instead of it being just a generic part then.

perhaps the solution i am looking for lies in the unsupervised training. I'll check that out as well then.

Oh. 1 more question. After creating your models in blender, how did you annotate them?

Thanks for your kind reply!

[ranahanocka]

I see, i kind of understand what you are trying to say. So it would be close to impossible to label the alien model parts with 1-3 and then the chair parts with 1-4 and then throw the model a object and say hey, segment it into different parts. The part labeled '1' would have to at least have similarities across the training sets instead of it being just a generic part then.

Exactly. Another option is to use MeshCNN as a binary classifier, to predict the boundaries between segments. This should be transferable across different shapes.

Oh. 1 more question. After creating your models in blender, how did you annotate them?

Did you check out the scripts in the Segmentation Wiki?

[iKessho]

Exactly. Another option is to use MeshCNN as a binary classifier, to predict the boundaries between segments. This should be transferable across different shapes.

That sounds like a plausible way to segment the parts that i'm looking for. Any advice on how to proceed?

[ranahanocka]

I think you can try to work with the data I have first, to see if it will work first on a single shape class (i.e., just aliens). And then you can extend it to see if it will work on aliens and humans, for example. You should search for all boundary edges (edges which have different segmentation IDs in their 1-ring, called the gemm data-structure in the code).