Object Activation Library and codes

[yyashpatel]

Hi !

how to define the object activation codes for the object as per the number of objects in the scene ? Say if I have only 2 objects in the scene what should be the activation library or say may be I have only 1 object what should be the activation library ? So basically how are the activation codes assigned to the individual objects in the scene ?

In the code I saw that you use torch.embedding and input indices 1 to 5 , so does this mean the code library gives 5 activation codes for the 5 objects (cube,dragon, face, candy, magic cube).......... but which code belongs to which object ? If I only wanted to render 3 objects out of the 5 objects then what should be done ?

Please help me with this concept of selecting the object codes @ybbbbt

[ybbbbt]

Hi, the code index is selected according to the labels in the segmentation images, which is manually assigned during the data preparation. You can modify the data loader to skip the decoupled learning of some objects.

[yyashpatel]

oh okay , I see you are using labels (1,2..5 ) and obtaining the object codes. But if I want to edit my own scene captured by a bunch of images and use my own objects (I have the mask images i.e. pixel value=1 for object location in the image), but then how do I select the object code for my own objects?

Thanks !

[ybbbbt]

Hi, yyashoatel,

I load the labeled segmentation masks at:

https://github.com/zju3dv/object_nerf/blob/e8b1a7a5ab0596babdf4400a9e8908f1bfdcf990/datasets/generic_dataset.py#L106-L118

Then, you can modify the instance_id and val_instance_id in config file to let the network knows which instance (with the specific IDs) to be trained and validated (visualized in Tensorboard):

https://github.com/zju3dv/object_nerf/blob/e8b1a7a5ab0596babdf4400a9e8908f1bfdcf990/config/scannet_base_0113_multi.yml#L35-L36

[yyashpatel]

Thanks, I get it !

one last question - Can I change the numbers used in the instance ids i.e. for example in the above file that you shared 4 represents near sofa and 6 represents center sofa.

I have my own objects (a box and a book) so what instance id should I assign to these objects. I know you have assigned the ids based on segmentation labels....but Basically is there a basis of selecting the instance id for an object or I can assign any number ?

Because I have the mask of all objects, but I do not have the segmentation labels, so on what basis should I select a number for instance ids for my objects. Should just assign any 2 numbers as instance ids?

It would be great if you can clarify this ..Thanks !

[ybbbbt]

You can assign any number of IDs, as long as the numbers match the input training data (i.e., instance_ids in batch) and the number for each object identity is always consistent across views:

https://github.com/zju3dv/object_nerf/blob/e8b1a7a5ab0596babdf4400a9e8908f1bfdcf990/datasets/generic_dataset.py#L269

During the inference, we assign the learnable object code according to the input IDs:

https://github.com/zju3dv/object_nerf/blob/e8b1a7a5ab0596babdf4400a9e8908f1bfdcf990/models/code_library.py#L23-L27

For example, you can assign the box as 1 and the book as 2, and also make sure that the per training ray's instance_ids in dataloader is properly assigned to these values.

Please feel free to let me know if you have any questions.

[yyashpatel]

Thanks for your reply. I understand it now.

Just one last thing, I want to only train and render the object, so I have done changes such that I only train the object branch. So the color loss goes to none sometimes because of the line as seen below ,

https://github.com/zju3dv/object_nerf/blob/e8b1a7a5ab0596babdf4400a9e8908f1bfdcf990/models/losses.py#L75

So does this affect the learning of the network ?

[yyashpatel]

Thanks for your help, I figured it out.