Dataset preprocessing

[ssh98son]

Hi, I'm trying to train shapo with my own synthetic dataset. Beforehand, I wanted to reproduce your results.

I tried to generate a dataset file and succeeded in the train set. However, it seems like some files are missing to generate the validation set. In the attached sdf_rgb_pretrained.tar.xz, I cannot find "Reconstructions" directory.

Because of it, the generation script says the following error. I ran the following command.

Here is my data structure. I followed the guidelines and was able to reproduce the training set, but not validation set.

Thank you a lot in advance.

[ssh98son]

Hi, while looking into the code, I also found that there are Smooth L1 Loss which are applied to depth output head. It seems like the depth output head is actually not used for shapo inference. Can I ask the purpose of it?

[zubair-irshad]

Hi @ssh98son

Thanks for your interest in our work and trying out the training code we released.

1. A bit hacky solution, but you could append a random latent vector to generate your validation data and run training. This can be generated for all instances as follows: latent_vec = torch.rand(64). The intuition for this is two-folds, we don't have access to GT SDF field for the validation or test set either during training or testing so even if we optimize over a new latent code given a pretrained SDF network, the validation error is not a great metric for shape latent vector to determine early stopping. Hence, we completely ignore this metric during validation and only determine early stopping based on the rotation and translation errors, which is a more sound metric. So a random latent code here would mean that your training would run smoothly, and it would just produce a very high val error for shape which you can safely ignore. A better way to do it would be to still include random latent vectors during data generation so that the training runs fine but you could comment out the shape_emb_loss and appearance_emb_loss from here depending on the prefix value i.e. only in val mode so these losses don't appear in your total loss calculation oinly furing validation.

2. Your observation is correct. This is just an auxillary loss. Please see our previous work CenterSnap for a motivation behind it. Specially, Table 3 ablation where we showed it helps sim2real transfer as we trying to infer noise free depth from noisy depth which brings synthetic and real depth domains more closer.

Hope it helps!