Closed OswaldoBornemann closed 11 months ago
ZENGXH
commented
12 months ago I think it's possible. Since the number of points is large, the overall loss scale can be high. A better check if things is going well is checking the reconstruction points: ideally, you will see the point clouds are well reconstructed (especially at the early iter), the latent points will look similar to the point cloud at the beginning and as training going on, it will be smoothed out.
OswaldoBornemann
commented
12 months ago I think it's possible. Since the number of points is large, the overall loss scale can be high. A better check if things is going well is checking the reconstruction points: ideally, you will see the point clouds are well reconstructed (especially at the early iter), the latent points will look similar to the point cloud at the beginning and as training going on, it will be smoothed out.
Yes. I will be smoothed out. And I found that my own dataset is very sensitive to the kl_weight. When the kl_weight approaches 0.5, I found that the latent points are totally different from the input point clouds. Based on this observation, I tried to decrease the kl weight and try to reach a situation where the latent points are slightly different from the input points, which is slightly smoothed out.
ZENGXH
commented
11 months ago This makes sense to me. kl_weight can be an important hyper-parameter to tune when the dataset changed. Hope the experiments go well!
OswaldoBornemann
commented
11 months ago This makes sense to me.
kl_weightcan be an important hyper-parameter to tune when the dataset changed. Hope the experiments go well!
Thanks a lot. Really appreciated.
@ZENGXH Hi, I am trying to train LION with my own dataset. My dataset contains nearly 800k point clouds, and the dimension of each point cloud is 6, which has both coordinate and color features. The number of points in each point cloud varies from 1,000 to 3,000. Considering the training time based on the size of the dataset, I reduced the
trainer.epochsfrom 8,000 to 25. And thelrfrom1e-3to1e-4. I trained the VAE on 4 A6000 GPUs, with 32 batch sizes. However, I found that the training loss is very high at the early stage. Is it normal or not? Thanks for sharing your insight.