Variable number of points for each pointcloud during training

[ghost]

Hi,

thanks for the nice work!

As in #41, I would like to train on point clouds with very different density. I exchanged tf_util.max_pool2d with tf.reduce_max as @charlesq34 suggested. However, as he also pointed out, the same number of points has to be used for each batch.

I tried to set the batch size to 1. But even when decreasing the learning rate by factor 10 or 100, the model did not converge anymore.

Is there a possibility to get the model to converge with batch size = 1? Does it make sense, to use varying size inputs for training? (To me, it intuitively does, since during datacapture with a scanner varying density does arise) If it does make sense, is there a way to train with varying point cloud density in tensorflow?

Sincerly, Barbara

-- Sidenote: It seems pytorch supports variable size inputs (https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278)

[jiachens]

Hi,

Have you resolved this?

[charlesq34]

HI @brbrhpt

As there are batch norm layers, using batch size = 1 would cause some issues. At training time, TensorFlow does require a fixed tensor size, but you can achieve the "effect" of training with variable number of points by randomly duplicating K existing points to N points (e.g. K is varying, N is fixed as 1024)-- for pointnet, the results would be roughly the same (just a difference caused by batchnorm) as training with K points.

At test time by setting batch size as 1, you can always test with variable number of points.

[offchan42]

@charlesq34 I tried training by duplicating the points and it works but not sure if it works roughly the same when testing on variable length. I want to ask to be sure about mean/std normalization of model input.

When training I have fixed 100 points (usually my dataset have around 40-70 points). So the extra points are padded by duplications. When testing I want to test with 40-70 points not 100 points.

Suppose that I do standard scaling for each point cloud on the X Y Z coordinate, should I do it before or after I duplicate the points?

If I duplicate before I normalize, this will affect the coordinate values. But maybe it will make the model generalize better by not relying on specific coordinate value? If I duplicate after I normalize, this will be like I'm just appending extra points when training. I'm not sure which approach is better. Please enlighten me.

[Maheshiitmandi]

Hi @off99555 I also want to know this. But i think, first copy the points and then normalize will be better because at end we need scaled or normalized data.

[offchan42]

@Maheshiitmandi But normalizing before duplications is also having its benefits. Suppose you have 70 points and you want to expand to 100 points, if you normalize before duplicates you will get the same prediction from the model (if you are doing classification), because most of the layers just do max pooling. And same prediction for 70 points and 100 points is what you want because the extra 30 points are just for padding.

[tkaleczyc-ats]

Hi! Does TF indeed require a fixed size input for all training samples or is this just the requirement of the same size of input within a batch? If it's the latter, maybe making artificial batches of a single point cloud permutated randomly could help?

[offchan42]

Let's say a batch consists of multiple clouds. And a cloud consists of multiple points. If the batch size is 32 it means you are training 32 clouds at a time. But we are talking about the number of points in a cloud here. It's not even a batch yet. It's required that you should have the same number of points for all the clouds so that the clouds can be concatenated into a cube-like tensor for training. It means you need constant batch size (can easily be done by duplicating, no normalization needed) and constant number of points per cloud (need to be done with duplication and normalization, that's why the issue about ordering arise)

[adosar]

HI @brbrhpt

As there are batch norm layers, using batch size = 1 would cause some issues. At training time, TensorFlow does require a fixed tensor size, but you can achieve the "effect" of training with variable number of points by randomly duplicating K existing points to N points (e.g. K is varying, N is fixed as 1024)-- for pointnet, the results would be roughly the same (just a difference caused by batchnorm) as training with K points.

At test time by setting batch size as 1, you can always test with variable number of points.

Could we also just zero padding instead of duplication?