Bug when decreasing the batch size value

[sgiraudot]

Hello,

I've been trying to test your program, using the Stanford dataset. During the training phase, I encountered a memory issue (not enough memory), so I tried to decrease the value of tt_batch_size (from 200000 to 5000, which might be overly small, but it was for the sake of the experiment). Training went well and finished without error, but when I am launching the test program, I get the following error:

Traceback (most recent call last):
  File "tc.py", line 31, in <module>
    run_net(config, "test")
  File "util/model.py", line 402, in run_net
    nn.test()
  File "util/model.py", line 367, in test
    out = self.sess.run(self.output, feed_dict=self.get_feed_dict(b))
  File "util/model.py", line 135, in get_feed_dict
    ret_dict = {self.c1_ind: expand_dim_to_batch2(b.conv_ind[0], bs),
  File "util/cloud.py", line 26, in expand_dim_to_batch2
    out_arr[0:sp[0], :] = array
ValueError: could not broadcast input array from shape (45210,9) into shape (5000,9)

It seems that the batch size is too small. Is there a limit to this batch size? How can I know it?

Thanks in advance

[tatarchm]

Hi,

For the Stanford dataset, every room is processed in a single network pass which means the batch size needs to be sufficient to fit the full room. During training if the scan does not fit in the batch it is simply discarded (i.e. not used for training). For evaluation this is not possible so your batch size really needs to be large enough. What GPU are you using? 200000 points is actually not that much.

[sgiraudot]

The GPU is a Nvidia GeForce GT 750M.

Does this mean that, if I want to test the program on my own data sets, I need to make sure that each input file I'm providing is smaller than the batch size?

[tatarchm]

Does this mean that, if I want to test the program on my own data sets, I need to make sure that each input file I'm providing is smaller than the batch size?

Not necessarily. This is the default regime for the indoor datasets (Stanford and ScanNet) but you can also sample random chunks from your scans during training which is done for the outdoor dataset (Semantic 3D). At test time, the scans are labeled using a sliding window. You can check out the Semantic3D experiments.

[sgiraudot]

Okay, thank you for that clarification, I understand better how it works now.

I think you can probably close the issue as my main concern was to be limited on other data sets. Thanks again!

[tatarchm]

Great. Let me know if other concerns pop up.