Issue with Extract point features

[the-king-of-yhq]

flatten = torch.sparse_coo_tensor(indices_non_zeros, weight_per_point, matrix_shape) RuntimeError: size is inconsistent with indices: for dim 0, size is 2 but found index 1100320154

[anhquancao]

which command did you run?

[the-king-of-yhq]

I perform this command: cd PaSCo/WaffleIron_mod python extract_point_features.py \ --path_dataset /gpfsdswork/dataset/SemanticKITTI \ --ckpt pretrained_models/WaffleIron-48-256__kitti/ckpt_last.pth \ --config configs/WaffleIron-48-256__kitti.yaml \ --result_folder /lustre/fsn1/projects/rech/kvd/uyl37fq/pasco_preprocess/kitti/waffleiron_v2 \ --phase val \ --num_workers 3 \ --num_votes 10 \ --batch_size 2

[anhquancao]

Do you use pytorch 1.11.0? And can you check if this self.im_idx is not empty?

[the-king-of-yhq]

I get the same error using both pytorch 1.13.0 and 1.11.0.

[anhquancao]

I am not sure why, I am asking the author of WaffleIron

[the-king-of-yhq]

OK！This error occurs when you run a little bit： 0%| | 2/23245 [00:03<12:14:45, 1.90s/it] flatten = torch.sparse_coo_tensor(indices_non_zeros, weight_per_point, matrix_shape) RuntimeError: size is inconsistent with indices: for dim 0, size is 2 but found index 140137744059904

[the-king-of-yhq]

Or can you provide the generated data waffleiron_v2?

[anhquancao]

Sorry, I didn't keep the files as our cluster has the policy of deleting data after 30 days unused. Still, you need this for later experiments with Robo3D

[the-king-of-yhq]

Thank you for your reply. I'll look for the error first

[anhquancao]

Another solution is to use the official code of WaffleIron and modify the file eval_kitti.py following extract_point_features.py.

Also, modify the kitti loader following here to keep only every fifth scan and here to also extract feature for training set..

This might take 1 or 2 days of work.

Then you can use the same command

python extract_point_features.py
--path_dataset /gpfsdswork/dataset/SemanticKITTI
--ckpt pretrained_models/WaffleIron-48-256__kitti/ckpt_last.pth
--config configs/WaffleIron-48-256__kitti.yaml
--result_folder /lustre/fsn1/projects/rech/kvd/uyl37fq/pasco_preprocess/kitti/waffleiron_v2
--phase val
--num_workers 3
--num_votes 10
--batch_size 2

[the-king-of-yhq]

Hi！ How much storage does waffleiron_v2 require？

[anhquancao]

Sorry, I never measure it.

[the-king-of-yhq]

Is 1T enough？ fq/pasco_preprocess/kitti/waffleiron_v2/sequences/00/seg_feats_tta/003615.pkl 16%|███████▊ | 3624/23245 [28:14<2:32:52, 2.14it/s] It need about 500G。

[anhquancao]

You should only save ~4000 files. Did you add this filter? We only need to extract features for scan 0, 5, 10, 15, ....

Additionally, you can optimize the dataset type used here. For example, vote -> uint8, embedding-> torch.float16

[the-king-of-yhq]

I mean doesn't he want to save 23245/5=4649 file? But I already need about 500GB running to 3624/5, so does he need about 4TB?

[anhquancao]

I didn't know it is that heavy. I suggest to optimize the data type here.

Also, I am saving embedding of 10 runs thus it has size of (10, N, 256). During training, I randomly take 1 of the them. Thus, maybe you can keep only 5 to save space.

[anhquancao]

Btw, could you tell me how did you solve the issue?

[the-king-of-yhq]

Sorry，I could not solve. I optimize the dataset type that vote -> uint8, embedding-> torch.float16, but it still need 600MB for only one pkl, i.e., 000000.pkl

[anhquancao]

I suggest change "embedding": embedding to "embedding": embedding[:1]. This probably will decrease memory by ~ 10 times

[the-king-of-yhq]

Does this affect the accuracy of the results？

[anhquancao]

If you retrain the model, I would say no. As for a pretrained model, I'm not certain -- you would need to evaluate it. In my view, this would primarily impact the variance of the predictions, which could influence uncertainty estimation. However, since I am using test-time augmentation, I don't expect the effect to be significant.