loiccordone
commented
1 year ago Hi Lennard, thanks for your interest in my work! You are indeed right in your comprehension of the voxel cubes, it is unfortunately the code that is wrong... However, the results of the paper were indeed achieved with binary voxel cubes.
For a little context, initially I used the MinkowskiEngine library to handle the sparse tensors and generate the datasets, and the coalesce function of MinkowskiEngine does not sum the features. The paper results were obtained with this approach. Before making this repository available, to make the code less dependent on an external library I replaced the MinkowskiEngine functions with torch.sparse functions. I remember that some time after I encountered the same issue as you where the data is non-binary, and I must have forgot to update this repo.
I am sorry to have mislead you and probably others, you can see in my most recent work that the intended way of constructing binary samples is to:
- instead of storing a tuple (coords, feats), directly store the coalesced sparse_tensor in bool format (e.g. in my other repo). Since the sparse_tensor is first coalesced (summed) then converted to a boolean, any values >= 1 will be stored as a True:
sparse_tensor = torch.sparse_coo_tensor( coords.t().to(torch.int32), feats, (self.T, quantized_h, quantized_w, self.C), )
sparse_tensor = sparse_tensor.coalesce().to(torch.bool)
- getitem() now simply returns `self.samples[index][0].to_dense().permute(0,3,1,2), self.samples[index][1]` (the dense sample and the target)
I will quickly push a PR to correct this, thank you again.
LennardBo
ShristiDasBiswas
axiong2156
For my team's internal training pipeline I've derived a dataset based on the GEN1DetectionDataset code. When I load the data with the same hyperparameters (tbin, T, sample_size etc.) I've noticed that once I coalesce the sparse data as in L42, I loose the binary structure of the data and receive values bigger than 1. My assumption was that due to the quantizing nature of the tbins there simply were multiple events falling into one pixel bin (voxel cubes as in the paper) and as per Pytorch's documentation regarding coalescing duplicate values are simply summed up when coalescing sparse COO tensors.
However, the paper explicitly talks of binary voxel cubes which makes sense since we want to treat them as spikes for the SNNs. I fail to see where in the code this non-bianry coalesced data is actually made into binary data again. Am I missing something? Does the data originally not contain any events falling into the same time bin or do you use here actually non-binary data?