Time window for coco dataset training and inference and time window and dataloader use for GEN1

[gwgknudayanga]

• coco.yaml is missing

• I just run the code for coco dataset. it seems the time window is always 1. According to your mem_update logic in this case it doesn't experience any voltage membrane decay and it just send the value of the received data clamp to 4 as maximum. So what is the time window that you have used to make it behave like spiking neurons?

• Even with that running i am getting following error when running. File "/home/atiye/SpikeYOLO/ultralytics/models/yolo/detect/val.py", line 146, in get_stats self.nt_per_class = np.bincount(stats[-1].astype(int), minlength=self.nc) # number of targets per class IndexError: list index out of range

• Another question is, how many time bins/time window you have used to organize the GEN1 dvs data? Can you please provide the code dataloader code for GEN1 as then the spiking neurons shows the membrane decay based real spiking neuron behavior?

• Also since mem_update is used for both train and validation it cannot see how binary spikes (1,0) are available during inference as mentioned in paper

[XinhaoLuo666]

1，3. We re-uploaded coco.yaml，you can try it again

2. you can change the time window in cfg/model/snn_yolov8.yaml [MS_GetT]
3. Time window of Gen1 is same as the paper reported. Gen1 dataset will upload a few weeks later
4. There is no function in the code to convert an integer into a pulse train, as this does not provide inference acceleration or power consumption on the GPU. But it's easy to implement, for example, you just need to turn a 3 into 3 1s

[gwgknudayanga]

@XinhaoLuo666, Regarding the GEN1 dataset evaluation, I organize each sample of the dataset to 5 time steps. each time step has 2 channels. And then i feed these to the network and trained your network . But the maximum mAP@50 that i could obtained in validation set is 36.1%. (I used horizontal,vertical as data augmentations. ). Don't know why i am getting these low values compared to the reported values in the paper.

[XinhaoLuo666]

For different T's, do you guarantee that their data-enhanced images are the same? Assuming that the image remains as it is in the first frame and the third image is flipped left and right, the model's ability to model the timing task will be significantly damaged

[gwgknudayanga]

@XinhaoLuo666 In GEN1 dataset there are no images and it's only the events, isn't it? I feed the direct events of a sample organizing them into a tensor of dimension (5,16,2,640,640) which are (T,N,C,H,W). So any augmentation happens to the whole tensor at once and hence similar to all 5 time steps in similar manner. I don't why the mAP@50 is 36.1% even after 200 iterations. It is better you can upload the code for GEN1 dataloading with guidelines/READme and also with sample weights file so that it is easy to reproduce the results and proced on top of it.

Thank you.

[feekyzz]

GEN1数据集如何使用呀，有没有哥们教一教我！