About MVSEC dataset

[Candy-Crusher]

Could you please consider sharing the code for processing the MVSEC dataset on GitHub as well?

[J-Cuadrado]

We have just added the directory 'mvsec_dataset' with the corresponding files, as well as some instructions to use them.

[Candy-Crusher]

Thank you kindly for generously sharing the code!

[Candy-Crusher]

Could I ask why you use the file valid_split_doubleseq.csv to duplicate the same data? In the DSECDatasetLite class, I noticed that you concatenate the first and second columns of the CSV file. Should I also perform this kind of processing for the MVSEC dataset? If possible, could you kindly share your MVSECDataset class as well?

[J-Cuadrado]

We do not duplicate the same data. When working with the DSEC Dataset, the model receives 21 consecutive event histograms as an input. However, during the frame duration optimization, we found the optimal duration of each histogram to be of roughly 9ms, i.e. 11 event histograms between two consecutive ground-truth labels. As such, for every optical flow ground-truth map, we feed the model the previous ~192ms, which requires us to use two consecutive event files. We could have stored 21 histograms per file, but that would have roughly doubled the necessary memory, so we opted for this approach to avoid data duplication.

When working with the MVSEC Dataset, though, since it is a smaller dataset with lower memory requirements, this situation does not pose any problem, so redundant data can be easily stored. Thus, with MVSEC, single .npz files containing at the same time the ground-truth data, the valid pixel mask and the 21 event histograms are created, and therefore no .csv file is needed to train on it.

All the files needed to train on MVSEC have been included in the _mvsecdataset directory, and the readme.md file has been edited accordingly.

[Candy-Crusher]

Thank you very much for your response and the newly shared file. It has been incredibly helpful! However, I encountered some difficulties when attempting to replicate the indoor part of the MVSEC dataset (Lmod = 1.85 for split1 while the AEE result in the paper is 0.58). I noticed that the files you uploaded were specifically for the outdoor portion. As a result, I made some modifications to the test_mvsec section in order to adapt it for the indoor scenario: valid_dataset = MVSECDataset(root='/xxx/OF_EV_SNN/mvsec_dataset/files/saved', split=split, condition='test'). I would like to inquire if there are any other areas that require modification or if it would be possible for you to kindly share the code specifically tailored for the indoor section. Your assistance would be greatly appreciated!

[J-Cuadrado]

All the codes you need are already provided. You simply need to choose the dataset corresponding to your condition (indoor or outdoor) and choose the path where your saved checkpoint is. Also, in the _testmvsec.py file, lines 86-88 are only needed for outdoor scenarios, you need to comment them for input splits (we have added a comment addressing this situation in the script, sorry for the inconvenience). You may also need to slightly modify the imported modules, depending on how you have stored them, as well as the save paths. Bear in mind that the example provided in this GitHub repository corresponds to a model pre-trained on DSEC. Our reported results on MVSEC, though, were obtained after training on the second dataset.

[Candy-Crusher]

Thank you so much for your generous share! The problem is solved and the result shown in the paper is correct!

[CatherineCloud]

Could you plz explain why you choose 1e-5 as the threshold for flow at a valid pixel? As in the paper you mentioned: "We followed a training approach akin to the one adopted for the DSEC dataset, i.e., we only considered pixels with either zero-valued ground-truth (x- and y- components of the optical flow vector below a small threshold thr =1e − 5) or with unknown flow values as invalid, and only trained on valid pixels."

[J-Cuadrado]

On the DSEC dataset, only pixels with a known ground-truth value are considered for training and evaluation. However, on the MVSEC dataset, only a dense flow map is provided. As such, zero-valued pixels in this dataset can be due to two different situations: either there is indeed no optical flow on those pixels, or they are originated by "holes" in the lidar measurements, and therefore unknown (the real optical flow value would in fact be nonzero). To filter out all of those pixels, we chose an arbitrary small threshold of 1e-5, and pixels were both the x- and y- optical flow components are below said threshold are neglected in training and evaluation.