Cannot reproduce copy last mse on Kitti-Caltech dataset

coxlab / prednet

Code and models accompanying "Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning"

https://arxiv.org/abs/1605.08104

MIT License

765 stars 260 forks source link

Cannot reproduce copy last mse on Kitti-Caltech dataset #73

Open liuem607 opened 4 years ago

liuem607 commented 4 years ago

Hi, we tried to calculate copy last testing MSE score on Kitti-Caltech datasets, but we got 0.0022 instead of 0.007 as reported in your paper. I'm really curious how did you get this number? Thanks,

bill-lotter commented 4 years ago

Hi sorry for the confusion, but the issue may be that Caltech and Kitti are actually two different datasets, where Kitti is used to train the model and then in the paper, Caltech is used to test, whereas here the evaluation script just uses the Kitti validation set. Retrieving the raw Caltech data relies on matlab scripts etc and is more complicated

SaulZhang commented 4 years ago

@liuem607 Did you normalize the intensity of the image to 0~1 during your mean square error calculation?And Ialso have another question，@bill-lotter，I see that there are two modes for calculating mean square error in your script(data_utils.py line 26), one is "all" and the another is "unique". Which one do you use when calculating the copy last frame? If it is convenient, I hope you can open source the code for calculating copy last frame mse and ssim.Very grateful for your help!