search

coxlab / prednet

Code and models accompanying "Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning"
https://arxiv.org/abs/1605.08104
MIT License
759 stars 259 forks source link

Longer Term Predictions #55

Closed mitkina closed 5 years ago

mitkina commented 5 years ago

Do you have any advice as to how to get the network to make longer term predictions (e.g., 2 seconds at 10 Hz) without completely deteriorating by the end?

Thanks!

bill-lotter commented 5 years ago

Hmm, is this the default model on KITTI? And doing t+1 prediction for each time step? Have you tried training it up to 20 timesteps or is this just in inference mode?

mitkina commented 5 years ago

This is not quite on the default KITTI data, but comparable. Doing t+1 works just fine - it's the actual longer term prediction (e.g., recursively predicting for 2 seconds from an input of 0.5 seconds) that deteriorates pretty quickly. I perform the fine-tune as suggested.

Thanks!

bill-lotter commented 5 years ago

Ah okay yeah it's probably just that the data gets non-deterministic as time goes on. Would probably need some type of GAN loss for longer term prediction.

mitkina commented 5 years ago

Got it. Do you have any context for how far you could push the prediction with the original KITTI data?

bill-lotter commented 5 years ago

I would say up to ~5 time steps it can still be okay. There are a few examples at the end of the page here: https://coxlab.github.io/prednet/. Figure 7 in the paper also has some quantitative info.

mitkina commented 5 years ago

Thanks! I was wondering if you tried pushing it any further than 0.5 second predictions. But I guess that's just the limit of the model! Thanks again for your help!

xiehousen commented 5 years ago

Ah okay yeah it's probably just that the data gets non-deterministic as time goes on. Would probably need some type of GAN loss for longer term prediction.

What you are talking about is using the discriminator network as a loss function or using the GAN network to get some loss function. Thanks for your reply.

bill-lotter commented 5 years ago

Thanks! I was wondering if you tried pushing it any further than 0.5 second predictions. But I guess that's just the limit of the model! Thanks again for your help!

No problem! Yeah there isn't a time step where it suddenly jumps up in MSE (it increases pretty linearly like Fig 7 suggests), but ~5 timesteps is a good rule of thumb.

bill-lotter commented 5 years ago

Ah okay yeah it's probably just that the data gets non-deterministic as time goes on. Would probably need some type of GAN loss for longer term prediction.

What you are talking about is using the discriminator network as a loss function or using the GAN network to get some loss function. Thanks for your reply.

Yeah it would be more along the lines of using a discriminator to add to the loss function.