search

tpfister / caffe-heatmap

Caffe with heatmap regression & spatial fusion layers. Useful for any CNN image position regression task.
http://www.robots.ox.ac.uk/~vgg/software/cnn_heatmap
Other
164 stars 99 forks source link

Temporal pooling. #9

Open chandak opened 8 years ago

chandak commented 8 years ago

Hi Tomas, could you please guide me on how I should implement the cross-channel weighted sum-pooling (temporal, basically) layer since I could not find anything on that ?

tpfister commented 8 years ago

Hi,

Sure, happy to help. Sorry, I haven't had the bandwidth to clean up the temporal code for release.

Given a video you'd want to:

  1. Precompute dense optical flow
  2. Modify the data layer so it reads in multiple frames and the optical flow between them
  3. Add a warping layer which uses the optical flow to warp the heatmaps from the top spatial fusion layer (conv5_fusion in our prototxt)
  4. Add a single 1x1 convolution on top of the warping layer output with an L2 loss to the ground truth (so you now have 3 losses) & give it a higher weight (say loss_weight: 5)
  5. Train :)

We trained a single-frame network first (SpatialNet + Spatial Fusion Layers), initialised the above temporal network with its weights (plus gaussians for the added 1x1 convolution) & continued training.

Pull requests accepted if you or someone else has time to reimplement this in our public fork! :)

Hope that helps, Tomas

chandak commented 8 years ago

Yes, that helps a lot Tomas. Thanks a lot. Will keep in touch in case I have any further queries.

Sincerely. On Mon, Apr 11, 2016 at 6:22 AM, Tomas Pfister notifications@github.com wrote:

Hi,

Sure, happy to help. Sorry, I haven't had the bandwidth to clean up the temporal code for release.

Given a video you'd want to:

  1. Precompute dense optical flow
  2. Modify the data layer so it reads in multiple frames and the optical flow between them
  3. Add a warping layer which uses the optical flow to warp the heatmaps from the top spatial fusion layer (conv5_fusion in our prototxt)
  4. Add a single 1x1 convolution on top of the warping layer output with an L2 loss to the ground truth (so you now have 3 losses) & give it a higher weight (say loss_weight: 5)
  5. Train :)

We trained a single-frame network first (SpatialNet + Spatial Fusion Layers), initialised the above temporal network with its weights (plus gaussians for the added 1x1 convolution) & continued training.

Hope that helps! Tomas

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://contactmonkey.com/api/v1/tracker?cm_session=d859a7fc-e9e1-4cb3-b843-91b6b27560cc&cm_type=link&cm_link=ce95b597-c770-40fb-9b30-5637b005eaa3&cm_destination=https://github.com/tpfister/caffe-heatmap/issues/9#issuecomment-208103982

Regards, Chinmay Mahesh Chandak. 3rd year undergraduate, Computer Science and Engineering, IIT Hyderabad.

zimenglan-sysu-512 commented 8 years ago

hi @tpfister, can you release the result of SpatialNet + Spatial Fusion Layers on flic dataset?

jinyixin621 commented 7 years ago

Hi @tpfister how do you use the optical flow to warp 64x64 heatmaps? Thank you very much

mindcont commented 7 years ago

Hi, @jinyixin621 Have you found the answer yet?

Hi @tpfister how do you use the optical flow to warp 64x64 heatmaps? Thank you very much

I do not know how they warp optical flow file which is use deepflow2 precomputed from dataset to heatmaps like they said in paper ,have this part code released? flowing-convnets

jinyixin621 commented 7 years ago

Hi, @mindcont I didn't get reply from the author, but I implement one method by my own understanding.

nitba commented 6 years ago

Hi @jinyixin621 , where can I find your implementation on warping heatmap layer?

Hi, @mindcont I didn't get reply from the author, but I implement one method by my own understanding.