Can I train a model with SBR to detect any dataset?

[Jar7]

Hello, I understand the model is designed to detect landmarks for a video which has been trained. However, I am wondering whether I can train a model with large scale dataset to do inference for any dataset, so that the model can do runtime inference.

May I have some questions:

1. Have you succeed train such a model?
2. The backbone I used is a pretrained model which can output quite stable landmarks (with slight jitter), and I would like to further improve this slight jitter. In this case, would SBR help with this? Or SBR can improve only poor performance model with limited training data?
3. In your code, within the lk operation the batch is split and trained by sequence . Could you please tell why not design to proceed by batch ?

Thank you.

[D-X-Y]

1, We failed to train one model to work fine for many other datasets.

2, SBR could help in this case, but we found that SBR is sensitive to many aspects, such as hyper-parameters and resolution. It would cost much time to tune it in another dataset/model.

3, Since different videos may have different LK iterations to converge, it is not that straightforward to implement it by batch.

[Jar7]

Thank you for your reply!

1. Could you please confirm that you failed to train the model by the way you mentioned in the following issue?

   You need to retrain the model on another video. If you want to use the trained model on any video, it requires to train this model on a very large number of videos.

Originally posted by @D-X-Y in https://github.com/facebookresearch/supervision-by-registration/issues/40#issuecomment-475639856

2. Could you please share any experience or suggestion on the initialization of the hyper-parameters ? How to decide the value? The training process is very slow, after one epoch the lk loss is almost the same (0.4-0.7), is it normal?

[D-X-Y]

Hi @Jar7 , 1, Yes, training the model with SBR on the target video is necessary to get a visually good effect.

2, It is hard to give a precise hyper-parameter for me right now. In my experience, I would debug to see the intermedia LK results on the training data, and see whether these hyper-parameters cause many failure cases or not.

[Jar7]

Hi @Jar7 , 1, Yes, training the model with SBR on the target video is necessary to get a visually good effect.

2, It is hard to give a precise hyper-parameter for me right now. In my experience, I would debug to see the intermedia LK results on the training data, and see whether these hyper-parameters cause many failure cases or not.

Thank you!

[Jar7]

Hi @D-X-Y, Excuse me, May I have further questions..

I found that the pretrained model cpm_vgg16-epoch-049-050.pth output stable landmarks for demo-sbr.mp4. Has the demo-sbr,mp4 been trained? How many epoch should be trained for each video?

Have you found why a model for any video cannot be trained? Is it because the lk operation is sensitive to the video? If the lkloss is changed to another opt-flow loss, whether another additive loss would help to stabilize the output of backbone?