spoonsso / dannce

MIT License
214 stars 30 forks source link

dannce-predict loss very small, but result same like normal but shift #138

Closed yuan0821 closed 1 year ago

yuan0821 commented 1 year ago

hi @davidhildebrand @spoonsso @data-hound

The loss of the prediction result is very low, and the relative position of each label is correct, but the result seems to be shifted as a whole, which does not match the video image. But the result of com-predict looks normal. Thank you so much for your time... ` viewGui = View3D(params, videos, skeleton);

pts3d = load('F:\dannce\demo\mouse20230129\DANNCE\predict_results\save_data_MAX0.mat')

viewGui.loadFrom3D(pts3d.pred) `

截屏2023-02-08 上午1 12 23 截屏2023-02-08 上午1 31 39
spoonsso commented 1 year ago

Can you plot the projected COM onto these images as well? Are you using a MAX model or an AVG model? If a MAX model, you may need to add the 3D COM coordinates to the predictions, depending on which version/branch you are using.

On Tue, Feb 7, 2023, 12:28 PM yuan0821 @.***> wrote:

hi @davidhildebrand https://github.com/davidhildebrand @spoonsso https://github.com/spoonsso @data-hound https://github.com/data-hound

The loss of the prediction result is very low, and the relative position of each label is correct, but the result seems to be shifted as a whole, which does not match the video image. But the result of com-predict looks normal. Thank you so much for your time... ` viewGui = View3D(params, videos, skeleton);

pts3d = load('F:\dannce\demo\mouse20230129\DANNCE\predict_results\save_data_MAX0.mat')

viewGui.loadFrom3D(pts3d.pred) `

[image: 截屏2023-02-08 上午1 12 23] https://user-images.githubusercontent.com/50692530/217318472-04e55bc1-0789-4a02-a367-40765b162316.png

— Reply to this email directly, view it on GitHub https://github.com/spoonsso/dannce/issues/138, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAW2P4ZECDIFUDLUASIN6E3WWKA2VANCNFSM6AAAAAAUUH5AAE . You are receiving this because you were mentioned.Message ID: @.***>

yuan0821 commented 1 year ago

Thank you @spoonsso I start with MAX and finetune with a 5-cam-pre-trained MAX network. I have four cameras, so I duplicate the camera1 video file as camera5. And also, copy the sync and label data in the .mat

net_type: MAX
train_mode: finetune
num_validation_per_exp: 0
max_num_samples: 1000
dannce_finetune_weights: 'F:\dannce\weight\MAX5\'

Please advise how to load com and dance predict results in the same image. I use the dannce_development branch. [https://github.com/spoonsso/dannce/tree/release_development]

And I also don't know how to add the 3d COM coordinates to the prediction; only use: new_predict=pred(:, :,1:16)+com(:,:);??? After the addition, the result same a little better, but still shifted.

截屏2023-02-08 上午2 04 56
yuan0821 commented 1 year ago

hi @spoonsso thank you for suggestion. the result is normal now after add COM corrdinates

for i=1:16
    new_predict(:,:,i)=dannce_predict.pred(:,:,i)+com_predict.com;
end

skeleton = load('F:\dannce\Label3D\skeletons\rat16.mat');
close all
viewGui = View3D(params, videos, skeleton);
viewGui.loadFrom3D(new_predict(framesToLabel,:,:));
截屏2023-02-08 上午2 39 49
spoonsso commented 1 year ago

Perfect, glad it is working now. By the way, the reason why the end of your tail is not being tracked well is because it is most likely outside of the 3D extent of the image volume.

@data-hound there is an addCOM argument in processing.savedata_tomat() that would solve this problem during dannce prediction, but it looks like it is not being used. Can we add this to the release you are working on?

data-hound commented 1 year ago

Yes, we can add this addCOM argument to the savedata_tomat(). This should be done for MAX model predictions, correct?

spoonsso commented 1 year ago

Yes, the AVG predictions already have the COM built in b/c they use the world grid coordinates in the calculation.

On Tue, Feb 7, 2023, 2:50 PM Anshuman Sabath @.***> wrote:

Yes, we can add this addCOM argument to the savedata_tomat(). This should be done for MAX model predictions, correct?

— Reply to this email directly, view it on GitHub https://github.com/spoonsso/dannce/issues/138#issuecomment-1421354161, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAW2P446HYD233MBOOZJDFLWWKRQDANCNFSM6AAAAAAUUH5AAE . You are receiving this because you were mentioned.Message ID: @.***>