search

JusperLee / CTCNet

An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits
Apache License 2.0
69 stars 16 forks source link

about frcnn_128_512.backbone.pth.tar #1

Closed BiboGao closed 1 year ago

BiboGao commented 1 year ago

Thanks for sharing your great work.

I was trying to run the model, however I didn't find the pretrained frcnn_128_512.backbone.pth.tar for videonet. Could you please share it? Thanks.

JusperLee commented 1 year ago

Ok, this pre-train model is in this link: lip-reading model

BiboGao commented 1 year ago

Thanks for your reply. I still got some issues when I ran eval.py:

CTCNet-main\nichang\models\base_av_model.py", line 69, in from_pretrain from . import get ImportError: cannot import name 'get' from 'nichang.models' (unknown location)

I didn't find a function called get.

And I printed conf["model_name"] and it showed "AVFRCNN2". I didn't find a model called AVFRCNN2.

Thanks.

JusperLee commented 1 year ago

I'm very sorry for any confusion I may have caused you.

AVFRCNN2 is the name of my previous model, which should now be replaced by the CTCNet model. You may need to modify the "model class" in the pre-trained model, for example, replace all instances of "AVFRCNN2" with "CTCNet". As I am currently preparing a paper for a conference, it may take me some time to adjust the names in the pre-trained model.

BiboGao commented 1 year ago

No worries. Thanks.

I tried to replace class CTCNet with "AVFRCNN2" and got the error about mismatch: Error(s) in loading state_dict for AVFRCNN2: Missing key(s) in state_dict: "masker.video_block.video.0.proj.conv.weight", "masker.video_block.video.0.proj.conv.bias", "masker.video_block.video.0.proj.norm.weight", "masker.video_block.video.0.proj.norm.bias", "masker.video_block.video.0.proj.norm.running_mean"...............

Could you please check for me when you are available? Thanks.

JusperLee commented 1 year ago

No worries. Thanks.

I tried to replace class CTCNet with "AVFRCNN2" and got the error about mismatch: Error(s) in loading state_dict for AVFRCNN2: Missing key(s) in state_dict: "masker.video_block.video.0.proj.conv.weight", "masker.video_block.video.0.proj.conv.bias", "masker.video_block.video.0.proj.norm.weight", "masker.video_block.video.0.proj.norm.bias", "masker.video_block.video.0.proj.norm.running_mean"...............

Could you please check for me when you are available? Thanks.

Ok. I will check quickly.

JusperLee commented 1 year ago

I have modified the URL for the checkpoints in the README. You can use the new checkpoints, and I have tested them without any issues.

BiboGao commented 1 year ago

Thanks. Yes, it worked.