Inferencing on a single video for CSL-Daily

[liu7826027]

I used the test_one_video.py code for inferring a single video from CSL-Daily. I changed num_classes to 2001 and encountered an error. Could you please guide me on what needs to be modified? The model is using the pretrained weights provided by you.

[hulianyuyy]

As we note in the readme.md, on csl-daily dataset, you should modify the model, with "To evaluate upon CSL-Daily with this checkpoint, you should remove the CorrNet block after layer2, i.e., comment line 102 and 145 in resnet.py and change the num from 3 to 2 in line 105, change self.alpha[1] & self.alpha[2] to self.alpha[0] & self.alpha[1] in line 147 & 149, respectively."

[liu7826027]

Yes, I just made the modification, and now I can perform inference. However, is it normal to have only two glosses in the output? What should I do?

[liu7826027]

As we note in the readme.md, on csl-daily dataset, you should modify the model, with "To evaluate upon CSL-Daily with this checkpoint, you should remove the CorrNet block after layer2, i.e., comment line 102 and 145 in resnet.py and change the num from 3 to 2 in line 105, change self.alpha[1] & self.alpha[2] to self.alpha[0] & self.alpha[1] in line 147 & 149, respectively."

Yes, I just made the modification, and now I can perform inference. However, is it normal to have only two glosses in the output? What should I do?

[hulianyuyy]

Normally, it should not produce only two words, but the model is not necessary to give exact translations. How you tried other videos?

[liu7826027]

通常，它不应该只产生两个单词，但模型不需要给出准确的翻译。你是如何尝试其他视频的？

When performing inference on the video S000544_P0000_T00 from the CSL-Daily dataset, and I haven't used any other sign language videos for inference yet, is it normal to have only two glosses in the output among the ones included in the dataset?Perhaps I need to perform inference on some other videos as well?

[hulianyuyy]

It seems that the label for this video is "担心 不 有 我 我 对 这 了解" where the model doesn't give correct outputs. Could you please try more input videos and observe their outputs?

[liu7826027]

It seems that the label for this video is "担心 不 有 我 我 对 这 了解" where the model doesn't give correct outputs. Could you please try more input videos and observe their outputs?

Yes, certainly. You said the tags I mentioned are correct. My output glosses are [['降' (drop), 0), ('抢' (snatch), 1)], and I don't know why. When using another video, the output glosses are [['看' (look), 0), ('山' (mountain), 1)]. Both are from videos in the dataset. I don't know where I went wrong, leading to this issue. I have already removed the second layer as required in the CorrNet block. I have attached some images for your reference. Could you please take a look?Perhaps there is an issue with gloss_dict.npy？At the very least, it shouldn't only output two words, right?

[hulianyuyy]

I have tried the same video (S000544_P0000_T00) with you, and i get the results as "output glosses : [[('担心', 0), ('不', 1), ('有', 2), ('我', 3), ('对', 4), ('这', 5), ('了解', 6)]]". Could you give me a full snapshot of your test_one_video.py?

[liu7826027]

我和你一起尝试了同样的视频（S000544_P0000_T00），我得到的结果为“输出光泽：[[（'担心'， 0）， （'不'， 1）， （'有'， 2）， （'我'， 3）， （'对'， 4）， （'这'， 5）， （'了解'， 6）]]”。你能给我一个完整的test_one_video.py快照吗？

Of course, I just tested another video again, and it's still the same, only outputting two. I'm attaching my script code.I have modified it into a text file for uploading. csl_daily_test_one_video.txt

[hulianyuyy]

I checked the test_one_video.py and found no error in it. I guess that the error may come from other locations. Could you please upload the resnet.py?

[liu7826027]

我检查了test_one_video.py，没有发现任何错误。我猜错误可能来自其他位置。你能上传 resnet.py 吗？

Okay, thank you very much for your patient responses. I am uploading my 'resnet.py' file. csl-daily-resnet.txt

[hulianyuyy]

It seems strange that i can't find any error in both files. Maybe you could load the weight for evaluation on csl-daliy dataset and see if the wer reaches 30.6. You may try to locate the errors step by step.

[liu7826027]

奇怪的是，我在这两个文件中都找不到任何错误。也许您可以在 csl-daliy 数据集上加载权重进行评估，看看 wer 是否达到 30.6。您可以尝试逐步找到错误。

Okay, my model is currently training. I'm planning to switch to a different device and give it a try. Thank you!!