Detailed code running steps

[kido1412y2y]

Hello, author. Could you please provide a specific code execution process, such as which code to run in the first step. I really want to reproduce your code, but my ability is really insufficient. I sincerely hope you can help me。

[ycmin95]

Thank you for your interest in our work. Currently, I do not have any plan to provide a step-by-step tutorial, but I will consider doing so in the future (maybe after June). You can reproduce the experiment following the README, as all necessary steps are clearly outlined there. If you have any questions, you can post them in an issue, which would be convenient for others to refer to as well.

[kido1412y2y]

Thank you very much for your comments and I will try to use it. Thank you sincerely for your answer.

[kido1412y2y]

Hello, author. I sincerely hope you can give me some help when you have time.

When I am running the following Data Preparation step 3

cd ./preprocess python data_preprocess.py --process-image --multiprocessing

The terminal display is as follows

(base)user5@com:~/Downloads/VAC_CSL-main/preprocess$ python: can't open file '/home/user5/Downloads/VAC_CSLR-main/preprocess/data_preprocess.py': [Errno 2] No such file or directory

Then, I run dataset_preprocess.py in the pycharm

FileNotFoundError: [Errno 2] No such file or directory: '../dataset/phoenix2014/phoenix-2014-multisigner/annotations/manual/dev.corpus.csv'

I have already done the second step of Data Preparation, I can also see the phienix2014 after the soft link in the dataset. home/public/ywy1/phoenix-2014-release home/user5/Downloads/VAC_CSL-main/dataset/phoenix-2014

[ycmin95]

You can check whether this file "dev.corpus.csv" exists, which is the annotation file of phienix2014 dataset.

[kido1412y2y]

Hello, author.Thank you for your answer. I checked the folder and found this file "dev.corpus.csv". I'm sorry to mislead you, my file name is wrong, Ha ha ha.

I see home/user5/Downloads/VAC_CSL-main/dataset/phoenix-2014

actually home/user5/Downloads/VAC_CSL-main/dataset/phienix-2014

Author, I have a little comment that you can edit again README Data Preparation step 3

before ln -s PATH_TO_DATASET/phoenix2014-release ./dataset/phienix2014

after ln -s PATH_TO_DATASET/phoenix2014-release ./dataset/phoenix2014

[kido1412y2y]

I encountered a problem while running the "python main.py --load-weights resnet18_slr_pretrained.pt --phase test"

File "/home/user5/Downloads/VAC_CSLR-main/utils/device.py"，line 57,in occupy.gpu
   torch.zeros(1).cuda(g)

RuntimeError: CUDA error : invalid device ordinal CUDA kernel errors might be asynchronously reponted at some other API call, so the stacktrace below might be inconect.For debugging consider passing cUDA_LAUNCH_BLOCKING=1.

I running "print(torch.cuda.device_count())"

show 1

I running if torch.cuda.is_available(): print('CUDA device is available.') else: print("CUDA device is not available.")

show CUDA device is available.

[ycmin95]

Thank you for pointing out this typo and it has been corrected. The default configuration file use two GPUs. You can modify it according to your actual situation. For example, if you only want to use one card (e.g., cuda:0), you can make the necessary changes accordingly:

python main.py --load-weights resnet18_slr_pretrained.pt --phase test --device 0

[kido1412y2y]

Hello, author. I ran into some problems while running the suggestions you gave me python main.py --load-weights resnet18_slr_pretrained.pt --phase test --device 0

Terminal display FileNotFoundError: [Errno 2] No such file or directory: 'resnet18_ sLr_pretrained.pt'

I downloaded the file on Baidu pan. The name of the file is "resnet18_slr_pretrained_distill25.pt" . I put it in this directory.

home/user5/Downloads/VAC_CSL-main/resnet18_slr_pretrained_distill25.pt (Didn't decompress it)

I running python main.py --load-weights resnet18_slr_pretrained_distill25.pt --phase test --device 0

Terminal display Loading model Loading model finished. Loading data train 5671 Apply training transform.

train 5671 Apply testing transform.

dev 540 Apply testing transform.

test 629 Apply testing transform.

Loading data finished . .git does not exist in current dir [ Thu Mar 23 17:09:44 2023 ] Nodel: slr_network.SLRModel. [ Thu Mar 23 17:09:44 2023 ] Weights: resnet18_slr_pretrained_distill25.pt. 0%| l 0/68 [00:00<?,?it/s] [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thnead-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc: 98] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) lW pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thead-pool after fork.(function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thnead-pool after fork. (function pthreadpool) [W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork.(function pthreadpool) lW pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork.(function pthreadpool) 0%| l 0/68 [00:00<?,?it/s] Traceback (most recent call last): File "/home/user5/Downloads/VAC_CSLR-main/main.py"，line 211,in processor.start( ) File "/home/user5/Downloads/VAC_CSLR-main/main.py",line 62,in start dev_wer = seq_eval(self.arg,self.data_loader[ " dev"],self.model，self.device, File "/home/user5/Downloads/VAC_CSLR-main/seq_scripts.py"，line 49， in seq_evalfor batch_idx, data in enumerate(tqdm(loader)): File "/hnome/com/anaconda3/envs/ywypytorch/lib/python3.9/site-packages/tqdm/std.py"，line 1178，in .iter for obj in iterable: File "/nome/com/anaconda/envs/ywypytonch/lib/python3.9/site-packages/torch/utils/data/dataloader.py",line 521,in .. e.t.. data = self._next_data() File " /hnome/con/enacondaB/envs/ywypytonch/lit/python3.9/site-packages/toreh/utils/data/dataloader.py",，line 1283，in _next_data return self.__process_data(data) File "/hore/con /anaconds/envs/ ymypytonet/1tib/pythons.9/site-packages/tonch/utils/data/dataloader.py"，line 1229，in process datadata.reraise( File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/site-packages/torch/_utils.py"，line 425，in reraise raise self.exc_type(msg) IndexError: Caught IndexError in DataLoader worker process 0. Original Traceback (most recent call last): File " /hone/conl/anacondal/envs/ymypytorcth/ltitl/python5.9/site-packages/torch/utils/datal_utils/wonker .py"line 287，in _worker_loop data = fetcher.fetch(index) File "/hnone/com/anaconda/envs/ywypytorch/Lio/python3.9/site-packages/torch/utits/datal_utils/fetch.py" ，line 44， in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/hore/con/anaconda/envs/ywypytorch/titb/python3.9/site-packages/torch/utils/data/ _uti1s/fetch.py"，line 44， in <listcomp; data = [self.dataset[idx] for idx in possibly_batched_index] File "/nome/user5/Downloads/VAC_CSLR-main/dataset/dataloadervideo.py" , line 47，in -.getitem.- input_data，label = self.normalize(input_data,label) File "/nome/user5/Downloads/VAC_CSLR-main/dataset/dataloader_video.py"，line 78，in normalize video,label = self.data_aug(video,label, file_id) File "/home/user5/Downloads/VAC_CSLR-main/utils/video_augmentation.py"，line 24，in ..call-. image = t(image) File "/home/user5/Downloads/VAC_CSLR-main/utils/video_augmentation.py", line 157，in ._call.. im_h, im_w, im_c = clip[0].shape IndexErrOr: list index out of range

[ycmin95]

Hi, @kido1412y2y，similar issue here, you can check whether processed images are found.

[kido1412y2y]

I see it, but I really don't understand. Is meaning there is an error on line 62 of the "dataloader_video.py"?

How do I check whether processed images are found, Could you please describe it in more detail? What are the areas I need to compare and where do I need to correct them?

[ycmin95]

During preprocessing, frames are extracted from video, the glob function here is adopted to load frames. When it fails (the path is wrong or the frames are not extracted), the img_list is empty and leads to IndexError.

[kido1412y2y]

Hello, author, I found that there seems to be a problem with data preparation, and the original image has not been successfully adjusted to 256x256.

Without this file "/dataset/phoenix2014/phoenix-2014-multisigner/features/fullFrame-256x256px"

Only this file"/dataset/phoenix2014/phoenix-2014-multisigner/features/fullFrame-210x260px" "/dataset/phoenix2014/phoenix-2014-multisigner/features/trackedRightHand-92x132px"

I running cd ./preprocess python data_preprocess.py --process-image --multiprocessing

Terminal display Generate information dict from ../dataset/phoenix2014/phoenix-2014-multisigner/annotations/manual/dev.corpus.csv 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 540/540 [00:00<00:00, 5993.77it/s] Resize image to 256x256px 0%| | 0/540 [00:00<?, ?it/s] multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/home/user5/Downloads/VAC_CSLR/preprocess/dataset_preprocess.py", line 73, in resize_dataset os.makedirs(rs_img_dir) File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/os.py", line 215, in makedirs makedirs(head, exist_ok=exist_ok) File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/os.py", line 215, in makedirs makedirs(head, exist_ok=exist_ok) File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/os.py", line 215, in makedirs makedirs(head, exist_ok=exist_ok) File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/os.py", line 225, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '../dataset/phoenix2014/phoenix-2014-multisigner/features/fullFrame-256x256px'

I running sudo python dataset_preprocess.py --process-image --multiprocessing

Terminal display [sudo] user5 password： File "dataset_preprocess.py", line 22 print(f"Generate information dict from {anno_path}") ^ SyntaxError: invalid syntax

I have modified the file permissions through chmod 777 drwxrwxrwx 4 2522 2000 4096 11月 3 2017 phoenix2014- release

[ycmin95]

Perhaps you need to check the permission of features and features/fullFrame-256x256px directories and there is no need to run the script with sudo.

[Dikay1]

Hello , when I am running : _python main.py --work-dir /home/user/VAC_CSLR-main/work_dir/baseline_res18/ --config /home/VACCSLR-main/configs/baseline.yaml --device 0 , the error will be reported as follows:

Loading model Loading model finished. Loading data train 5671 Apply training transform.

train 5671 Apply testing transform.

dev 540 Apply testing transform.

test 629 Apply testing transform.

Loading data finished. .git does not exist in current dir [ Mon Mar 27 08:21:39 2023 ] Model: slr_network.SLRModel. [ Mon Mar 27 08:21:39 2023 ] Weights: []. 0%| | 0/540 [00:01<?, ?it/s] Traceback (most recent call last): File "/home/user/VAC_CSLR-main/main.py", line 211, in processor.start() File "/home/user/VAC_CSLR-main/main.py", line 63, in start "dev", 6667, self.arg.work_dir, self.recoder, self.arg.evaluate_tool) File "/home/user/VAC_CSLR-main/seq_scripts.py", line 56, in seq_eval ret_dict = model(vid, vid_lgt, label=label, label_lgt=label_lgt) File "/home/user/anaconda3/envs/ctc_torch1.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/home/user/VAC_CSLR-main/slr_network.py", line 97, in forward else self.decoder.decode(outputs, lgt, batch_first=False, probs=False) File "/home/user/VAC_CSLR-main/utils/decode.py", line 28, in decode return self.BeamSearch(nn_output, vid_lgt, probs) File "/home/user/VAC_CSLR-main/utils/decode.py", line 42, in BeamSearch beam_result, beam_scores, timesteps, out_seq_len = self.ctc_decoder.decode(nn_output, vid_lgt) File "/home/user/anaconda3/envs/ctc_torch1.8/lib/python3.6/site-packages/ctcdecode/init.py", line 40, in decode output, timesteps, scores, out_seq_len) RuntimeError: Not enough space

After I set batch_size and test_batch_size to 1, this problem will still occur. Is there not enough graphics card memory or something else?(gpu can run normally) Looking forward to your reply.

[ycmin95]

Hi, @Dikay1. I have not met this problem before, it seems like a problem of ctcdecoder, you can find some information from this issue.

[xxxiaosong]

Hello , when I am running : _python main.py --work-dir /home/user/VAC_CSLR-main/work_dir/baseline_res18/ --config /home/VACCSLR-main/configs/baseline.yaml --device 0 , the error will be reported as follows:

Loading model Loading model finished. Loading data train 5671 Apply training transform.

train 5671 Apply testing transform.

dev 540 Apply testing transform.

test 629 Apply testing transform.

Loading data finished. .git does not exist in current dir [ Mon Mar 27 08:21:39 2023 ] Model: slr_network.SLRModel. [ Mon Mar 27 08:21:39 2023 ] Weights: []. 0%| | 0/540 [00:01<?, ?it/s] Traceback (most recent call last): File "/home/user/VAC_CSLR-main/main.py", line 211, in processor.start() File "/home/user/VAC_CSLR-main/main.py", line 63, in start "dev", 6667, self.arg.work_dir, self.recoder, self.arg.evaluate_tool) File "/home/user/VAC_CSLR-main/seq_scripts.py", line 56, in seq_eval ret_dict = model(vid, vid_lgt, label=label, label_lgt=label_lgt) File "/home/user/anaconda3/envs/ctc_torch1.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, kwargs) File "/home/user/VAC_CSLR-main/slr_network.py", line 97, in forward else self.decoder.decode(outputs, lgt, batch_first=False, probs=False) File "/home/user/VAC_CSLR-main/utils/decode.py", line 28, in decode return self.BeamSearch(nn_output, vid_lgt, probs) File "/home/user/VAC_CSLR-main/utils/decode.py", line 42, in BeamSearch beam_result, beam_scores, timesteps, out_seq_len = self.ctc_decoder.decode(nn_output, vid_lgt) File "/home/user/anaconda3/envs/ctc_torch1.8/lib/python3.6/site-packages/ctcdecode/init**.py", line 40, in decode output, timesteps, scores, out_seq_len) RuntimeError: Not enough space

After I set batch_size and test_batch_size to 1, this problem will still occur. Is there not enough graphics card memory or something else?(gpu can run normally) Looking forward to your reply.

Hello. I had the same problem. Have you solved it yet?

[Dikay1]

Hi, @xxxiaosong .It's been a bit long and I can't remember some of it, I may have reinstalled ctcdecoder.

[xxxiaosong]

Hi, @xxxiaosong .It's been a bit long and I can't remember some of it, I may have reinstalled ctcdecoder.

Thanks for your reply.

[kido1412y2y]

Hello author ! When I run : python main.py --work-dir ./work_dir/vac/ --config ./configs/baseline.yaml --device 0 ，the error occurred:

/(vac) user2@com:~/data/VAC_CSLR-main$ python main.py --work-dir ./work_dir/vac/ --config ./configs/baseline.yaml --device 0 Loading model /opt/anaconda3/envs/vac/lib/python3.7/site-packages/torchvision/models/_utils.py:209: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead. f"The parameter '{pretrained_param}' is deprecated since 0.13 and may be removed in the future, " /opt/anaconda3/envs/vac/lib/python3.7/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=ResNet18_Weights.IMAGENET1K_V1. You can also use weights=ResNet18_Weights.DEFAULT to get the most up-to-date weights. warnings.warn(msg) Loading model finished. Loading data train 5671 Apply training transform.

train 5671 Apply testing transform.

dev 540 Apply testing transform.

test 629 Apply testing transform.

Loading data finished. .git does not exist in current dir [ Thu Jul 13 21:34:29 2023 ] Parameters: {'work_dir': './work_dir/vac/', 'config': './configs/baseline.yaml', 'random_fix': True, 'device': '0', 'phase': 'train', 'save_interval': 5, 'random_seed': 0, 'eval_interval': 1, 'print_log': True, 'log_interval': 50, 'evaluate_tool': 'sclite', 'feeder': 'dataset.dataloader_video.BaseFeeder', 'dataset': 'phoenix14', 'dataset_info': {'dataset_root': './dataset/phoenix2014/phoenix-2014-multisigner', 'dict_path': './preprocess/phoenix2014/gloss_dict.npy', 'evaluation_dir': './evaluation/slr_eval', 'evaluation_prefix': 'phoenix2014-groundtruth'}, 'num_worker': 10, 'feeder_args': {'mode': 'test', 'datatype': 'video', 'num_gloss': -1, 'drop_ratio': 1.0, 'prefix': './dataset/phoenix2014/phoenix-2014-multisigner', 'transform_mode': False}, 'model': 'slr_network.SLRModel', 'model_args': {'num_classes': 1296, 'c2d_type': 'resnet18', 'conv_type': 2, 'use_bn': 1, 'share_classifier': False, 'weight_norm': False}, 'load_weights': None, 'load_checkpoints': None, 'decode_mode': 'beam', 'ignore_weights': [], 'batch_size': 2, 'test_batch_size': 4, 'loss_weights': {'SeqCTC': 1.0}, 'optimizer_args': {'optimizer': 'Adam', 'base_lr': 0.0001, 'step': [20, 35], 'learning_ratio': 1, 'weight_decay': 0.0001, 'start_epoch': 0, 'nesterov': False}, 'num_epoch': 40}

0%| | 0/2835 [00:00<?, ?it/s] Traceback (most recent call last): File "main.py", line 211, in processor.start() File "main.py", line 45, in start self.device, epoch, self.recoder) File "/home/user2/data/VAC_CSLR-main/seq_scripts.py", line 18, in seq_train for batch_idx, data in enumerate(tqdm(loader)): File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/tqdm/std.py", line 1178, in iter for obj in iterable: File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in next data = self._next_data() File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data return self._process_data(data) File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data data.reraise() File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise raise exception IndexError: Caught IndexError in DataLoader worker process 0. Original Traceback (most recent call last): File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop data = fetcher.fetch(index) File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/user2/data/VAC_CSLR-main/dataset/dataloader_video.py", line 47, in getitem input_data, label = self.normalize(input_data, label) File "/home/user2/data/VAC_CSLR-main/dataset/dataloader_video.py", line 78, in normalize video, label = self.data_aug(video, label, file_id) File "/home/user2/data/VAC_CSLR-main/utils/video_augmentation.py", line 24, in call image = t(image) File "/home/user2/data/VAC_CSLR-main/utils/video_augmentation.py", line 119, in call if isinstance(clip[0], np.ndarray): IndexError: list index out of range

Can you help me please, Thank you very much!

I have successfully used you provide the pretrained models to inference, and have also obtained the same results. python main.py --load-weights resnet18_slr_pretrained_distill25.pt --phase test --device 0