Training with custom data, what should I do?

[ToanICV]

Hi. Thank you for your works and open it. I'm trying to train with custom dataset from scratch. I have videos, glosses and texts as well. I plan to train with SingleStream first to have a baseline. I have trained with G2T and result is good (BLEU4 score: 43.76, ROUGE: 70.23). Now I'm training S2G but the result is not good (WER always ~ 95-100 and loss ~60). I believe that I missing somethings. So may you tell me a road map or something familiar to train with custom dataset? Thank you so much.

[ToanICV]

I have only 2 x GPUs P40 24GB. Train command:

python3 -m torch.distributed.launch --nproc_per_node 2 --use_env training.py --config experiments/configs/SingleStream/vsl-edu_s2g.yaml

This is my configs:

task: S2G data: input_data: videos #features, gloss zip_file: /root/working/vsl_edu_v2_15.zip input_streams:

• rgb
  dataset_name: vsl-edu level: word #word or char txt_lowercase: true max_sent_length: 400 train: /root/input/vsl-edu-labels/myVSL_p2_15.train dev: /root/input/vsl-edu-labels/myVSL_p2_15.dev test: /root/input/vsl-edu-labels/myVSL_p2_15.test transform_cfg: img_size: 224 aug_hflip: false color_jitter: false bottom_area: 0.9 csl_cut: False csl_resize:
• 224
• 224 center_crop: false center_crop_size: 270 randomcrop_threshold: 1 aspect_ratio_min: 0.75 aspect_ratio_max: 1.3 temporal_augmentation: tmin: 0.5 tmax: 1.5 testing: cfg: recognition: beam_size: 5 training: random_seed: 44 overwrite: False model_dir: /root/working/TwoStreamNetworkVi/TwoStreamNetwork/experiments/outputs/SingleStream/vsl-edu_s2g shuffle: True num_workers: 4 batch_size: 1 total_epoch: 100 keep_last_ckpts: 1 validation: unit: epoch freq: 1 cfg: recognition: beam_size: 1 optimization: optimizer: Adam learning_rate: default: 5.0e-2 weight_decay: 0.001 betas:
• 0.9
• 0.998 scheduler: cosineannealing t_max: 40 model: RecognitionNetwork: GlossTokenizer: gloss2id_file: /root/input/vsl-edu-labels/gloss2ids_vsl_p2_15.pkl s3d: pretrained_ckpt: pretrained_models/s3ds_glosscls_ckpt use_block: 4 freeze_block: 1 visual_head: input_size: 832 hidden_size: 512 ff_size: 2048 pe: True ff_kernelsize:
• 3
• 3

[HebaRaslan]

peace be upon you how you custom data to G2T task ...can you tell me the roadmap for it

[len2618187]

Have you found a solution?

[2000ZRL]

Well, it is difficult to debug given just the configs... Basically, to train on your custom dataset, you only need to use your own split files (xxx.train, xxx.dev, xxx.test), and modify the dataloader if necessary. For the S2G task, you only need to make sure the video-gloss pairs are correct.