How to train unsupervised MT Language Model without parallel data?

[sylyoung]

I do not understand why I cannot run the training without provide parallel data. I get following error: File "train.py", line 323, in <module> ./data/en-kk/processed/en-kk/valid.en.pth not found ./data/en-kk/processed/en-kk/test.en.pth not found ./data/en-kk/processed/en-kk/valid.kk.pth not found ./data/en-kk/processed/en-kk/test.kk.pth not found

In this closed issue someone suggests

You can make empty or random files for those.

Originally posted by @fallcat in https://github.com/facebookresearch/XLM/issues/264#issuecomment-643005057

But it does not work on my end with empty maked-up files (valid.en.pth etc.) Loading data from ./data/en-kk/processed/en-kk/valid.en.pth ... ... EOFError: Ran out of input magic_number = pickle_module.load(f, **pickle_load_args)

My setting is as follows: export NGPU=2 python -m torch.distributed.launch --nproc_per_node=$NGPU train.py \ --exp_name test_enkk_mlm \ --dump_path ./dumped/ \ --data_path ./data/en-kk/processed/en-kk/ \ --lgs "en-kk" \ --clm_steps "" \ --mlm_steps "en,kk" \ --emb_dim 1024 \ --n_layers 6 \ --n_heads 8 \ --dropout 0.1 \ --attention_dropout 0.1 \ --gelu_activation true \ --batch_size 32 \ --bptt 256 \ --optimizer "adam,lr=0.0001" \ --epoch_size 200000 \ --validation_metrics "_valid_mlm_ppl" \ --stopping_criterion "_valid_mlm_ppl,10"

[sylyoung]

nvm I figured it out. need to change script to supply and process test/val mono data to the model