How to use new BPE codes and vocab to generate parallel data

[dineshkh]

I have a few questions.

1. I have used the data in the folder https://github.com/facebookresearch/CodeGen/tree/main/data/test_dataset learned BPE codes and vocab using Monolingual Functions mode. I want to know how to use the .tok files in the zip file https://dl.fbaipublicfiles.com/transcoder/test_set/transcoder_test_set.zip and generate files like test.cpp_sa-java_sa.cpp_sa.pth using my bpe codes and vocab?

2. What is the contents of file test.cpp_sa-java_sa.cpp_sa.pth ? Also what is the difference between files test.cpp_sa-java_sa.cpp_sa.pth and test.cpp_sa-java_sa.java_sa.pth ?

3. I first preprocessed data in Monolingual mode, learned BPE codes, and then did my MLM training. Then I preprocessed the data in Monolingual Functions mode and learned new BPE codes and vocab. My question is, which vocab have you used to train CodeGen? Also, why two different BPE codes are learned?

[baptisteroziere]

Hi

1. You just need to pass your .code and .vocab files to preprocess.py using the fastbpe_vocab_path and fastbpe_code_path parameters. It will reload your BPE instead of computing a new one.
2. test.cpp_sa-java_sa.cpp_sa.pth contains the cpp functions for cpp-java translation and test.cpp_sa-java_sa.java_sa.pth contains the java functions.
3. We used the BPE codes and vocab learned from the monolingual dataset (used for MLM) when computing the monolingual function dataset using the parameters I mentioned in 1.

[dineshkh]

Thanks @brozi for the reply. Few more clarifications questions.

1. In case we don't have unit tests, then in AE and BT training do we require to create 'train.cpp_sa-java_sa.java_sa.pth', val.cpp_sa-java_sa.java_sa.pth, and test.cpp_sa-java_sa.java_sa.pth files ? In case what will be the value of --bt_steps parameter ?
2. If we have some parallel train set then just adding mt_steps will train the model on parallel train set ? Also what kind of loss function is used in mt_steps and bt_steps?
3. What is use of line 539 in evaluator.py

[baptisteroziere]

Hi @dineshkh,

1. You still need to keep the same bt_steps parameters no matter what your test/valid set is. If you don't have unit tests, you can set --eval_computation to false and your models will be evaluated with metrics such as BLEU, token accuracy, and perplexity. Our code would still look for parallel test/valid files to compute these metrics. You could either modify the code to not do any evaluation or add files with just one line with a simple example. Note: 'train.cpp_sa-java_sa.java_sa.pth' would be a train file with parallel cpp/java data and we don't use anything like that in TransCoder.
2. Yes the simplest way to use a parallel train set is to add mt_steps to train to translate. Both mt and bt steps are training using the cross-entropy. The difference is that we generate the source sentence in back-translation (allowing to train with monolingual datasets) while mt_steps use the sentence from the dataset.
3. It just limits the time spent evaluating MLM. It would probably be better to clean this up by making turning it into a parameter.

[dineshkh]

Thanks @brozi.

Can you please tell me how MT and BT steps are training using the cross-entropy loss ? Is it summing binary cross-entropy over all the tokens in the ground truth sequence? Any references or material you can point ?

[cadddr]

@brozi thanks for your explanations, but to me it is not unclear how the parallel files like train.cpp_sa-java_sa.java_sa.pth are produced. the preprocessing pipeline seems to process each language independently. at which stage are they connected? tokenization, bpe, or binarization stage? do I need to put both languages in a single json, somehow? thanks