questions about dict.txt and data samples specify methods

[freexxxyyy]

1. which step generates the dict.txt? It seems be generated during "fairseq-preprocess", but "fairseq-preprocess" also have a parameter "--srcdict $DICT_FILE".
2. how do you make the machine know the end of a data sample(which is a function in this case)? It seems that you use "\n", but functions also have "\n", I am confused about this.
3. Also, if fairseq-train FILENAME_OF_FIRST_DATASAMPLE: FILENAME_OF_SECOND_DATASAMPLE : FILENAME_OF_THIRD_DATASAMPLE:.....:FILENAME_OF_NTH_DATASAMPLE, will it work?

I am new to this. Thanks.

[wasiahmad]

1. Check the binarize.sh file.
2. We do not model newline (except python, we use NEW_LINE).
3. Study fairseq, I do not have the answer.

[freexxxyyy]

Thanks. But I want to know why you start from the 4th line and why you replace all numbers with 100 in

cut -f1 $SPM_VOCAB | tail -n +4 | sed "s/$/ 100/g" >$DICT_FILE

Also, I am still confused about how you split different training samples. How does the model which part belongs to one training data sample.

Thanks

[wasiahmad]

Because first three tokens are <s> </s> and <unk> and the numbers do not matter, so I just used a constant.

We do not split training samples, they are functions. If they are lengthy (>512), we truncated them.

[freexxxyyy]

Thanks. But the model will think each function as one training samples? How does the model know where a function ends?

[wasiahmad]

In fairseq, we use </s> to indicate end-of-sequence.

[freexxxyyy]

where do you insert between functions to indicate the end-of-sequence? I see that you add "\n" after each function in https://github.com/wasiahmad/PLBART/blob/main/data/github/preprocessing/src/utils.py#L128

Thanks