Could you please guide me on the steps if we want to pre-train the model only for the downstream tasks which require only one Programming language (Eg. Clone detection). Does this library incorporates this scenario, or do I need to make certain changes to accommodate it?
Also, the use of number of GPUs has been hardcoded for 8 GPU and not taken as an input argument from the user. Will this work if we have only 1 or 2 GPUs on the system?
What is the early stopping criteria for pre-training? What is the hyperparameter for the same
Yes, you need to make small changes. For example, you can only use your target language(s) during pretraining as mentioned here. If you do not need a certain language, just skip preparing its corresponding pretraining data.
This is not true. You can always vary the number of GPUs (see here). Please, make sure to adjust other hyper-parameters to achieve the desired batch size.
We used a fixed number of pre-training steps (100k). You can use validation loss as a stopping criterion too.
Hi @wasiahmad,
I have following queries:
Could you please guide me on the steps if we want to pre-train the model only for the downstream tasks which require only one Programming language (Eg. Clone detection). Does this library incorporates this scenario, or do I need to make certain changes to accommodate it?
Also, the use of number of GPUs has been hardcoded for 8 GPU and not taken as an input argument from the user. Will this work if we have only 1 or 2 GPUs on the system?
What is the early stopping criteria for pre-training? What is the hyperparameter for the same
Thank you, Aman