train code complete from zero

[yuandaxing]

Hi， I try to train code complete from zero， below is two settings， and the performance is bad:

1. train from utf-8 encode, the default BPE encoding, 1 batch_size, 3 GPU card, 500K iteration
2. train from ascii encode, 1 batch_size, 3 GPU card, 500K iteration and method 2 is just map ascii from 1 to N, the size is much less, and the performance of the two settings ard bad. do you train by finetune the release model of gpt2? can you share your training settings?

[iedmrc]

I should say that, currently, Galois is not at its best stage. I made just a little bit finetuning works but still, results are not bad. These are the configs/parameters that I used:

• Base model: GPT-2 345M
• Encoder: Default BPE
• Batch size: 1
• Gradient accumulation: 5 (As I remember)
• Optimizer: Adam
• Learning rate: 1e-5 or 1e-4
• Device: 1 V100
• Iteration: I interrupted when loss and unconditional generations were well enough but it took almost 48 hours.
• Dataset: ~45K preprocessed python files. Make sure that your dataset is well enough. Don't forget that GPT2 is powerful enough to learn any well-structured (I mean all programming languages). So should either find the best parameters experimentally or make sure that your dataset is well enough.

Nowadays, I suggest who wants to train a generative model to use huggingface's transformers library.

[yuandaxing]

great thanks for your detailed help