Hi, I increased the number of layers for the file with the error, and found that the error is still reported. I want to ask two questions. - Githubissues

CodedotAl / gpt-code-clippy

Full description can be found here: https://discuss.huggingface.co/t/pretrain-gpt-neo-for-open-source-github-copilot-model/7678?u=ncoop57

Apache License 2.0

3.29k stars 220 forks source link

Hi, I increased the number of layers for the file with the error, and found that the error is still reported. I want to ask two questions. #75

Closed BitcoinNLPer closed 2 years ago

BitcoinNLPer commented 2 years ago

Hi, I increased the number of layers for the file with the error, and found that the error is still reported. I want to ask two questions.

May I just clone this project and install the corresponding library, and then I can run run_clm_streaming_flax.py? Do I need to download more than 200g of training corpus into my local server? Or does it download automatically? Can you provide a more detailed pre-training guide?
The server I use has a GPU, so why do I run run_clm_streaming_flax.py directly and use the CPU by default?

Thanks!

Originally posted by @BitcoinNLPer in https://github.com/CodedotAl/gpt-code-clippy/issues/74#issuecomment-955669643