GaganHonor
commented
1 year ago ╰(°▽°)╯ohhhhhhhhhhhhh The error message you are getting is:
Got to 86
Got to 92
Got to 94This means that the code is reaching line 86, 92, and 94 of the example_instructions.py file. These lines are responsible for loading the model checkpoint, initializing the model parallel, and initializing the pipeline.
The reason why the code is hanging at this step is because the model checkpoint is too large to fit on the CPU. The CodeLlama-34b-Instruct model has a size of 1.6GB, which is larger than the 1.8GB of RAM that is available on the CPU.
To fix this error, you need to move the model checkpoint to the GPU. You can do this by running the following command:
cp CodeLlama-34b-Instruct.ckpt /tmp/CodeLlama-34b-Instruct.ckptOnce you have moved the model checkpoint to the GPU, you need to update the example_instructions.py file to load the model checkpoint from the GPU. You can do this by changing the map_location argument to "cuda".
The updated code should look like this:
checkpoint = torch.load("/tmp/CodeLlama-34b-Instruct.ckpt", map_location="cuda")Once you have made these changes, you should be able to run the example_instructions.py file without any errors.
Here are some additional things you can try:
- Make sure that you are using the correct version of PyTorch. The
torchruncommand requires PyTorch version 1.8 or higher. - Make sure that the
nccllibrary is installed in the same location as your PyTorch distribution. - Try running the
torchruncommand with the--use_glooflag. This will use the Gloo backend instead of NCCL.
If you are still having trouble, you can ask for help here ❇️❇️ I hope this helps!
mmr-crexi
Morning! I need help getting the models to run a second time, on a new instance.
Yesterday, I registered for and downloaded the models onto an AWS sagemaker instance. Everything worked fine and I was able to run
pip install -e .And from there experiment with the models. I shut down the instance and this morning started it again. I reran the pip installation, but now, everything hangs at this step:
This same code would finish loading the model after 8 seconds or so and be good to go. I've tried this with the 7b instruct model, the 13b instruct, and the 34b instruct; all worked fine yesterday, none work today.
How can I make this work? Did I forget some crucial step?
For the rest of this bug report, it's basically how I arrived at the conclusion that
checkpoint = torch.load(ckpt_path, map_location="cpu")Is not working, and I'm not sure why. Once I get to that point, the RAM usage rises from 1.8GB to 28.9GB, so it looks like it's at least found the first file in the checkpoint. This instance g5.12xlarge has 196GB and 4 24GB GPUs (and everything worked yesterday).
To figure this all out, I went into generation.py in the llama directory, and I added in some line number inspections. I added in lines like:
The code in generation now looks like:
and the run output looks like:
which is
checkpoint = torch.load(ckpt_path, map_location="cpu")My
pip freeze: