ArtemBiliksin
commented
3 weeks ago Hello, @Jun-Howie!
Most likely you did not have enough RAM.
You are using nsamples=1024, the Qwen2.5-32B-Instruct model, and the --offload_activations key. Using the --offload_activations key means that the inps (of size [1024, 4096, 5120]) and outs (of size [1024, 4096, 5120]) tensors will be stored in RAM. Here 1024 is the value of nsamples, 4096 is the default model_seqlen value, 5120 is the hidden_size value of the Qwen2.5-32B-Instruct model.
Let's calculate how much RAM you will need for inps and outs. In your case, the data type of the inps and outs tensors will be bfloat16, i.e. 2 bytes per tensor parameter. Hence,
- inps: 1024 4096 5120 * 2 / 1024 / 1024 = 40960 Mb,
- outs: 1024 4096 5120 * 2 / 1024 / 1024 = 40960 Mb.
In total we get 40960 + 40960 = 81920 Mb. You use the --offload_activations key, so this memory will be used in RAM.
You can get around this problem by taking a smaller value of nsamples (for example, nsamples=512) or by using multiple GPU devices without the --offload_activations key.
Jun-Howie
log (computing)
(aqlm) root@f9f90a551b02:~/xinglin-data/AQLM# bash train.sh wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information. wandb: (1) Create a W&B account wandb: (2) Use an existing W&B account wandb: (3) Don't visualize my results wandb: Enter your choice: 3 wandb: You chose "Don't visualize my results" wandb: Tracking run with wandb version 0.18.2 wandb: W&B syncing is set to
offlinein this directory.wandb: Run
wandb onlineor set WANDB_MODE=online to enable cloud syncing.============ Load model... ============ Loading checkpoint shards: 100%|██████████████████████████████| 17/17 [00:01<00:00, 11.35it/s] Loading pretrained model ... Model loaded sucсessfully ...
============ Quantizing model... ============ Loading data ... /root/xinglin-data/AQLM/src/datautils.py:219: FutureWarning: You are using
torch.loadwithweights_only=False(the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value forweights_onlywill be flipped toTrue. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user viatorch.serialization.add_safe_globals. We recommend you start settingweights_only=Truefor any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. data = torch.load(name)[:nsamples] Loaded data from /root/xinglin-data/AQLM/train.pt; len(data)=1024 sequencesStarting AQ quantization ... catching layer inputs from data train.sh: line 23: 28722 Killed python main.py $MODEL_PATH $DATASET_PATH --nsamples=1024 --val_size=0 --num_codebooks=1 --nbits_per_codebook=16 --in_group_size=32 --relative_mse_tolerance=0.01 --finetune_batch_size=32 --finetune_max_epochs=10 --finetune_early_stop=3 --finetune_keep_best --local_batch_size=1 --offload_activations --wandb --resume --save $SAVE_PATH
configure
export CUDA_VISIBLE_DEVICES=0 # or e.g. 0,1,2,3 export MODEL_PATH=/root/xinglin-data/model/Qwen/Qwen2.5-32B-Instruct export DATASET_PATH=/root/xinglin-data/AQLM/train.pt export SAVE_PATH=/root/xinglin-data/Qwen2 export WANDB_PROJECT=MY_AQ_EXPS export WANDB_NAME=COOL_EXP_NAME
python main.py $MODEL_PATH $DATASET_PATH \ --nsamples=1024 \ --val_size=0 \ --num_codebooks=1 \ --nbits_per_codebook=16 \ --in_group_size=32 \ --relative_mse_tolerance=0.01 \ --finetune_batch_size=32 \ --finetune_max_epochs=10 \ --finetune_early_stop=3 \ --finetune_keep_best \ --local_batch_size=1 \ --offload_activations \ --wandb \ --resume \ --save $SAVE_PATH