"error" in training - AttributeError: 'CastOutputToFloat' object has no attribute 'weight', RuntimeError: Only Tensors of floating point and complex dtype can require gradients

[GreenTeaBD]

WSL2 Ubuntu, new install, I get the following error after it downloads the weights and tries to train. Sorry I can't give more details, but I'm really not sure what's going on.

Number of samples: 534 Traceback (most recent call last): File "/home/ckg/.local/lib/python3.10/site-packages/gradio/routes.py", line 394, in run_predict output = await app.get_blocks().process_api( File "/home/ckg/.local/lib/python3.10/site-packages/gradio/blocks.py", line 1075, in process_api result = await self.call_function( File "/home/ckg/.local/lib/python3.10/site-packages/gradio/blocks.py", line 884, in call_function prediction = await anyio.to_thread.run_sync( File "/home/ckg/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/home/ckg/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/home/ckg/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, args) File "/home/ckg/.local/lib/python3.10/site-packages/gradio/helpers.py", line 587, in tracked_fn response = fn(args) File "/home/ckg/github/simple-llama-finetuner/main.py", line 164, in tokenize_and_train model = peft.prepare_model_for_int8_training(model) File "/home/ckg/.local/lib/python3.10/site-packages/peft/utils/other.py", line 72, in prepare_model_for_int8_training File "/home/ckg/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1614, in getattr raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'CastOutputToFloat' object has no attribute 'weight'

[lxe]

What hardware are you running on? Any other console traceback?

[GreenTeaBD]

I just figured out an important thing while I was typing this comment. I'll include what I was writing before at the end but it appears to finetune correctly if I kill it and run main.py again. If I start finetuning once and then abort, I will get that error on every other attempt. I don't know if it actually will finetune successfully because it's currently running, but yeah, that seems like it's the problem, errors out on a 2nd attempt after an abort.

I did try deleting the leftover directory after the abort, in case that was getting in the way, but that didn't seem to do it. Much less of a problem now since killing the script and restarting isn't a big hassle but, still, probably not running as expected.

Anyway, the original comment: Windows 11 (but in WSL2), otherwise working WSL install for diffusers and GPT-Neo finetuning (but, in their own environments) A 5900x, 32GB ram, 100GB swap for WSL2 (I needed a lot for GPT-Neo stuff), and a 4090 for the gpu.

Nothing else to the traceback, everything else in the console is, what I think is, just normal stuff before (llama-finetuner) ckg@Ryzen9:~/github/simple-llama-finetuner$ python main.py

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

CUDA SETUP: CUDA runtime path found: /home/ckg/anaconda3/envs/llama-finetuner/lib/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 8.9 CUDA SETUP: Detected CUDA version 117 CUDA SETUP: Loading binary /home/ckg/.local/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so... Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch(). Loading base model... Loading base model... The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. The tokenizer class you load from this checkpoint is 'LLaMATokenizer'. The class this function is called from is 'LlamaTokenizer'. The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. The tokenizer class you load from this checkpoint is 'LLaMATokenizer'. The class this function is called from is 'LlamaTokenizer'.

In case it helps, the output of nvcc -V is nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Tue_May__3_18:49:52_PDT_2022 Cuda compilation tools, release 11.7, V11.7.64 Build cuda_11.7.r11.7/compiler.31294372_0

And going through my bash history it looks like I installed everything with git clone https://github.com/lxe/simple-llama-finetuner conda create -n llama-finetuner python=3.10 conda activate llama-finetuner conda install -y cuda -c nvidia/label/cuda-11.7.0 conda install -y pytorch=1.13.1 pytorch-cuda=11.7 -c pytorch cd simple-llama-finetuner/ pip install -r requirements.txt python main.py

[GreenTeaBD]

Also, quick question that doesn't need its own issue. What's the significance of the 2 empty spaces in between each entry in the training data?

I had finetuned GPT-Neo a bunch, and I'm trying to wrap my head around the differences. And I haven't been able to find out if finetuning LLaMA uses <|endoftext|> tokens or not, or if there's another way to do it. Is that what the two empty lines are adding?

[lxe]

Also, quick question that doesn't need its own issue. What's the significance of the 2 empty spaces in between each entry in the training data?

If you want to have newlines or empty lines in each of your samples, it helps. It's just the easiest way to format samples.

[lxe]

I just rewrote the whole thing. Still seeing the issue?