jvhoffbauer
commented
1 year ago I have the same issue - it occurs when running an 8bit model in the following docker container
FROM nvidia/cuda:11.7.0-cudnn8-devel-ubuntu22.04
RUN apt update
RUN apt install git -y
RUN apt install wget -y
RUN apt install python3 python3-pip -y
# Install dependencies (one-by-one for better caching)
#RUN pip install --upgrade pip
RUN pip install torch
RUN pip install transformers
RUN pip install datasets
RUN pip install evaluate
RUN pip install xformers
RUN pip install wandb
RUN pip install peft
RUN pip install trl
RUN pip install scipy
RUN pip install accelerate
RUN pip install scikit-learn
RUN pip install pandas
RUN pip install bleurt@https://github.com/google-research/bleurt/archive/b610120347ef22b494b6d69b4316e303f5932516.zip#egg=bleurt
RUN git clone https://github.com/EleutherAI/lm-evaluation-harness
RUN pip install -e lm-evaluation-harness
RUN git clone https://github.com/timdettmers/bitsandbytes.git
# CUDA_VERSIONS in {110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 120}
# make argument in {cuda110, cuda11x, cuda12x}
# if you do not know what CUDA you have, try looking at the output of: python -m bitsandbytes
ENV CUDA_VERSION=117
RUN cd bitsandbytes && git checkout ac5550a0238286377ee3f58a85aeba1c40493e17
RUN cd bitsandbytes && make cuda11x
RUN cd bitsandbytes && python3 setup.py install
#RUN pip install bitsandbytes
#RUN python3 check_bnb_install.py
# Init wandb
#COPY ./wandb /wandb
ENV WANDB_CONFIG_DIR=/wandb
ENV HF_DATASETS_CACHE="/hf_cache/datasets"
ENV HUGGINGFACE_HUB_CACHE="/hf_cache/hub"
# Copy the code
COPY . /code
# Set the working directory
WORKDIR /code
# Install a useful helper to check bitsandbytes installation. Only works at runtime.
RUN wget https://gist.githubusercontent.com/TimDettmers/1f5188c6ee6ed69d211b7fe4e381e713/raw/4d17c3d09ccdb57e9ab7eca0171f2ace6e4d2858/check_bnb_install.py 
sumukshashidhar
pribadihcr
TimDettmers
Ar770
piperino11
basteran
shashank140195
mikecipolla
swumagic
github-actions[bot]
PyroGenesis
adrian-branescu
monk1337
0-hero
hayoung-jeremy
ionutmodo
ZhouFang-Intel
FoolPlayer
khayamgondal
suzewei
zhuconv
matthewdouglas
RaccoonOnion
NuoJohnChen
surdarla
Boltzmachine
crinoiddream
zihaohe123
suhyeok-jang
sreemanti-abacusai
krjoha
rodaw92
yz26cn
Davido111200
Problem
Hello, I'm getting this weird cublasLt error on a lambdalabs H100 with cuda 118, pytorch 2.0.1, python3.10 Miniconda while trying to fine-tune a 3B param open-llama using LORA with 8bit loading. This only happens if we turn on 8bit loading. Lora alone or 4bit loading (qlora) works.
The same commands did work 2 weeks ago and stopped working a week ago.
I've tried bitsandbytes version 0.39.0 and 0.39.1 as prior versions don't work with H100. Source gives me a different issue as mentioned in Env section.
Expected
No error
Reproduce
Setup Miniconda then follow https://github.com/OpenAccess-AI-Collective/axolotl 's readme on lambdalabs and run the default open llama lora config.
Trace
0.39.0
Env
python -m bitsandbyteson main branch: I get error same as here https://github.com/TimDettmers/bitsandbytes/issues/382
on 0.39.0
Misc
All related issues:
Also tried install
cudatoolkitvia conda.