NameError: name 'torch' is not defined

pseudotensor commented 1 year ago

System Info

transformers version: 4.32.0

Platform: Linux-5.19.0-38-generic-x86_64-with-glibc2.35
Python version: 3.10.9
Huggingface_hub version: 0.16.4
Safetensors version: 0.3.1
Accelerate version: 0.22.0
Accelerate config: - compute_environment: LOCAL_MACHINE
- distributed_type: MULTI_GPU
- mixed_precision: bf16
- use_cpu: False
- debug: False
- num_processes: 1
- machine_rank: 0
- num_machines: 1
- gpu_ids: 0
- rdzv_backend: static
- same_network: True
- main_training_function: main
- downcast_bf16: no
- tpu_use_cluster: False
- tpu_use_sudo: False
PyTorch version (GPU?): 2.0.1+cu117 (True)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?:
Using distributed or parallel set-up in script?:

Who can help?

@ArthurZucker and @younesbelkada

Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction

There is conditional in /home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/transformers/utils/bitsandbytes.py that in new transformers (not prior) leaves torch undefined if bitsandbytes can't be used. E.g. for CPU. Then one hits:

    model = model_loader(
  File "/home/jon/miniconda3/envs/alpaca/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 516, in from_pretrained
    return model_class.from_pretrained(
  File "/home/jon/miniconda3/envs/alpaca/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3091, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/home/jon/miniconda3/envs/alpaca/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3471, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/home/jon/miniconda3/envs/alpaca/lib/python3.10/site-packages/transformers/modeling_utils.py", line 744, in _load_state_dict_into_meta_model
    set_module_quantized_tensor_to_device(
  File "/home/jon/miniconda3/envs/alpaca/lib/python3.10/site-packages/transformers/utils/bitsandbytes.py", line 59, in set_module_quantized_tensor_to_device
    if old_value.device == torch.device("meta") and device not in ["meta", torch.device("meta")] and value is None:
NameError: name 'torch' is not defined

Repro:

Install bitsandbytes
export CUDA_VISIBLE_DEVICES=
python checkbits.py

from transformers import BitsAndBytesConfig

base_model = 'OpenAssistant/reward-model-deberta-v3-large-v2'
false = False
null = None
model_kwargs = {'local_files_only': False, 'resume_download': True,
                'use_auth_token': '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', 'trust_remote_code': True,
                'offload_folder': 'fooodasf3/offline_folder', 'revision': None, 'device_map': {'': 'cpu'},
                'quantization_config': BitsAndBytesConfig(**{
                    "bnb_4bit_compute_dtype": "bfloat16",
                    "bnb_4bit_quant_type": "fp4",
                    "bnb_4bit_use_double_quant": false,
                    "llm_int8_enable_fp32_cpu_offload": false,
                    "llm_int8_has_fp16_weight": false,
                    "llm_int8_skip_modules": null,
                    "llm_int8_threshold": 6.0,
                    "load_in_4bit": false,
                    "load_in_8bit": false,
                    "quant_method": "bitsandbytes"
                })
                }
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(base_model, **model_kwargs)

based upon code here: https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2#how-to-use

Expected behavior

No failure

ydshieh commented 1 year ago

Hi,

What's your bitsandbytes version? cc @younesbelkada .

torch is imported under is_bitsandbytes_available(), so might be a version issue.

if is_bitsandbytes_available():
    import bitsandbytes as bnb
    import torch

pseudotensor commented 1 year ago

(h2ogpt) jon@pseudotensor:~/h2ogpt$ pip freeze | grep bits
bitsandbytes==0.41.1

Latest on pypi, I don't think relevant.

younesbelkada commented 1 year ago

This is because we changed a bit the is_bitsandbytes_available() condition, https://github.com/huggingface/transformers/blob/main/src/transformers/utils/import_utils.py#L539 as you can see if no GPU is available things should behave as bitsandbytes is not installed. I also think users should be aware that bnb can't be used under a non-GPU env. EDIT: it is a bad idea to raise an error if no GPU is installed

younesbelkada commented 1 year ago

Let me dig a bit and get back to you

flavioloss commented 1 year ago

Restarting runtime and running bitsandbytes==0.40.2 worked for me

github-actions[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

guigarfr commented 3 months ago

Still happening to you guys?:

python -m transformations.integrations.bitsandbytes

I get the following:

python -m transformers.integrations.bitsandbytes
Traceback (most recent call last):
  File "/home/*****/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/*****/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/*****/.pyenv/versions/3.8.12/envs/mlrisktrain/lib/python3.8/site-packages/transformers/integrations/bitsandbytes.py", line 331, in <module>
    def dequantize_bnb_weight(weight: torch.nn.Parameter, state=None):
NameError: name 'torch' is not defined

younesbelkada commented 3 months ago

@guigarfr this should be now fixed on transformers main branch, can you try to install transformers from source?

guigarfr commented 3 months ago

It's strange, @younesbelkada . It only happened if the project had a test with freezetime library. Otherwise it would work with exactly the same packages installed. I mocked in a different way and the problem dissapeared. I just wanted to write this comment in the issue in case someone has the same problem with the mock of time.

huggingface / transformers