OSError: could not get source code when loading a model using a for loop

[daegonYu]

An error occurs when loading a model using a for loop as shown below. What could be the problem?

for peft_model_id in peft_model_ids:    
    print(peft_model_id)
    model, tokenizer = FastLanguageModel.from_pretrained(
        model_name = peft_model_id, 
        max_seq_length = 3096,
        dtype = None,
        load_in_4bit = True,
        # device_map='auto',
    )
    FastLanguageModel.for_inference(model)

{
    "name": "OSError",
    "message": "could not get source code",
    "stack": "---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[14], line 3
      1 for peft_model_id in peft_model_ids:    
      2     print(peft_model_id)
----> 3     model, tokenizer = FastLanguageModel.from_pretrained(
      4         model_name = peft_model_id, 
      5         max_seq_length = 3096,
      6         dtype = None,
      7         load_in_4bit = True,
      8         # device_map='auto',
      9     )
     10     FastLanguageModel.for_inference(model)

File ~/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/loader.py:332, in FastLanguageModel.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, token, device_map, rope_scaling, fix_tokenizer, trust_remote_code, use_gradient_checkpointing, resize_model_vocab, revision, *args, **kwargs)
    329     tokenizer_name = None
    330 pass
--> 332 model, tokenizer = dispatch_model.from_pretrained(
    333     model_name        = model_name,
    334     max_seq_length    = max_seq_length,
    335     dtype             = dtype,
    336     load_in_4bit      = load_in_4bit,
    337     token             = token,
    338     device_map        = device_map,
    339     rope_scaling      = rope_scaling,
    340     fix_tokenizer     = fix_tokenizer,
    341     model_patcher     = dispatch_model,
    342     tokenizer_name    = tokenizer_name,
    343     trust_remote_code = trust_remote_code,
    344     revision          = revision if not is_peft else None,
    345     *args, **kwargs,
    346 )
    348 if resize_model_vocab is not None:
    349     model.resize_token_embeddings(resize_model_vocab)

File ~/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/llama.py:1790, in FastLlamaModel.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, token, device_map, rope_scaling, fix_tokenizer, model_patcher, tokenizer_name, trust_remote_code, **kwargs)
   1787 Trainer._inner_training_loop = _fast_inner_training_loop
   1789 # Fix gradient accumulation
-> 1790 patch_gradient_accumulation_fix(Trainer)
   1792 # Save tokenizer for inference purposes
   1793 tokenizer.padding_side = \"left\" # Force inference

File ~/anaconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/_utils.py:1220, in patch_gradient_accumulation_fix(Trainer)
   1217 # Also fix up loss scaling ie negate loss *= self.args.gradient_accumulation_steps
   1218 if \"num_items_in_batch\" not in inspect.signature(Trainer.training_step).parameters: return
-> 1220 function = inspect.getsource(Trainer.training_step)
   1221 where = function.find(\"def\")
   1222 function = function.split(\"\
\")

File ~/anaconda3/envs/unsloth_env/lib/python3.11/inspect.py:1258, in getsource(object)
   1252 def getsource(object):
   1253     \"\"\"Return the text of the source code for an object.
   1254 
   1255     The argument may be a module, class, method, function, traceback, frame,
   1256     or code object.  The source code is returned as a single string.  An
   1257     OSError is raised if the source code cannot be retrieved.\"\"\"
-> 1258     lines, lnum = getsourcelines(object)
   1259     return ''.join(lines)

File ~/anaconda3/envs/unsloth_env/lib/python3.11/inspect.py:1240, in getsourcelines(object)
   1232 \"\"\"Return a list of source lines and starting line number for an object.
   1233 
   1234 The argument may be a module, class, method, function, traceback, frame,
   (...)
   1237 original source file the first line of code was found.  An OSError is
   1238 raised if the source code cannot be retrieved.\"\"\"
   1239 object = unwrap(object)
-> 1240 lines, lnum = findsource(object)
   1242 if istraceback(object):
   1243     object = object.tb_frame

File ~/anaconda3/envs/unsloth_env/lib/python3.11/inspect.py:1077, in findsource(object)
   1075     lines = linecache.getlines(file)
   1076 if not lines:
-> 1077     raise OSError('could not get source code')
   1079 if ismodule(object):
   1080     return lines, 0

OSError: could not get source code"
}

[danielhanchen]

Yep working on a fix - apologies on the issue!

[GabrielPimenta1999]

How can I install the version prior to the start of this bug?

[daegonYu]

@GabrielPimenta1999 Sorry for the late reply

Package Version

---

accelerate 1.0.1 aiohappyeyeballs 2.4.3 aiohttp 3.10.10 aiosignal 1.3.1 asttokens 2.4.1 attrs 24.2.0 bitsandbytes 0.44.1 certifi 2024.8.30 charset-normalizer 3.4.0 comm 0.2.2 datasets 3.0.2 debugpy 1.8.7 decorator 5.1.1 dill 0.3.8 docstring_parser 0.16 exceptiongroup 1.2.2 executing 2.1.0 filelock 3.13.1 frozenlist 1.5.0 fsspec 2024.9.0 gmpy2 2.1.2 hf_transfer 0.1.8 huggingface-hub 0.26.2 idna 3.10 importlib_metadata 8.5.0 ipykernel 6.29.5 ipython 8.29.0 ipywidgets 8.1.5 jedi 0.19.1 Jinja2 3.1.4 jupyter_client 8.6.3 jupyter_core 5.7.2 jupyterlab_widgets 3.0.13 markdown-it-py 3.0.0 MarkupSafe 2.1.3 matplotlib-inline 0.1.7 mdurl 0.1.2 mpmath 1.3.0 multidict 6.1.0 multiprocess 0.70.16 nest_asyncio 1.6.0 networkx 3.3 ninja 1.11.1.1 numpy 2.1.2 packaging 24.1 pandas 2.2.3 parso 0.8.4 peft 0.13.2 pexpect 4.9.0 pickleshare 0.7.5 pillow 10.2.0 pip 24.2 platformdirs 4.3.6 prompt_toolkit 3.0.48 propcache 0.2.0 protobuf 3.20.3 psutil 6.1.0 ptyprocess 0.7.0 pure_eval 0.2.3 pyarrow 18.0.0 Pygments 2.18.0 python-dateutil 2.9.0 pytz 2024.2 PyYAML 6.0.2 pyzmq 26.2.0 regex 2024.9.11 requests 2.32.3 rich 13.9.3 safetensors 0.4.5 sentencepiece 0.2.0 setuptools 75.1.0 shtab 1.7.1 six 1.16.0 stack-data 0.6.2 sympy 1.13.1 tokenizers 0.20.1 torch 2.5.1 torchaudio 2.5.1+cu121 torchvision 0.20.1+cu121 tornado 6.4.1 tqdm 4.66.6 traitlets 5.14.3 transformers 4.46.1 triton 3.1.0 trl 0.11.4 typing_extensions 4.11.0 tyro 0.8.14 tzdata 2024.2 unsloth 2024.10.7 unsloth_zoo 2024.10.5 urllib3 2.2.3 wcwidth 0.2.13 wheel 0.44.0 widgetsnbextension 4.0.13 xformers 0.0.28.post3 xxhash 3.5.0 yarl 1.17.1 zipp 3.20.2

[lochana]

Any update on this?? Thanks.