StarcoderV2 models not merging

rombodawg commented 6 months ago

When merging any starcoderV2 models this error always comes up "safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer"

Complete error bellow

D:\Merging_models_on_D_drive\mergekit5\mergekit>mergekit-yaml D:\Merging_models_on_D_drive\mergekit5\mergekit\examples\Sauronties.yml "E:\Open_source_ai_chatbot\OOBA_10\text-generation-webui-main\models\UniverseCoder-7b"
C:\Users\USER\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\bitsandbytes\cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
'NoneType' object has no attribute 'cadam32bit_grad_fp32'
Warmup loader cache: 100%|████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 300.00it/s]
  0%|                                                                                                           | 1/3098 [00:00<00:34, 90.90it/s]
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "D:\Merging_models_on_D_drive\mergekit5\mergekit\mergekit-yaml.exe\__main__.py", line 7, in <module>
  File "C:\Users\USER\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Merging_models_on_D_drive\mergekit5\mergekit\mergekit\options.py", line 76, in wrapper
    f(*args, **kwargs)
  File "D:\Merging_models_on_D_drive\mergekit5\mergekit\mergekit\scripts\run_yaml.py", line 47, in main
    run_merge(
  File "D:\Merging_models_on_D_drive\mergekit5\mergekit\mergekit\merge.py", line 90, in run_merge
    for _task, value in exec.run():
  File "D:\Merging_models_on_D_drive\mergekit5\mergekit\mergekit\graph.py", line 191, in run
    res = task.execute(**arguments)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Merging_models_on_D_drive\mergekit5\mergekit\mergekit\io\tasks.py", line 78, in execute
    x = loader.get_tensor(name, device=self.device or "cpu")
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Merging_models_on_D_drive\mergekit5\mergekit\mergekit\io\lazy_tensor_loader.py", line 135, in get_tensor
    self.current_shard = TensorLoader.get(
                         ^^^^^^^^^^^^^^^^^
  File "D:\Merging_models_on_D_drive\mergekit5\mergekit\mergekit\io\loader.py", line 45, in get
    return safetensors.safe_open(
           ^^^^^^^^^^^^^^^^^^^^^^
safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer

cg123 commented 6 months ago

This looks a lot like a partial or corrupted download of the model. Are you able to load it in other software from that path?

CaptainHer0 commented 6 months ago

Same thing here, Ive been pounding my head on this issue for over 12 hours.

Huggingface Login successful

C:\Users\USER\mergekit>mergekit-yaml F:\AIBaseMergeModels\ProfessorSensei\mergekit-config.yml F:\AIBaseMergeModels\ProfessorSensei\ProfessorSensei-v1.0 --cuda --write-model-card --lazy-unpickle --low-cpu-memory --out-shard-size 1B --safe-serialization Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Scripts\mergekit-yaml.exe__main.py", line 7, in File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1157, in call return self.main(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 783, in invoke return callback(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\USER\mergekit\mergekit\options.py", line 76, in wrapper f(args, **kwargs) File "C:\Users\USER\mergekit\mergekit\scripts\run_yaml.py", line 45, in main yaml.safe_load(config_source) File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\yaml__init__.py", line 125, in safe_load return load(stream, SafeLoader) ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\yaml__init__.py", line 81, in load return loader.get_single_data() ^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\yaml\constructor.py", line 49, in get_single_data node = self.get_single_node() ^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\yaml\composer.py", line 36, in get_single_node document = self.compose_document() ^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\yaml\composer.py", line 58, in compose_document self.get_event() File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\yaml\parser.py", line 118, in get_event self.current_event = self.state() ^^^^^^^^^^^^ File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\yaml\parser.py", line 193, in parse_document_end token = self.peek_token() ^^^^^^^^^^^^^^^^^ File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\yaml\scanner.py", line 129, in peek_token self.fetch_more_tokens() File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\yaml\scanner.py", line 223, in fetch_more_tokens return self.fetch_value() ^^^^^^^^^^^^^^^^^^ File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\yaml\scanner.py", line 577, in fetch_value raise ScannerError(None, None, yaml.scanner.ScannerError: mapping values are not allowed here in "", line 2, column 11: parameters:

rombodawg commented 6 months ago

@cg123 Yea sorry i forgot to mention, i got it working, i didnt finish my download, it got interrupted, i did end up merging the model just fine.

cg123 commented 6 months ago

@cg123 Yea sorry i forgot to mention, i got it working, i didnt finish my download, it got interrupted, i did end up merging the model just fine.

Glad you got it working!

@CaptainHer0 that looks like a syntax error in your YAML config file. Could you post the config you're trying to use?

CaptainHer0 commented 6 months ago

### [solved] My YAML file was not nested properly. I've only used Python for less than a week. Lesson learned.

Something:
  - something-else
    another-thing
  - something-else2
    another-thing2

@cg123

Windows 11 Hone - 23H2 (OS build: 22631.3296) Python 3.12.2 (I have tried 3.12.1 & 3.10) PIP v24.0

*My inspiration to use these 4 models came from theprofessor155B. I wanted to include the entire updated meditron-70b model and the new WizardMath-7b with bfloat16. Maybe the linear merge is the wrong type for this?

Steps: huggingface-cli login Login successful

git clone https://github.com/cg123/mergekit.git
cd mergekit
pip install -e .

C:\Users\USER\mergekit>mergekit-yaml F:\mergemodels\professorsensei\config.yaml F:\mergemodels\professorsensei\professorsensei-v1.0 --cuda --out-shard-size 1B --lazy-unpickle`

RESULTS IN ERRORS

"config.yaml" UTF-8

model_name = professorsensei-v1.0
merge_method linear
slices
sources
model F:\mergemodels\cognitivecomputations\dolphin-2.2-70b
layer_range [0, 1]
model F:\mergemodels\epfl-llm\meditron-70b
layer_range [0, 1]
parameters
weight 0
sources
model F:\mergemodels\cognitivecomputations\dolphin-2.2-70b
layer_range [1, 25]
sources
model F:\mergemodels\WizardLM\WizardMath-7B-V1.1
layer_range [1, 17]
sources
model F:\mergemodels\epfl-llm\meditron-70b
layer_range [1, 40]
sources
model F:\mergemodels\migtissera\Synthia-70B-v1.2b
layer_range [10, 25]
sources
model F:\mergemodels\cognitivecomputations\dolphin-2.2-70b
layer_range [25, 45]
sources
model F:\mergemodels\WizardLM\WizardMath-7B-V1.1
layer_range [17, 35]
sources
model F:\mergemodels\migtissera\Synthia-70B-v1.2b
layer_range [45, 65]
sources
model F:\mergemodels\epfl-llm\meditron-70b
layer_range [40, 80]
sources
model F:\mergemodels\cognitivecomputations\dolphin-2.2-70b
layer_range [45, 79]
sources
model F:\mergemodels\epfl-llm\meditron-70b
layer_range [79, 80]
model F:\mergemodels\cognitivecomputations\dolphin-2.2-70b
layer_range [79, 80]
parameters
weight 0
dtype bfloat16
tokenizer_source model:F:\mergemodels\cognitivecomputations\dolphin-2.2-70b

I have tried referencing the HF repositories as well.

ERROR:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Scripts\mergekit-yaml.exe\__main__.py", line 7, in <module>
  File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\mergekit\mergekit\options.py", line 76, in wrapper
    f(*args, **kwargs)
  File "C:\Users\USER\mergekit\mergekit\scripts\run_yaml.py", line 44, in main
    merge_config: MergeConfiguration = MergeConfiguration.model_validate(
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\pydantic\main.py", line 509, in model_validate
    return cls.__pydantic_validator__.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**pydantic_core._pydantic_core.ValidationError: 1 validation error for MergeConfiguration
  Input should be a valid dictionary or instance of MergeConfiguration [type=model_type, input_value='model_name = "professors...ations\\dolphin-2.2-70b', input_type=str]
    For further information visit https://errors.pydantic.dev/2.6/v/model_type**

I have all the model(s) files downloaded on my SSD (F:) Thank You! 👍

arcee-ai / mergekit

StarcoderV2 models not merging #192