segmentation fault python3 airllm2.py

[taozhiyuai]

Mac, native conda, mlx installed.

`(native) taozhiyu@603e5f4a42f1 downloads % pip show mlx airllm Name: mlx Version: 0.11.1 Summary: A framework for machine learning on Apple silicon. Home-page: https://github.com/ml-explore/mlx Author: MLX Contributors Author-email: mlx@group.apple.com License: Location: /Users/taozhiyu/miniconda3/envs/native/lib/python3.12/site-packages Requires: Required-by:

Name: airllm Version: 2.8.3 Summary: AirLLM allows single 4GB GPU card to run 70B large language models without quantization, distillation or pruning. Home-page: https://github.com/lyogavin/Anima/tree/main/air_llm Author: Gavin Li Author-email: gavinli@animaai.cloud License: Location: /Users/taozhiyu/miniconda3/envs/native/lib/python3.12/site-packages Requires: accelerate, huggingface-hub, optimum, safetensors, scipy, torch, tqdm, transformers Required-by: (native) taozhiyu@603e5f4a42f1 downloads % python3 airllm2.py found index file... found_layers:{'model.embed_tokens.': False, 'model.layers.0.': False, 'model.layers.1.': False, 'model.layers.2.': False, 'model.layers.3.': False, 'model.layers.4.': False, 'model.layers.5.': False, 'model.layers.6.': False, 'model.layers.7.': False, 'model.layers.8.': False, 'model.layers.9.': False, 'model.layers.10.': False, 'model.layers.11.': False, 'model.layers.12.': False, 'model.layers.13.': False, 'model.layers.14.': False, 'model.layers.15.': False, 'model.layers.16.': False, 'model.layers.17.': False, 'model.layers.18.': False, 'model.layers.19.': False, 'model.layers.20.': False, 'model.layers.21.': False, 'model.layers.22.': False, 'model.layers.23.': False, 'model.layers.24.': False, 'model.layers.25.': False, 'model.layers.26.': False, 'model.layers.27.': False, 'model.layers.28.': False, 'model.layers.29.': False, 'model.layers.30.': False, 'model.layers.31.': False, 'model.layers.32.': False, 'model.layers.33.': False, 'model.layers.34.': False, 'model.layers.35.': False, 'model.layers.36.': False, 'model.layers.37.': False, 'model.layers.38.': False, 'model.layers.39.': False, 'model.layers.40.': False, 'model.layers.41.': False, 'model.layers.42.': False, 'model.layers.43.': False, 'model.layers.44.': False, 'model.layers.45.': False, 'model.layers.46.': False, 'model.layers.47.': False, 'model.layers.48.': False, 'model.layers.49.': False, 'model.layers.50.': False, 'model.layers.51.': False, 'model.layers.52.': False, 'model.layers.53.': False, 'model.layers.54.': False, 'model.layers.55.': False, 'model.layers.56.': False, 'model.layers.57.': False, 'model.layers.58.': False, 'model.layers.59.': False, 'model.layers.60.': False, 'model.layers.61.': False, 'model.layers.62.': False, 'model.layers.63.': False, 'model.layers.64.': False, 'model.layers.65.': False, 'model.layers.66.': False, 'model.layers.67.': False, 'model.layers.68.': False, 'model.layers.69.': False, 'model.layers.70.': False, 'model.layers.71.': False, 'model.layers.72.': False, 'model.layers.73.': False, 'model.layers.74.': False, 'model.layers.75.': False, 'model.layers.76.': False, 'model.layers.77.': False, 'model.layers.78.': False, 'model.layers.79.': False, 'model.norm.': False, 'lm_head.': False} some layer splits found, some are not, re-save all layers in case there's some corruptions. 0%| | 0/83 [00:00<?, ?it/s]Loading shard 1/30 zsh: segmentation fault python3 airllm2.py (native) taozhiyu@603e5f4a42f1 downloads % /Users/taozhiyu/miniconda3/envs/native/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' (native) taozhiyu@603e5f4a42f1 downloads %

`

`from airllm import AutoModel

MAX_LENGTH = 128

could use hugging face model repo id:

model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct")

or use model's local path...

model = AutoModel.from_pretrained("/Users/taozhiyu/Downloads/Meta-Llama-3-70B-Instruct")

input_text = [ 'What is the capital of United States?', ]

input_tokens = model.tokenizer(input_text, return_tensors="pt", return_attention_mask=False, truncation=True, max_length=MAX_LENGTH, padding=False)

generation_output = model.generate( input_tokens['input_ids'].cuda(), max_new_tokens=20, use_cache=True, return_dict_in_generate=True)

output = model.tokenizer.decode(generation_output.sequences[0])

print(output) `

[Proryanator]

What macbook are you using? An M3 Max? 🤔 I've seen an issue like this here as well as in the coreml stable diffusion repo specific to M3 macbooks.

[taozhiyuai]

What macbook are you using? An M3 Max? 🤔 I've seen an issue like this here as well as in the coreml stable diffusion repo specific to M3 macbooks.

m3 max 128gb

[Proryanator]

My suspicion yeah, very odd. I have a 36GB M3 Max.