Closed tobinsouth closed 9 months ago
It's worth noting that when running models we want to use efficient ones, and it seems like GPTNeoX models have fewer FLOPs per param than GPTNeo models. Plus, many of the best open-source LLMs build off the NeoX template.
Added info: The same error occurs for other LM architectures
e.g. LlamaForCausalLM
from transformers import AutoModelForCausalLM, AutoTokenizer
pretrained_model = AutoModelForCausalLM.from_pretrained('JackFram/llama-160m')
tokenizer = AutoTokenizer.from_pretrained("JackFram/llama-160m")
P.S. you might want to clear your cache if you download too many models: rm -r ~/.cache/huggingface/hub/*
tracked here: https://github.com/sonos/tract/issues/1269
Generate settings throws an error when using any Huggingface GPTNeoXForCausalLM model (interestingly it doesn't for GPTNeoForCausalLM models).
TL;DR:
Device and Operating System
ezkl 5.0.8 transformers 4.35 Ubuntu 22.04 40 core Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz 1005 GB Memory
Replication code