jorgeantonio21
commented
7 months ago This is running through another internal library, the repo is not yet publicly available, but I can share more details. That said, the Llama integration logic basically follows the candle example. This is the code for fetching and loading llama models
fn fetch(
api_key: String,
cache_dir: PathBuf,
config: ModelConfig,
) -> Result<Self::LoadData, ModelError> {
let device = device(config.device_id())?;
let dtype = DType::from_str(&config.dtype())?;
let api = ApiBuilder::new()
.with_progress(true)
.with_token(Some(api_key))
.with_cache_dir(cache_dir)
.build()?;
let model_type = ModelType::from_str(&config.model_id())?;
let repo_id = model_type.repo().to_string();
let revision = model_type.default_revision().to_string();
let repo = api.repo(Repo::with_revision(
repo_id.clone(),
RepoType::Model,
revision,
));
let config_file_path = repo.get("config.json")?;
let tokenizer_file_path = repo.get("tokenizer.json")?;
let model_weights_file_paths = if &repo_id == "TinyLlama/TinyLlama-1.1B-Chat-v1.0" {
vec![repo.get("model.safetensors")?]
} else {
hub_load_safetensors(&repo, "model.safetensors.index.json")?
};
let mut file_paths = Vec::with_capacity(2 + model_weights_file_paths.len());
file_paths.extend(vec![config_file_path, tokenizer_file_path]);
file_paths.extend(model_weights_file_paths);
Ok(Self::LoadData {
device,
dtype,
file_paths,
model_type: ModelType::from_str(&config.model_id())?,
use_flash_attention: config.use_flash_attention(),
})
}
fn load(load_data: Self::LoadData) -> Result<Self, ModelError> {
info!("Loading Llama model ...");
let start = Instant::now();
let device = load_data.device;
let dtype = load_data.dtype;
let (model, tokenizer_filename, config) = {
let config_filename = load_data.file_paths[0].clone();
let config: LlamaConfig = serde_json::from_slice(&std::fs::read(config_filename)?)?;
let tokenizer_filename = load_data.file_paths[1].clone();
let config = config.into_config(load_data.use_flash_attention);
let vb = unsafe {
VarBuilder::from_mmaped_safetensors(&load_data.file_paths[2..], dtype, &device)?
};
(model::Llama::load(vb, &config)?, tokenizer_filename, config)
};
let tokenizer = Tokenizer::from_file(tokenizer_filename)?;
info!("Loaded Llama model in {:?}", start.elapsed());
Ok(Self {
device,
model,
model_type: load_data.model_type,
tokenizer,
config,
dtype,
})
}The load of tokenizer seems to be resulting in the above warnings..
EricLBuehler
LaurentMazare
oovm
vody-am
kczimm
julien-blanchon
donkey-donkey
BradyBonnette
super-fun-surf
sidharthrajaram
timwalls
okpatil4u
louis030195
zachcp
vrdn-23
This issue aims at keeping track of the models that would be interesting to get added to candle. Feel free to make a comment to mention a new model, or vote for a model already in the list.
Added recently: