janekb04
commented
1 year ago I tried converting the LLAMA models to the CTranslate2 format, but I got stuck at creating a proper LLamaLoader:
@register_loader("LlamaConfig")
class LlamaLoader(ModelLoader):
...converters/transformers.py has some examples, but I don't know enough to be able to convert the Hugging Face implementation of Llama (Config, Tokenizer, Model, Model with something added to it) to the CTranslate2 format. As far as I understood the code, the converter is basically, a kind of reimplementation of the original model that will be run by CTranslate2's "execution engine". Supposedly, for someone who knows LLMs well, this could be relatively straightforward, as all the other conversion scripts look rather short. Also, having read the LLama paper, I saw that they don't really talk a lot about its internals. The entire description of LLama's fits in a few paragraphs while its initial implementation is 238 lines.
2.2 Architecture
Following recent work on large language models, our network is based on the transformer architecture (Vaswani et al., 2017). We leverage various improvements that were subsequently proposed, and used in different models such as PaLM. Here are the main difference with the original architecture, and where we were found the inspiration for this change (in bracket): Pre-normalization [GPT3]. To improve the training stability, we normalize the input of each transformer sub-layer, instead of normalizing the output. We use the RMSNorm normalizing function, introduced by Zhang and Sennrich (2019). SwiGLU activation function [PaLM]. We replace the ReLU non-linearity by the SwiGLU activation function, introduced by Shazeer (2020) to improve the performance. We use a dimension of 2/3 4d instead of 4d as in PaLM. Rotary Embeddings [GPTNeo]. We remove the absolute positional embeddings, and instead, add rotary positional embeddings (RoPE), introduced by Su et al. (2021), at each layer of the network. The details of the hyper-parameters for our different models are given in Table 2.
2.4 Efficient Implementation
We make several optimizations to improve the training speed of our models. First, we use an efficient implementation of the causal multi-head attention to reduce memory usage and runtime. This implementation, available in the
xformerslibrary, is inspired by Rabe and Staats (2021) and uses the backward from Dao et al. (2022). This is achieved by not storing the attention weights and not computing the key/query scores that are masked due to the causal nature of the language modeling task. To further improve training efficiency, we reduced the amount of activations that are recomputed during the backward pass with checkpointing. More precisely, we save the activations that are expensive to compute, such as the outputs of linear layers. This is achieved by manually implementing the backward function for the transformer layers, instead of relying on the PyTorch autograd. To fully benefit from this optimization, we need to reduce the memory usage of the model by using model and sequence parallelism, as described by Korthikanti et al. (2022). Moreover, we also overlap the computation of activations and the communication between GPUs over the network (due toall_reduceoperations) as much as possible. When training a 65B-parameter model, our code processes around 380 tokens/sec/GPU on 2048 A100 GPU with 80GB of RAM. This means that training over our dataset containing 1.4T tokens takes approximately 21 days.
Quote from LLaMA: Open and Efficient Foundation Language Models Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, Guillaume Lample
guillaumekln
av
hobodrifterdavid
Palmik
qZhang88
BrightXiaoHan
what about adding llama to the pre-trained models list guys?