Closed mikowals closed 9 months ago
This both speeds up loading the models and reduced memory use. Prior to this I think we may have had 3 sets of the weights existing at a time. Memory usage on my M1 Pro now peaks ~8GB instead of ~12GB when running TinyLlama-1B.
Llamatune picks up the speedup in file loading. Tokens per second is unchanged. This is the output from the stories110M benchmark.
stories110M
This both speeds up loading the models and reduced memory use. Prior to this I think we may have had 3 sets of the weights existing at a time. Memory usage on my M1 Pro now peaks ~8GB instead of ~12GB when running TinyLlama-1B.