search

rustformers / llm

[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models
https://docs.rs/llm/latest/llm/
Apache License 2.0
6.08k stars 362 forks source link

Making results independent from threadcount/batch size (from llama.cpp) #67

Closed KerfuffleV2 closed 1 year ago

KerfuffleV2 commented 1 year ago

This may be something to keep an eye on: https://github.com/ggerganov/llama.cpp/pull/439

Looks like the corresponding code is here: https://github.com/rustformers/llama-rs/blob/bf7bdbcfff3114dcbdafb6eb7eed58f04f19b1c3/llama-rs/src/lib.rs#L1203

According to the comments in the pull, it should trade a small amount of performance for less memory usage. However, at least one user commented they saw more memory use (not sure what size model).

KerfuffleV2 commented 1 year ago

Seems like this is rearing its head again: https://github.com/ggerganov/llama.cpp/issues/603#issuecomment-1490399524

philpax commented 1 year ago

Interesting... not sure what to make of that. I suppose we could do more performance testing: I wonder if https://github.com/rustformers/llama-rs/issues/87#issuecomment-1489268539 is related?

KerfuffleV2 commented 1 year ago

I did some more testing with combinations of 6 and 12 threads, 16 and 32 bit memory and ID on or off. This was with a 7B Alpaca model, generating 80 tokens and just timing the total run time. It was pretty consistent though, and I did the ID (increased_determinism) one three times just to be sure.

Sorted by type:

ID Threads Memory TPS
Y 6 32 2.7860
Y 6 32 2.9353
Y 6 32 2.7716
N 6 32 3.7873
N 6 32 3.6932
Y 12 32 3.4560
Y 12 32 3.4309
Y 12 32 3.4439
N 12 32 3.8925
N 12 32 3.8376
Y 6 16 3.2821
Y 6 16 3.2599
Y 6 16 3.2978
N 6 16 3.6942
N 6 16 3.7541
Y 12 16 3.4277
Y 12 16 3.4325
Y 12 16 3.4361
N 12 16 3.8940
N 12 16 3.8127

Sorted by TPS (higher is better):

ID Threads Memory TPS
N 12 16 3.8940
N 12 32 3.8925
N 12 32 3.8376
N 12 16 3.8127
N 6 32 3.7873
N 6 16 3.7541
N 6 16 3.6942
N 6 32 3.6932
Y 12 32 3.4560
Y 12 32 3.4439
Y 12 16 3.4361
Y 12 16 3.4325
Y 12 32 3.4309
Y 12 16 3.4277
Y 6 16 3.2978
Y 6 16 3.2821
Y 6 16 3.2599
Y 6 32 2.9353
Y 6 32 2.7860
Y 6 32 2.7716

In the TPS sorted version, ID off comes out ahead pretty consistently. It's not a massive difference most of the time.

I don't know why ID + 6 threads + 32bit memory looks so bad but those results were consistent.

Anyway... It probably would make sense to go back to the behavior with the commandline argument that toggles it with it automatically getting enabled when a seed is specified. The difference is definitely enough to care about in the 32bit memory/6 thread case.

Not sure if it's just a thing with low numbers of threads in general or what.

KerfuffleV2 commented 1 year ago

Can anyone else replicate the performance differences I saw? If so, do we want to go back to the previous proposed behavior where there's a CLI flag which gets automatically enabled when --seed is supplied?

philpax commented 1 year ago

Sorry, haven't had the time to test. Seems like you've done some pretty thorough testing and other people have reported regressions in upstream, so I'm leaning towards your suggested solution of apply-only-when-seed-is-specified.

I'll reopen this issue for now...

KerfuffleV2 commented 1 year ago

I think this is the same thing, right?

https://github.com/ggerganov/llama.cpp/pull/775

I'm guessing people would want to use that approach rather than trying to invest time in making the existing on better.

edit: I've been testing the changes. llama.cpp seems much faster than a few days ago. Also it seemed like it would get much slower as the context increased, now it's still pretty fast even after generating a page of text.

Not 100% sure it's all this change, but something changed recently that made a big difference.

philpax commented 1 year ago

Yes, that seems related. Looks like we'll need to sync with them soon...