search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
5.14k stars 353 forks source link

CPUs clocking way down while koboldcpp is generating tokens. #318

Open CallMeAl1973 opened 1 year ago

CallMeAl1973 commented 1 year ago

Hi all, I just noticed something interesting. For some reason, when koboldcpp is generating tokens, the clock speeds of my CPUs are actually slowing way down. This is a Lenovo P360 Tiny. The computer has a 12th gen i9, an NVME SSD, 64 gigs of ram and is running Debian 12. I used the cpufreq-info utility twice to gather the below info. The first snapshot was with Koboldcpp running but not generating tokens. The second snapshot was during the token generation process. Is this behavior expected? If not, how can I help to gather more useful info? Thanks,

Before

cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: intel_pstate CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 1: driver: intel_pstate CPUs which run at the same hardware frequency: 1 CPUs which need to have their frequency coordinated by software: 1 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 2: driver: intel_pstate CPUs which run at the same hardware frequency: 2 CPUs which need to have their frequency coordinated by software: 2 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 3: driver: intel_pstate CPUs which run at the same hardware frequency: 3 CPUs which need to have their frequency coordinated by software: 3 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 4: driver: intel_pstate CPUs which run at the same hardware frequency: 4 CPUs which need to have their frequency coordinated by software: 4 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 5: driver: intel_pstate CPUs which run at the same hardware frequency: 5 CPUs which need to have their frequency coordinated by software: 5 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 6: driver: intel_pstate CPUs which run at the same hardware frequency: 6 CPUs which need to have their frequency coordinated by software: 6 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.29 GHz. analyzing CPU 7: driver: intel_pstate CPUs which run at the same hardware frequency: 7 CPUs which need to have their frequency coordinated by software: 7 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 800 MHz. analyzing CPU 8: driver: intel_pstate CPUs which run at the same hardware frequency: 8 CPUs which need to have their frequency coordinated by software: 8 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.90 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.90 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 9: driver: intel_pstate CPUs which run at the same hardware frequency: 9 CPUs which need to have their frequency coordinated by software: 9 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.90 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.90 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 10: driver: intel_pstate CPUs which run at the same hardware frequency: 10 CPUs which need to have their frequency coordinated by software: 10 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.90 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.90 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 11: driver: intel_pstate CPUs which run at the same hardware frequency: 11 CPUs which need to have their frequency coordinated by software: 11 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.90 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.90 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 12: driver: intel_pstate CPUs which run at the same hardware frequency: 12 CPUs which need to have their frequency coordinated by software: 12 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 13: driver: intel_pstate CPUs which run at the same hardware frequency: 13 CPUs which need to have their frequency coordinated by software: 13 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 14: driver: intel_pstate CPUs which run at the same hardware frequency: 14 CPUs which need to have their frequency coordinated by software: 14 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 15: driver: intel_pstate CPUs which run at the same hardware frequency: 15 CPUs which need to have their frequency coordinated by software: 15 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 799 MHz. analyzing CPU 16: driver: intel_pstate CPUs which run at the same hardware frequency: 16 CPUs which need to have their frequency coordinated by software: 16 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 17: driver: intel_pstate CPUs which run at the same hardware frequency: 17 CPUs which need to have their frequency coordinated by software: 17 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 18: driver: intel_pstate CPUs which run at the same hardware frequency: 18 CPUs which need to have their frequency coordinated by software: 18 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 19: driver: intel_pstate CPUs which run at the same hardware frequency: 19 CPUs which need to have their frequency coordinated by software: 19 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 20: driver: intel_pstate CPUs which run at the same hardware frequency: 20 CPUs which need to have their frequency coordinated by software: 20 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 21: driver: intel_pstate CPUs which run at the same hardware frequency: 21 CPUs which need to have their frequency coordinated by software: 21 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 22: driver: intel_pstate CPUs which run at the same hardware frequency: 22 CPUs which need to have their frequency coordinated by software: 22 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 23: driver: intel_pstate CPUs which run at the same hardware frequency: 23 CPUs which need to have their frequency coordinated by software: 23 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz.

While

cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: intel_pstate CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 1: driver: intel_pstate CPUs which run at the same hardware frequency: 1 CPUs which need to have their frequency coordinated by software: 1 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 2: driver: intel_pstate CPUs which run at the same hardware frequency: 2 CPUs which need to have their frequency coordinated by software: 2 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 3: driver: intel_pstate CPUs which run at the same hardware frequency: 3 CPUs which need to have their frequency coordinated by software: 3 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 471 MHz. analyzing CPU 4: driver: intel_pstate CPUs which run at the same hardware frequency: 4 CPUs which need to have their frequency coordinated by software: 4 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 5: driver: intel_pstate CPUs which run at the same hardware frequency: 5 CPUs which need to have their frequency coordinated by software: 5 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 6: driver: intel_pstate CPUs which run at the same hardware frequency: 6 CPUs which need to have their frequency coordinated by software: 6 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 7: driver: intel_pstate CPUs which run at the same hardware frequency: 7 CPUs which need to have their frequency coordinated by software: 7 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 8: driver: intel_pstate CPUs which run at the same hardware frequency: 8 CPUs which need to have their frequency coordinated by software: 8 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.90 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.90 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 9: driver: intel_pstate CPUs which run at the same hardware frequency: 9 CPUs which need to have their frequency coordinated by software: 9 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.90 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.90 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 10: driver: intel_pstate CPUs which run at the same hardware frequency: 10 CPUs which need to have their frequency coordinated by software: 10 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.90 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.90 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 11: driver: intel_pstate CPUs which run at the same hardware frequency: 11 CPUs which need to have their frequency coordinated by software: 11 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.90 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.90 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 1.40 GHz. analyzing CPU 12: driver: intel_pstate CPUs which run at the same hardware frequency: 12 CPUs which need to have their frequency coordinated by software: 12 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 13: driver: intel_pstate CPUs which run at the same hardware frequency: 13 CPUs which need to have their frequency coordinated by software: 13 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 14: driver: intel_pstate CPUs which run at the same hardware frequency: 14 CPUs which need to have their frequency coordinated by software: 14 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 15: driver: intel_pstate CPUs which run at the same hardware frequency: 15 CPUs which need to have their frequency coordinated by software: 15 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 480 MHz. analyzing CPU 16: driver: intel_pstate CPUs which run at the same hardware frequency: 16 CPUs which need to have their frequency coordinated by software: 16 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 17: driver: intel_pstate CPUs which run at the same hardware frequency: 17 CPUs which need to have their frequency coordinated by software: 17 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 18: driver: intel_pstate CPUs which run at the same hardware frequency: 18 CPUs which need to have their frequency coordinated by software: 18 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 19: driver: intel_pstate CPUs which run at the same hardware frequency: 19 CPUs which need to have their frequency coordinated by software: 19 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 20: driver: intel_pstate CPUs which run at the same hardware frequency: 20 CPUs which need to have their frequency coordinated by software: 20 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 21: driver: intel_pstate CPUs which run at the same hardware frequency: 21 CPUs which need to have their frequency coordinated by software: 21 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 22: driver: intel_pstate CPUs which run at the same hardware frequency: 22 CPUs which need to have their frequency coordinated by software: 22 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz. analyzing CPU 23: driver: intel_pstate CPUs which run at the same hardware frequency: 23 CPUs which need to have their frequency coordinated by software: 23 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 3.60 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 400 MHz.

Prerequisites

Please answer the following questions for yourself before submitting an issue.

Expected Behavior

Please provide a detailed written description of what you were trying to do, and what you expected llama.cpp to do.

Current Behavior

Please provide a detailed written description of what llama.cpp did, instead.

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

$ lscpu

$ uname -a

$ python3 --version
$ make --version
$ g++ --version

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

  1. step 1
  2. step 2
  3. step 3
  4. etc.

Failure Logs

Please include any relevant log snippets or files. If it works under one configuration but not under another, please provide logs for both configurations and their corresponding outputs so it is easy to see where behavior changes.

Also, please try to avoid using screenshots if at all possible. Instead, copy/paste the console output and use Github's markdown to cleanly format your logs for easy readability.

Example environment info:

llama.cpp$ git log | head -1
commit 2af23d30434a677c6416812eea52ccc0af65119c

llama.cpp$ lscpu | egrep "AMD|Flags"
Vendor ID:                       AuthenticAMD
Model name:                      AMD Ryzen Threadripper 1950X 16-Core Processor
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca sme sev
Virtualization:                  AMD-V

llama.cpp$ python3 --version
Python 3.10.9

llama.cpp$ pip list | egrep "torch|numpy|sentencepiece"
numpy                         1.24.2
numpydoc                      1.5.0
sentencepiece                 0.1.97
torch                         1.13.1
torchvision                   0.14.1

llama.cpp$ make --version | head -1
GNU Make 4.3

$ md5sum ./models/65B/ggml-model-q4_0.bin
dbdd682cce80e2d6e93cefc7449df487  ./models/65B/ggml-model-q4_0.bin

Example run with the Linux command perf

llama.cpp$ perf stat ./main -m ./models/65B/ggml-model-q4_0.bin -t 16 -n 1024 -p "Please close your issue when it has been answered."
main: seed = 1679149377
llama_model_load: loading model from './models/65B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 8192
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 64
llama_model_load: n_layer = 80
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 22016
llama_model_load: n_parts = 8
llama_model_load: ggml ctx size = 41477.73 MB
llama_model_load: memory_size =  2560.00 MB, n_mem = 40960
llama_model_load: loading model part 1/8 from './models/65B/ggml-model-q4_0.bin'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 2/8 from './models/65B/ggml-model-q4_0.bin.1'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 3/8 from './models/65B/ggml-model-q4_0.bin.2'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 4/8 from './models/65B/ggml-model-q4_0.bin.3'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 5/8 from './models/65B/ggml-model-q4_0.bin.4'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 6/8 from './models/65B/ggml-model-q4_0.bin.5'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 7/8 from './models/65B/ggml-model-q4_0.bin.6'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723
llama_model_load: loading model part 8/8 from './models/65B/ggml-model-q4_0.bin.7'
llama_model_load: .......................................................................................... done
llama_model_load: model size =  4869.09 MB / num tensors = 723

system_info: n_threads = 16 / 32 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |

main: prompt: 'Please close your issue when it has been answered.'
main: number of tokens in prompt = 11
     1 -> ''
 12148 -> 'Please'
  3802 -> ' close'
   596 -> ' your'
  2228 -> ' issue'
   746 -> ' when'
   372 -> ' it'
   756 -> ' has'
  1063 -> ' been'
  7699 -> ' answered'
 29889 -> '.'

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000

Please close your issue when it has been answered.
@duncan-donut: I'm trying to figure out what kind of "support" you need for this script and why, exactly? Is there a question about how the code works that hasn't already been addressed in one or more comments below this ticket, or are we talking something else entirely like some sorta bugfixing job because your server setup is different from mine??
I can understand if your site needs to be running smoothly and you need help with a fix of sorts but there should really be nothing wrong here that the code itself could not handle. And given that I'm getting reports about how it works perfectly well on some other servers, what exactly are we talking? A detailed report will do wonders in helping us get this resolved for ya quickly so please take your time and describe the issue(s) you see as clearly & concisely as possible!!
@duncan-donut: I'm not sure if you have access to cPanel but you could try these instructions. It is worth a shot! Let me know how it goes (or what error message, exactly!) when/if ya give that code a go? [end of text]

main: mem per token = 71159620 bytes
main:     load time = 19309.95 ms
main:   sample time =   168.62 ms
main:  predict time = 223895.61 ms / 888.47 ms per token
main:    total time = 246406.42 ms

 Performance counter stats for './main -m ./models/65B/ggml-model-q4_0.bin -t 16 -n 1024 -p Please close your issue when it has been answered.':

        3636882.89 msec task-clock                #   14.677 CPUs utilized
             13509      context-switches          #    3.714 /sec
              2436      cpu-migrations            #    0.670 /sec
          10476679      page-faults               #    2.881 K/sec
    13133115082869      cycles                    #    3.611 GHz                      (16.77%)
       29314462753      stalled-cycles-frontend   #    0.22% frontend cycles idle     (16.76%)
    10294402631459      stalled-cycles-backend    #   78.39% backend cycles idle      (16.74%)
    23479217109614      instructions              #    1.79  insn per cycle
                                                  #    0.44  stalled cycles per insn  (16.76%)
     2353072268027      branches                  #  647.002 M/sec                    (16.77%)
        1998682780      branch-misses             #    0.08% of all branches          (16.76%)

     247.802177522 seconds time elapsed

    3618.573072000 seconds user
      18.491698000 seconds sys
gustrd commented 1 year ago

Probably it's the temperature throttle.

CallMeAl1973 commented 1 year ago

Any way to turn it off or adjust parameters? The fans aren’t even noticeably running. Will need to figure out how to get a temperature report from the Linux CLI.

gustrd commented 1 year ago

At Windows I use ThrottleStop to configure parameters like PROCHOT.

I don't know a Linux similar.

https://www.techpowerup.com/download/techpowerup-throttlestop/

LostRuins commented 1 year ago

You can also try adjusting the process priority to see if it makes a difference.

CallMeAl1973 commented 1 year ago

I’ve tried the –highpriority parameter and have also tried to renice the process to -20. Neither of those settings seem to have any significant effect. The CPUs clock down to 400 MHz either way. The best speed I’ve gotten was by actually disabling Speed Shift in the bios. Doing that locks my CPU clock speed at 1.4 GHz. That takes my performance on the airoboros-33b-gpt4-1.4.ggmlv3.q3_K_S.bin model from about 0.8T/s to 1.2T/s.

LostRuins commented 1 year ago

Another thing to try is messing with the thread counts. You can also check out the latest version which fixed a few earlier speed regressions.

CallMeAl1973 commented 1 year ago

So this is getting really strange. I’ve played with this some more and I’m actually seeing better performance when I use the powersave governor than when I use the performance governor. With the performance governor, I’m seeing about 0.8 T/s on 30b models, while with the powersave governor, I’m seeing about 1.3 t/s with the same models. Is there any way to roughly estimate how many t/s I should be expecting from my system? This is a Lenovo P360 tiny with an i9-12900T, 64 gigs ofDDR5,NVIDIA T1000, and an NVMe SSD.

LostRuins commented 1 year ago

I can only compare against my own machine. For 30b, I get about 900ms per token, I am offloading 10 layers to my RTX2060 and the rest is on Intel 9th gen i7 (cant remember exact model)