search

kvcache-ai / ktransformers

A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations
Apache License 2.0
745 stars 39 forks source link

Attempting to increase output to 16k results in crash during output #104

Open bitbottrap opened 1 month ago

bitbottrap commented 1 month ago

Increased max_new_tokens to 16384 and received the following crash. Code is updated as of creation of this issue. The prompt was a bit long having an additional 250 lines of source code context.

Unfortunately, Between this and another reported issue (updated) I have not been able to get a useful number of response tokens for coding purposes out of KT.

Oct 24 11:37:11 sv-ai kernel: NVRM: Xid (PCI:0000:01:00): 43, pid=63681, name=pt_main_thread, Ch 0000000c Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [0,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [1,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [2,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [3,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [4,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [5,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [6,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [7,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [8,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [9,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [10,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [11,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [12,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [13,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [14,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [15,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [16,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [17,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [18,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [19,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [20,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [21,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [22,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [23,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [24,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [25,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [26,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [27,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [28,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [29,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [30,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [31,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [32,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [33,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [34,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [35,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [36,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [37,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [38,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [39,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [40,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [41,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [42,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [43,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [44,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [45,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [46,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [47,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [48,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [49,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [50,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [51,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [52,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [53,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [54,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [55,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [56,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [57,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [58,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [59,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [60,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [61,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [62,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai ktransformers[63681]: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [63,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed. Oct 24 11:37:11 sv-ai kernel: pt_main_thread[63681]: segfault at 0 ip 0000709ea8ace279 sp 00007fff53546ca0 error 4 in libcuda.so.560.35.03[709ea86b0000+4b6000] likely on CPU 113 (core 49, socket 0) Oct 24 11:37:11 sv-ai kernel: Code: 84 dc 00 00 00 83 3d 9d c4 c6 01 02 48 8b 0f 77 09 f6 05 a1 c4 c6 01 01 74 10 44 8b 05 94 c4 c6 01 45 85 c0 0f 84 97 00 00 00 <48> 8b 09 4c 8b 41 70 8b 4f 10 4c 01 c1 48 29 ca 8b 48 20 01 f2 48

qiyuxinlin commented 4 weeks ago

Could you please provide the yaml file and launch command? Based on the above error message, we cannot determine which operator the error occurred in.