Comparision with faster-whisper

[geekodour]

I did a very rough comparison of https://github.com/guillaumekln/faster-whisper and whisper.cpp, turns out faster-whisper is faster than whisper.cpp in CPU.

For eg. It takes faster-whisper 14seconds with the small.en, whereas with whisper.cpp it's 46seconds. What causes this slowness? Or I am not setting parameters correctly, I tried keeping the beam size and threads similar.

I have a suspicion that I am not doing the comparison correctly, it'll be awesome if someone more knowledgeable can tell why faster-whisper is faster on CPU

I think I am comparing int8(faster-whisper) to int4(https://huggingface.co/ggerganov/whisper.cpp) quantization here. But not sure how much of a difference should it make.

See comparison here: https://gist.github.com/geekodour/8734b3bf22b8ede61fb5bfc92ce68fe3

[bobqianic]

I've just submitted a pull request that aims to address a few of these issues. Currently, OpenBLAS isn't active on the Windows platform, even though the previously released binary file is named whisper-blas-bin-x64. When OpenBLAS is enabled, it boosts CPU inferencing speeds by a factor of 3-4. I ran some tests on my i7-12700H, using the -w 2 flag for matrix multiplication. I found that it achieves at least 50% of the theoretical maximum (OpenBLAS enabled).

[fire]

Does this mean we're at 30 seconds compared to faster-whispher's 14 seconds?

[guillaumekln]

For this example there are 2 main reasons explaining why faster-whisper is faster:

• On x64 it uses oneMKL (for Intel) or oneDNN (for AMD) which are typically much faster than OpenBLAS or hand-written kernels (see https://github.com/ggerganov/whisper.cpp/discussions/589#discussioncomment-5289714)
• When using a beam size > 1, it executes a batched beam search while whisper.cpp runs multiple decoder passes (see #1048)

Disclaimer: I'm the author of faster-whisper.

[vadi2]

Just throwing in there that faster-whisper is quicker than whisper.cpp on the GPU as well.

Using an RTX 4080 on Ubuntu 22.04, a 12min audio sample takes 3.4min to transcribe using whisper.cpp with a medium model while faster-whisper does it in 30s using the higher quality large-v2 model. medium model brings it down to 20s. It's seriously impressive.

[nchudleigh]

@guillaumekln has batched beam search still not been implemented?

[guillaumekln]

The related issue #1048 is still open so I don't think it is implemented yet.

[nchudleigh]

Going to take a crack at bringing over the implementation from llama.

Fair warning, I am not very experienced with C/C++. Will link the PR here once ready for review.

[bobqianic]

Could you run another test on the latest version of whisper.cpp? I'm curious to see how much we've improved since last month. You can find the latest version in PR #1243. Thanks! @geekodour

Please use OpenBLAS (64bit) eg. openblas64-dev , and use the following command for testing: ./main -bs 5 -bo 5 -t 8 -f steve2.wav -m models/ggml-small.en.bin

[bobqianic]

Going to take a crack at bringing over the implementation from llama.

Fair warning, I am not very experienced with C/C++. Will link the PR here once ready for review.

Any progress?

[bobqianic]

Could you run another test on the latest version of whisper.cpp? I'm curious to see how much we've improved since last month. You can find the latest version in PR #1243. Thanks! @geekodour

Please use OpenBLAS (64bit) eg. openblas64-dev , and use the following command for testing: ./main -bs 5 -bo 5 -t 8 -f steve2.wav -m models/ggml-small.en.bin

In terms of CPU performance, whisper.cpp* isn't lagging too far behind. To give you an idea, our latest tests were run on an i7-12700H, using 4 threads and a beam size of 5.

Please note:

1. We haven't added batch decoding yet
2. Parallel sampling is still on the to-do list
3. Assisted generation is also not available at this time

Small model on CPU

diffusion2023-07-03.wav 27m:49s	Implementation	Precision	Beam size	Time
whisper.cpp (1ee2707)	FP32	5	9m:45s
faster-whisper (ad388cd)	FP32	5	7m:30s

[ggerganov]

@bobqianic We should better have the batched decoding implemented before additional tests. Without it whisper.cpp will always be significantly slower

[bobqianic]

@bobqianic We should better have the batched decoding implemented before additional tests. Without it whisper.cpp will always be significantly slower

Agree.

[nchudleigh]

No progress on this yet from me. Will update here with draft PR when I have something.

[InflexCZE]

I did some measurements of my own that I'd like to share along with some observations and maybe perf recommendations for this great project.

Audio Context size 1500 (default)	#	Method	Compute type	Relative speedup	Mean time	Std Dev
#​1	WhisperCpp	-	1.0	4723.80	15.13
#​27	WhisperCpp	BLAS	1.72	2745.40	14.72
#​2	CT2InsideCpp	BLAS f32	2.28	2068.30	18.09
#​3	CT2InsideCpp	BLAS i8	1.59	2976.30	27.16
#​4	CT2InsideCpp	MKL f32	2.76	1712.00	24.04
#​5	CT2InsideCpp	MKL i8	2.14	2210.60	28.87
#​6	CT2	BLAS f32	2.68	1762.10	33.39
#​7	CT2	BLAS i8	1.78	2646.40	16.97
#​8	CT2	MKL f32	3.37	1403.10	26.42
#​9	CT2	MKL i8	2.52	1873.60	8.66
#​10	CT2 (No MEL)	BLAS f32	2.73	1730.90	12.55
#​11	CT2 (No MEL)	BLAS i8	1.81	2615.70	15.70
#​12	CT2 (No MEL)	MKL f32	3.43	1379.00	13.19
#​13	CT2 (No MEL)	MKL i8	2.54	1856.70	10.24

Audio Context size 512 (Seems to strike well accuracy vs performance #166)	#	Method	Compute type	Relative speedup	Mean time	Std Dev
#​14	WhisperCpp	-	1.0	1349.00	37.21
#​28	WhisperCpp	BLAS	1.76	766.60	12.42
#​15	CT2InsideCpp	BLAS f32	1.48	909.00	20.83
#​16	CT2InsideCpp	BLAS i8	1.29	1042.60	12.34
#​17	CT2InsideCpp	MKL f32	2.42	556.40	7.62
#​18	CT2InsideCpp	MKL i8	1.99	677.30	10.56
#​19	CT2	BLAS f32	1.71	791.10	11.68
#​20	CT2	BLAS i8	1.45	932.00	6.88
#​21	CT2	MKL f32	3.03	444.80	12.61
#​22	CT2	MKL i8	2.38	566.40	7.69
#​23	CT2 (No MEL)	Blas f32	1.76	767.90	4.25
#​24	CT2 (No MEL)	Blas i8	1.48	910.90	3.51
#​25	CT2 (No MEL)	MKL f32	3.15	427.80	7.35
#​26	CT2 (No MEL)	MKL i8	2.44	552.90	20.40

Legend: CT2 - https://github.com/OpenNMT/CTranslate2 CT2 (No MEL) - MEL was pre-computed before benchmark. Just for perf-comparison of MEL vs Encode-Decode stages WhisperCpp - Just downloaded and compiled as is, with /arch:AVX2, not sure what compute type is default CT2InsideCpp - WhisperCpp frontend, Encode-Decode stages replaced with callbacks to CT2 MKL - https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl.html BLAS - https://github.com/OpenMathLib/OpenBLAS

Obsevations: 1) Gj on finding this Audio Context size reduction trick. I don't care if it's documented feature of the model or not, it works and makes processing on embedded devices actually viable. Thank you!

2) In https://github.com/ggerganov/whisper.cpp/discussions/589#discussioncomment-5265713 the question was raised how MKL compares to OpenBLAS. In https://github.com/ggerganov/whisper.cpp/discussions/589#discussioncomment-5289714 guillaumekln observed ~70% better perf thanks to MKL. I second this, with a small caviat. In my measurements this was the case only for AC=512 (#​19 vs #​21 => 78%). For full AC (#​6 vs #​8) the improvement was only 25%. The difference likely comes from much smaller data used and prolly smaller model too. Still, very nice gains, no issues with installation on my side, so it could be very low hanging fruit for ggml improvement.

3) INT8 compute mode turned out to be perf regression across the board compared to FLOAT32. This was very interesting for me, I belive this is result of small data and model running on "sufficiently powerful" HW. What is normally memory-bound program, managed to comfortably fit into L3 at every computation step thus turning the program into compute-bound problem, so the memory footprint reduction had no effect and additional compute load on i8 coversions increased the times. Will definitely follow up on this when porting to RPi. Much less cache, could become memory-bound again on f32s. Let me know if you have different hypotesis on this.

4) The #​2 #​3 #​4 #​5 vs #​6 #​7 #​8 #​9 respectively and #​15 #​16 #​17 #​18 vs #​19 #​20 #​21 #​22 respectively show overhead of Whisper.cpp frontend vs pure CT2 model. This is particularly interesting as it shows that there is potential to optimize the Whisper.cpp by 12-25% just by reducing data shuffles, runtime allocations and runtime tensor graph preparations.

For reproduction I attach sources of my benchmark as well as patch to Whisper.cpp that was used to expose MEL and replace Encode-Decode steps.

Much code here :)

```c++ #define AUDIO_CONTEXT_SIZE 1500 #define COMPUTE_TYPE FLOAT32 #define BEAM_SIZE 5 std::string StringifyWhisperCpp(std::vector samples) { whisper_context_params cparams = whisper_context_default_params(); cparams.use_gpu = false; whisper_context* ctx = whisper_init_from_file_with_params("\\models\\ggml-base.bin", cparams); auto samples32 = Int16ToFP32(samples); auto wparams = whisper_full_default_params(WHISPER_SAMPLING_BEAM_SEARCH); wparams.n_threads = 1; wparams.audio_ctx = AUDIO_CONTEXT_SIZE; wparams.no_timestamps = true; wparams.print_special = false; wparams.token_timestamps = false; wparams.beam_search.beam_size = BEAM_SIZE; Benchmark(__func__, [&] { if (whisper_full(ctx, wparams, samples32.data(), samples32.size()) != 0) { throw std::exception("failed to process audio"); } return 0; }); std::string output; const int n_segments = whisper_full_n_segments(ctx); for (int i = 0; i < n_segments; ++i) { output += whisper_full_get_segment_text(ctx, i); } whisper_free(ctx); return output; } #ifdef CT2 std::string StringifyCT2(std::vector samples, bool precomputedMEL = false) { std::vector> prompts{ {50258, 50259, 50359, 50363} }; auto mels = WhisperCppMel(nullptr, samples); try { std::string path = "\\faster-whisper\\models\\base"; auto modelFile = ctranslate2::models::Model::load ( path, Device::CPU, 0, ComputeType::COMPUTE_TYPE ); auto model = ctranslate2::models::WhisperReplica::create_from_model(*modelFile); auto featuresSize = precomputedMEL ? (dim_t)mels.size() / 80 : AUDIO_CONTEXT_SIZE * 2; StorageView features { Shape{ {1, 80, featuresSize} }, DataType::FLOAT32, Device::CPU }; whisper_context* whisper_context = nullptr; if (precomputedMEL) { features.copy_from(mels.data(), mels.size(), Device::CPU, /*synchronous*/ true); } else { whisper_context = WhisperCppMelContext(); } auto result = Benchmark(__func__, [&] { if (precomputedMEL == false) { auto mel = WhisperCppMel(whisper_context, samples); features.copy_from(mel.data(), mel.size(), Device::CPU, /*synchronous*/ true); } return model->generate ( features, prompts, models::WhisperOptions { .beam_size = BEAM_SIZE, .patience = 1, .length_penalty = 1, .repetition_penalty = 1.01, .no_repeat_ngram_size = 0, .max_length = 448, .sampling_temperature = 1.0, .return_scores = false,//true, .return_no_speech_prob = false,//true, .max_initial_timestamp_index = 50, .suppress_blank = false,//true, .suppress_tokens = {-1}, } ); }); std::string output = ""; for (auto strings : result[0].sequences) for (auto string : strings) { output += string; } if(whisper_context != nullptr) { whisper_free(whisper_context); } return output; } catch (std::exception& e) { auto message = e.what(); printf("%s\n", message); } } std::string StringifyCT2WithCpp(std::vector samples) { std::string path = "\\faster-whisper\\models\\base"; auto modelFile = ctranslate2::models::Model::load ( path, Device::CPU, 0, ComputeType::COMPUTE_TYPE ); auto model = ctranslate2::models::WhisperReplica::create_from_model(*modelFile); struct CallbackContext { models::WhisperReplica* model; whisper_context* whisper; std::string stringOut; std::vector lastTokens; StorageView features; } callback_context { .model = model.get(), .features = { Shape {{1, 80, AUDIO_CONTEXT_SIZE * 2}}, DataType::FLOAT32, Device::CPU } }; whisper_context_params cparams = whisper_context_default_params(); cparams.use_gpu = false; cparams.custom_encoder.custom_encoder_context = &callback_context; cparams.custom_encoder.custom_encoder_callback = [](void* custom_encoder_context, ggml_tensor* mel_in, ggml_tensor* encoded_tokens_out) { auto& ctx = *(CallbackContext*) custom_encoder_context; std::vector> prompts{ {50258, 50259, 50359, 50363} }; assert(mel_in->type == GGML_TYPE_F32); ctx.features.copy_from((float*)mel_in->data, mel_in->ne[0] * mel_in->ne[1], Device::CPU, /*synchronous*/ true); auto results = ctx.model->generate ( ctx.features, prompts, models::WhisperOptions { .beam_size = BEAM_SIZE, .patience = 1, .length_penalty = 1, .repetition_penalty = 1.01, .no_repeat_ngram_size = 0, .max_length = 448, .sampling_temperature = 1.0, .return_scores = false,//true, .return_no_speech_prob = false,//true, .max_initial_timestamp_index = 50, .suppress_blank = false,//true, .suppress_tokens = {-1}, } ); auto& tokensOut = results[0].sequences_ids; //assert(encoded_tokens_out->type == GGML_TYPE_I32); //ggml_backend_tensor_set(encoded_tokens_out, tokensOut.data(), 0, sizeof(size_t) * tokensOut.size()); //for (auto tokens : tokensOut) //for (auto token : tokens) //{ // ctx.stringOut += whisper_token_to_str(ctx.whisper, token); //} ctx.lastTokens.clear(); for (auto tokens : results[0].sequences_ids) for (auto token : tokens) { ctx.lastTokens.push_back((whisper_token)token); } }; cparams.custom_encoder.custom_decoder_context = &callback_context; cparams.custom_encoder.custom_decoder_callback = [] ( void* custom_decoder_context, void* submit_tokens_context, void(*submit_tokens)(void* submit_tokens_context, whisper_token* tokens, size_t tokens_count) ) { auto& ctx = *(CallbackContext*) custom_decoder_context; submit_tokens(submit_tokens_context, ctx.lastTokens.data(), ctx.lastTokens.size()); }; whisper_context* ctx = whisper_init_from_file_with_params("\\models\\ggml-base.bin", cparams); callback_context.whisper = ctx; auto samples32 = Int16ToFP32(samples); auto wparams = whisper_full_default_params(WHISPER_SAMPLING_BEAM_SEARCH); wparams.n_threads = 1; wparams.audio_ctx = AUDIO_CONTEXT_SIZE; wparams.no_timestamps = true; wparams.print_special = false; wparams.token_timestamps = false; wparams.beam_search.beam_size = BEAM_SIZE; Benchmark(__func__, [&] { if (whisper_full(ctx, wparams, samples32.data(), samples32.size()) != 0) { throw std::exception("failed to process audio"); } return 0; }); std::string output; const int n_segments = whisper_full_n_segments(ctx); for (int i = 0; i < n_segments; ++i) { output += whisper_full_get_segment_text(ctx, i); } whisper_free(ctx); return output; } template auto Benchmark(const char* name, T&& callback) { printf("%s\n", name); using time = decltype(ggml_time_ms()); // Wamup run auto firstResult = callback(); std::vector runs; for (int i = 0; i < 10; ++i) { auto begin = ggml_time_ms(); callback(); auto elapsed = ggml_time_ms() - begin; runs.push_back(elapsed); } auto sum = std::accumulate(runs.begin(), runs.end(), 0); auto mean = (float)sum / (float)runs.size(); printf("%.2f;", mean); float variance = 0; for (auto run : runs) { auto diff = run - mean; variance += diff * diff; } auto stdDev = sqrt(variance / (float)(runs.size() - 1)); printf("%.2f;", stdDev); for (auto run : runs) { printf("%i;", (int)run); } printf("\n"); return firstResult; } std::vector FilterMel(const std::vector& raw_mels, int n_ctx = AUDIO_CONTEXT_SIZE) { auto mel_offset = 0; auto mel_inp_n_mel = 80; auto mel_inp_n_len = (int)(raw_mels.size() / mel_inp_n_mel); const int i0 = std::min(mel_offset, mel_inp_n_len); const int i1 = std::min(mel_offset + 2 * n_ctx, mel_inp_n_len); std::vector mels; mels.resize(mel_inp_n_mel * (i1 - i0)); for (int j = 0; j < mel_inp_n_mel; ++j) { for (int i = i0; i < i1; ++i) { mels[j * 2 * n_ctx + (i - i0)] = raw_mels[j * mel_inp_n_len + i]; } } return mels; } whisper_context* WhisperCppMelContext() { whisper_context_params cparams = whisper_context_default_params(); cparams.use_gpu = false; return whisper_init_from_file_with_params("\\models\\ggml-base.bin", cparams); } std::vector WhisperCppMel(whisper_context* acceptedContext, std::vector samples, int contextSize = AUDIO_CONTEXT_SIZE) { whisper_context* ctx = acceptedContext; if (ctx == nullptr) { ctx = WhisperCppMelContext(); } auto samples32 = Int16ToFP32(samples); if (whisper_pcm_to_mel(ctx, samples32.data(), samples32.size(), /*n_threads*/1) != 0) { fprintf(stderr, "failed to process audio\n"); return {}; } auto* mel = whisper_extract_mel(ctx); auto mels = FilterMel(mel->data, contextSize); if (acceptedContext == nullptr) { whisper_free(ctx); } return mels; } ``` ```patch diff --git a/whisper.cpp b/whisper.cpp index f601197..d39275d 100644 --- a/whisper.cpp +++ b/whisper.cpp @@ -351,6 +351,7 @@ static const std::map> g_lang = { { "yue", { 99, "cantonese", } }, }; +/* struct whisper_mel { int n_len; int n_len_org; @@ -358,6 +359,7 @@ struct whisper_mel { std::vector data; }; +*/ struct whisper_filters { int32_t n_mel; @@ -815,6 +817,8 @@ struct whisper_state { whisper_openvino_context * ctx_openvino = nullptr; #endif + whisper_custom_encoder custom_encoder; + // [EXPERIMENTAL] token-level timestamps data int64_t t_beg = 0; int64_t t_last = 0; @@ -1625,7 +1629,7 @@ static bool whisper_encode_external(const whisper_state & wstate) { const bool use_openvino = wstate.ctx_openvino != nullptr; #endif - return use_coreml || use_openvino; + return use_coreml || use_openvino || wstate.custom_encoder.custom_encoder_callback != nullptr; } static struct ggml_cgraph * whisper_build_graph_conv( @@ -2059,6 +2063,13 @@ static bool whisper_encode_internal( whisper_coreml_encode(wstate.ctx_coreml, mel->ne[0], mel->ne[1], (float *) mel->data, (float *) wstate.embd_enc->data); #elif defined(WHISPER_USE_OPENVINO) whisper_openvino_encode(wstate.ctx_openvino, mel, wstate.embd_enc); +#else + wstate.custom_encoder.custom_encoder_callback + ( + wstate.custom_encoder.custom_encoder_context, + mel, + wstate.embd_enc + ); #endif } } @@ -2758,7 +2769,7 @@ static void log_mel_spectrogram_worker_thread(int ith, const std::vector // ref: https://github.com/openai/whisper/blob/main/whisper/audio.py#L110-L157 static bool log_mel_spectrogram( - whisper_state & wstate, + int64_t* time_out, const float * samples, const int n_samples, const int /*sample_rate*/, @@ -2837,7 +2848,10 @@ static bool log_mel_spectrogram( mel.data[i] = (mel.data[i] + 4.0)/4.0; } - wstate.t_mel_us += ggml_time_us() - t_start_us; + if(time_out != nullptr) + { + *time_out += ggml_time_us() - t_start_us; + } // Dump log_mel_spectrogram if (debug) { @@ -2853,6 +2867,37 @@ static bool log_mel_spectrogram( return true; } +static bool log_mel_spectrogram +( + whisper_state& wstate, + const float* samples, + const int n_samples, + const int sample_rate, + const int frame_size, + const int frame_step, + const int n_mel, + const int n_threads, + const whisper_filters& filters, + const bool debug, + whisper_mel& mel +) +{ + return log_mel_spectrogram + ( + &wstate.t_mel_us, + samples, + n_samples, + sample_rate, + frame_size, + frame_step, + n_mel, + n_threads, + filters, + debug, + mel + ); +} + // split text into tokens // // ref: https://github.com/openai/gpt-2/blob/a74da5d99abaaba920de8131d64da2862a8f213b/src/encoder.py#L53 @@ -2970,6 +3015,8 @@ struct whisper_state * whisper_init_state(whisper_context * ctx) { whisper_state * state = new whisper_state; + state->custom_encoder = ctx->params.custom_encoder; + state->backend = whisper_backend_init(ctx->params); if (!state->backend) { WHISPER_LOG_ERROR("%s: whisper_backend_init() failed\n", __func__); @@ -3052,13 +3099,20 @@ struct whisper_state * whisper_init_state(whisper_context * ctx) { } // encoder allocator - if (!whisper_encode_external(*state)) { - bool ok = whisper_allocr_graph_init(state->alloc_encode, ctx->backend, - [&]() { - return whisper_build_graph_encoder(*ctx, *state); - }); + if (!whisper_encode_external(*state)) + { + bool ok = whisper_allocr_graph_init + ( + state->alloc_encode, + ctx->backend, + [&]() + { + return whisper_build_graph_encoder(*ctx, *state); + } + ); - if (!ok) { + if (!ok) + { WHISPER_LOG_ERROR("%s: failed to init encoder allocator\n", __func__); whisper_free_state(state); return nullptr; @@ -3400,6 +3454,23 @@ int whisper_pcm_to_mel(struct whisper_context * ctx, const float * samples, int return whisper_pcm_to_mel_with_state(ctx, ctx->state, samples, n_samples, n_threads); } +int whisper_pcm_to_mel_no_state +( + const float* samples_in, + int n_samples, + float* mel_out, + size_t* mel_size_in_out, + int n_threads +) +{ + //if (!log_mel_spectrogram(*state, samples, n_samples, WHISPER_SAMPLE_RATE, WHISPER_N_FFT, WHISPER_HOP_LENGTH, ctx->model.filters.n_mel, n_threads, ctx->model.filters, false, state->mel)) { + WHISPER_LOG_ERROR("%s: failed to compute mel spectrogram\n", __func__); + return -1; +// } +// +// return 0; +} + // same as whisper_pcm_to_mel, but applies a Phase Vocoder to speed up the audio x2 (PV without phase lock is not good) int whisper_pcm_to_mel_phase_vocoder_with_state(struct whisper_context * ctx, struct whisper_state * state, const float * samples, int n_samples, int n_threads) { if (!log_mel_spectrogram(*state, samples, n_samples, WHISPER_SAMPLE_RATE, 2 * WHISPER_N_FFT, 2 * WHISPER_HOP_LENGTH, ctx->model.filters.n_mel, n_threads, ctx->model.filters, false, state->mel)) { @@ -3453,6 +3524,11 @@ int whisper_set_mel( return whisper_set_mel_with_state(ctx, ctx->state, data, n_len, n_mel); } +whisper_mel* whisper_extract_mel(struct whisper_context* ctx) +{ + return &ctx->state->mel; +} + int whisper_encode_with_state(struct whisper_context * ctx, struct whisper_state * state, int offset, int n_threads) { if (!whisper_encode_internal(*ctx, *state, offset, n_threads, nullptr, nullptr)) { WHISPER_LOG_ERROR("%s: failed to eval\n", __func__); @@ -5181,6 +5257,52 @@ int whisper_full_with_state( int best_decoder_id = 0; + //if (state->custom_encoder.custom_encoder_callback != nullptr) + //{ + // seek += 100 * WHISPER_CHUNK_SIZE; + // continue; + //} + + if (state->custom_encoder.custom_decoder_callback != nullptr) + { + auto& decoder = state->decoders[best_decoder_id]; + + // TAGS: WHISPER_DECODER_INIT + decoder.sequence.tokens.clear(); + decoder.sequence.result_len = 0; + decoder.sequence.sum_logprobs_all = 0.0; + decoder.sequence.sum_logprobs = -INFINITY; + decoder.sequence.avg_logprobs = -INFINITY; + decoder.sequence.entropy = 0.0; + decoder.sequence.score = -INFINITY; + + decoder.seek_delta = 100 * WHISPER_CHUNK_SIZE; + + decoder.failed = false; + decoder.completed = false; + decoder.has_ts = false; + + if (params.grammar_rules != nullptr) + { + decoder.grammar = whisper_grammar_init(params.grammar_rules, params.n_grammar_rules, params.i_start_rule); + } + else + { + decoder.grammar = {}; + } + + state->custom_encoder.custom_decoder_callback + ( + state->custom_encoder.custom_decoder_context, + &decoder, + [](void* context, whisper_token* tokens, size_t tokens_count) + { + auto& decoderTokens = ((whisper_decoder*)context)->sequence.tokens; + decoderTokens.insert(decoderTokens.begin(), tokens, tokens + tokens_count); + } + ); + } + else // TODO: Needs formatting, but I don't want to blow patch for (int it = 0; it < (int) temperatures.size(); ++it) { const float t_cur = temperatures[it]; @@ -5685,13 +5807,16 @@ int whisper_full_with_state( //WHISPER_LOG_DEBUG("prompt_init.size() = %d, prompt.size() = %d, result_len = %d, seek_delta = %d\n", prompt_init.size(), prompt.size(), result_len, seek_delta); // update prompt_past - prompt_past.clear(); - if (prompt.front() == whisper_token_prev(ctx)) { - prompt_past.insert(prompt_past.end(), prompt.begin() + 1, prompt.end() - prompt_init.size()); - } - - for (int i = 0; i < result_len; ++i) { - prompt_past.push_back(tokens_cur[i].id); + if(state->custom_encoder.custom_decoder_callback == nullptr) + { + prompt_past.clear(); + if (prompt.front() == whisper_token_prev(ctx)) { + prompt_past.insert(prompt_past.end(), prompt.begin() + 1, prompt.end() - prompt_init.size()); + } + + for (int i = 0; i < result_len; ++i) { + prompt_past.push_back(tokens_cur[i].id); + } } if (!tokens_cur.empty() && ctx->model.n_loaded > 0) { diff --git a/whisper.h b/whisper.h index a5371eb..708dbad 100644 --- a/whisper.h +++ b/whisper.h @@ -6,6 +6,7 @@ #include #include #include +#include #ifdef __GNUC__ # define WHISPER_DEPRECATED(func, hint) func __attribute__((deprecated(hint))) @@ -34,6 +35,8 @@ #define WHISPER_HOP_LENGTH 160 #define WHISPER_CHUNK_SIZE 30 +#define WHISPER_USE_CT2 + #ifdef __cplusplus extern "C" { #endif @@ -84,9 +87,30 @@ extern "C" { typedef int32_t whisper_token; typedef int32_t whisper_seq_id; + struct whisper_custom_encoder + { + void* custom_encoder_context; + void(*custom_encoder_callback) + ( + void* custom_encoder_context, + ggml_tensor* mel_in, + ggml_tensor* encoded_tokens_out + ); + + void* custom_decoder_context; + void(*custom_decoder_callback) + ( + void* custom_decoder_context, + void* submit_tokens_context, + void(*submit_tokens)(void* submit_tokens_context, whisper_token* tokens, size_t tokens_count) + ); + }; + struct whisper_context_params { bool use_gpu; int gpu_device; // CUDA device + + whisper_custom_encoder custom_encoder; }; typedef struct whisper_token_data { @@ -217,6 +241,15 @@ extern "C" { int n_samples, int n_threads); + WHISPER_API int whisper_pcm_to_mel_no_state + ( + const float* samples_in, + int n_samples, + float* mel_out, + size_t* mel_size_in_out, + int n_threads + ); + WHISPER_API int whisper_pcm_to_mel_with_state( struct whisper_context * ctx, struct whisper_state * state, @@ -257,6 +290,16 @@ extern "C" { int n_len, int n_mel); + struct whisper_mel { + int n_len; + int n_len_org; + int n_mel; + + std::vector data; + }; + + WHISPER_API whisper_mel* whisper_extract_mel(struct whisper_context* ctx); + // Run the Whisper encoder on the log mel spectrogram stored inside the default state in the provided whisper context. // Make sure to call whisper_pcm_to_mel() or whisper_set_mel() first. // offset can be used to specify the offset of the first frame in the spectrogram. ```

Disclaimers: 1) Runtimes measure only actual evaluation. Loading times of the models, teardown and first warmup run is not considered. Reasoning is simple, on short transcriptions we typically care about response time so the model will be preloaded, on large transcriptions the loading time amortizes to nothing.

2) My implementation of Audio Context size reduction is not correct for multi-pass decoding (+30s clips). Whisper.cpp nailed this down, I did not bother, it is out of my target use-case for now.

3) All measurements were done on Win10 i7-9700K MSVC Whisper Base model (multi-lang) and following ~7s audio voice command: https://github.com/Picovoice/rhino/blob/4a69bd13dceed859911f4e360dfde0cdb9d9fbf0/resources/audio_samples/test_within_context.wav

[bobqianic]

WhisperCpp - Just downloaded and compiled as is, with /arch:AVX2, not sure what compute type is default

That's FP32. I believe if you give the OpenBLAS version a try, you'll find its performance quite similar to CT2, with hardly any noticeable difference. (v1.5.4)

[InflexCZE]

Great, I completely missed that in docs, thanks. Should be BIG RED, if you want perf, enable this :)

So for the BLAS backend I added measurements #​27 and #​28 to the tables above. Solid ~1.7x perf improvement, on AC 512 it actually beats CT2. On 1500 there is something more to it tho, not even close.

In the ggml code I also noticed there is some support for MKL too, tried to measure it, but it's throwing on me, not sure why. Maybe you'll see what's going on? Maybe support for it is not finished yet?

Intel oneMKL ERROR: Parameter 9 was incorrect on entry to cblas_sgemm.

 ne1  = 1500
 ne01 = 512
 ne10 = 512
 ne00 = 512

 cblas_sgemm(CblasRowMajor, CblasNoTrans, CblasTrans,
          ne1, ne01, ne10,
         1.0f,    y, ne10,
                  x, ne00,
         0.0f,    d, ne01);

.exe!ggml_compute_forward_mul_mat(const ggml_compute_params * params, ggml_tensor * dst) Line 10629 C
.exe!ggml_compute_forward(ggml_compute_params * params, ggml_tensor * tensor) Line 16061    C
.exe!ggml_graph_compute_thread(void * data) Line 18157  C
.exe!ggml_graph_compute(ggml_cgraph * cgraph, ggml_cplan * cplan) Line 18490    C
.exe!ggml_backend_cpu_graph_compute(ggml_backend * backend, ggml_cgraph * cgraph) Line 809  C
.exe!ggml_backend_graph_compute_async(ggml_backend * backend, ggml_cgraph * cgraph) Line 282    C
.exe!ggml_backend_graph_compute(ggml_backend * backend, ggml_cgraph * cgraph) Line 276  C
.exe!ggml_graph_compute_helper(ggml_backend * backend, ggml_cgraph * graph, int n_threads) Line 190 C++
.exe!whisper_encode_internal(whisper_context & wctx, whisper_state & wstate, const int mel_offset, const int n_threads, bool(*)(void *) abort_callback, void * abort_callback_data) Line 2088   C++
.exe!whisper_full_with_state(whisper_context * ctx, whisper_state * state, whisper_full_params params, const float * samples, int n_samples) Line 5247  C++
.exe!whisper_full(whisper_context * ctx, whisper_full_params params, const float * samples, int n_samples) Line 5943    C++

Full context of locals

``` blck_0 -3689348814741910324 const __int64 blck_1 -3689348814741910324 const __int64 d 0x0000026db2e0e080 {-431602080.} float * desired_wsize 1048576 const unsigned __int64 dr0 -3689348814741910324 const __int64 dr1 -3689348814741910324 const __int64 dst 0x0000026dff7242d0 {type=GGML_TYPE_F32 (0) backend=GGML_BACKEND_TYPE_CPU (0) buffer=0x0000026dfb5ace80 {...} ...} ggml_tensor * from_float_to_vec_dot 0x00007ff62c9bf856 {.exe!ggml_fp32_to_fp16_row} void(*)(const float *, void *, int) i02 0 const __int64 i03 0 const __int64 i12 0 __int64 i13 0 __int64 ir010 -3689348814741910324 const __int64 ir011 -3689348814741910324 const __int64 ir110 -3689348814741910324 const __int64 ir111 -3689348814741910324 const __int64 ith 0 const int ith0 -3689348814741910324 const __int64 ith1 -3689348814741910324 const __int64 nb00 2 const unsigned __int64 nb0 4 const unsigned __int64 nb01 1024 const unsigned __int64 nb1 2048 const unsigned __int64 nb02 524288 const unsigned __int64 nb2 3072000 const unsigned __int64 nb03 524288 const unsigned __int64 nb3 3072000 const unsigned __int64 nb10 4 const unsigned __int64 nb11 2048 const unsigned __int64 nb12 3072000 const unsigned __int64 nb13 3072000 const unsigned __int64 ne_plane 262144 const __int64 ne00 512 const __int64 ne0 512 const __int64 ne01 512 const __int64 ne1 1500 const __int64 ne02 1 const __int64 ne2 1 const __int64 ne03 1 const __int64 ne3 1 const __int64 ne10 512 const __int64 ne11 1500 const __int64 ne12 1 const __int64 ne13 1 const __int64 nr0 -3689348814741910324 const __int64 nr1 -3689348814741910324 const __int64 nrc -3689348814741910324 __int64 nth 1 const int nth0 -3689348814741910324 const __int64 nth1 -3689348814741910324 const __int64 params 0x000000f7a3cf9b18 {type=GGML_TASK_TYPE_COMPUTE (1) ith=0 nth=1 ...} const ggml_compute_params * r2 1 const __int64 r3 1 const __int64 row_size 14757395258967641292 const unsigned __int64 src0 0x0000026dfcc79ca0 {type=GGML_TYPE_F16 (1) backend=GGML_BACKEND_TYPE_CPU (0) buffer=0x0000026de331ee70 {...} ...} const ggml_tensor * src1 0x0000026dff723c90 {type=GGML_TYPE_F32 (0) backend=GGML_BACKEND_TYPE_CPU (0) buffer=0x0000026dfb5ace80 {...} ...} const ggml_tensor * src1_col_stride 14757395258967641292 const unsigned __int64 src1_cont true const bool t0 0 __int64 tmp 0x000000f7a3cf8fe0 {-107374176., -107374176., -107374176., -107374176., -107374176., -107374176., -107374176., ...} float[32] type GGML_TYPE_F16 (1) ggml_type vec_dot 0x00007ff62d234110 {.exe!ggml_vec_dot_f16(int, float *, unsigned __int64, unsigned short *, unsigned __int64, unsigned short *, unsigned __int64, int)} void(*)(int, float *, unsigned __int64, const void *, unsigned __int64, const void *, unsigned __int64, int) vec_dot_num_rows 1 const __int64 vec_dot_type GGML_TYPE_F16 (1) ggml_type wdata 0xcccccccccccccccc const void * x 0x0000026dbdd91070 const void * y 0x0000026db2b20080 {-0.218084723} const float * ```

[AIWintermuteAI]

It looks like compiled with OpenBLAS is actually worse on Raspberry Pi 5 (1743.21 ms vs. 6232.27 ms - I ran jfk sample a few times, while the numbers differed slightly, the overall result was the same).