Limited GPU Utilization with NVIDIA RTX 4000 Ada Gen

[James-Shared-Studios]

I am experiencing limited GPU utilization with the NVIDIA RTX 4000 Ada Gen card while running on Windows 10 1809 CPU: AMD EPYC 3251 8-Core Processor 2.5 GHz RAM: 32GB GPU: NVIDIA RTX 4000 Ada Gen 20 GB CUDA Toolkit Version: 12.3 GPU Driver Version: 546.12

Python code:

    device = 'cuda'
    compute_type = 'int8_float16'
    model_size = 'medium.en'

    print(f"Loading model...")

    start_time = time.time()
    model = WhisperModel(model_size, device=device, 
                         compute_type=compute_type)
    end_time = time.time()
    execution_time = end_time - start_time
    print(f"Model loading time: {execution_time:.2f} seconds")
    folder_path = r"C:\Users\XYZ\Downloads\AI voice"
    max_new_tokens = 10
    beam_size = 10

    for filename in os.listdir(folder_path):
        if filename.endswith(".mp3") or filename.endswith(".m4a") or filename.endswith(".mp4") or filename.endswith(".wav"):
            file_path = os.path.join(folder_path, filename)
            print(f"Transcribing file: {file_path}")
            start_time = time.time()
            segments, _ = model.transcribe(file_path,
                                           beam_size=beam_size,
                                           max_new_tokens=max_new_tokens,
                                           word_timestamps = False,
                                           prepend_punctuations = "",
                                           append_punctuations = "",
                                           language="en", condition_on_previous_text=False)
            for segment in segments:
                print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
            end_time = time.time()
            execution_time = end_time - start_time
            print(f"Execution time: {execution_time:.2f} seconds")
            total_processing_time += execution_time

While running my code, I'm only observing around 10% GPU utilization.

However, the same code achieves 100% utilization on an NVIDIA GeForce RTX 4070.

[Napuh]

Try to repeat the test but show the CUDA graph which shows CUDA utilization.

To do that, click here: And select CUDA

[James-Shared-Studios]

for CUDA it's barely reached 70% utilization

[Napuh]

How does it compare with a bigger model?

[phineas-pta]

u should compare speed

utilization matters less

[James-Shared-Studios]

u should compare speed

utilization matters less

The average processing time with GeForce 4070 is 0.16 seconds, compared to 0.51 seconds with RTX 4000 Ada. I would expect faster performance from RTX 4000 Ada, that's why I was wondering if the RTX 4000 Ada has been limited in some way.

[James-Shared-Studios]

How does it compare with a bigger model?

the same results for large-v1, large-v2 and large-v3

[phineas-pta]

I would expect faster performance from RTX 4000 Ada

no u should expect the inverse: 4070 is faster

[James-Shared-Studios]

I would expect faster performance from RTX 4000 Ada

no u should expect the inverse: 4070 is faster

why is that? could you provide more context please? Thank you.

[phineas-pta]

since the model can fit to gpu, vram is not a factor, it comes down to memory bandwidth (more impactful when cuda cores count isnt much different)

u can take a look at their theoretical fp32 & fp16 performance:

• https://www.techpowerup.com/gpu-specs/rtx-4000-ada-generation.c4171
• https://www.techpowerup.com/gpu-specs/geforce-rtx-4070.c3924

[James-Shared-Studios]

4070 FP16 (half) 29.15 TFLOPS vs RTX 4000 Ada FP16 (half) 26.73 TFLOPS (1:1) so RTX 4000 Ada should not be three times slower than 4070, correct?

[phineas-pta]

the execution time is too short, there's additionally i/o overhead

for better benchmark, use longer audio/video to reduce overhead time part

[James-Shared-Studios]

the execution time is too short, there's additionally i/o overhead

for better benchmark, use longer audio/video to reduce overhead time part

That makes sense. I will try a longer audio and see if it improves the results. Thank you so much for your help.

[Napuh]

What's the conclusion? @James-Shared-Studios