search

huggingface / open_asr_leaderboard

Apache License 2.0
50 stars 20 forks source link

RTF on CPU #11

Open csukuangfj opened 1 year ago

csukuangfj commented 1 year ago

Is there a plan to also list the RTF on CPU?

nithinraok commented 9 months ago

Very good point, @Vaibhavs10

csukuangfj commented 1 month ago

Is there any progress?

Embedded systems, mobile systems, etc, usually have no GPUs.

It would be great if RTFs on CPU are also included.

galv commented 1 month ago

I think that if you want to see it happen, you should do it. I'm definitely curious, but the problem is that you don't just benchmark a particular system without taking the time to understand the system and making sure that you are running it correctly.

I don't think RTFx is an appropriate metric for embedded systems or mobile systems. Since you are running at batch size 1, you probably care much more about latency. Since it's a mobile system, you can assume that RTFx must be at least 1 anyway. I assume this purpose is some kind of interactive system.