google / compare-codecs

Apache License 2.0
49 stars 23 forks source link

Consider encoding speed in comparison #53

Closed sijchen closed 9 years ago

sijchen commented 9 years ago

It will be much better, either if the file-size comparison can be put under a similar encoding speed, or the results can include speed data along with the filesize-PSNR data. The current default setting of x264 is '-preset slow', which can give very different filesize/PSNR results with '-preset ultrafast' or '-preset superfast' .

For most of the codec, the speed info can be read from the encoder output to screen, for example, x264 outputs "encoded xxx frames, xxx.xx fps, xxx.xx kb/s" at the end of encoding.

mstorsjo commented 9 years ago

Instead of just documenting the speed, it might be useful with a separate comparison table/setup for "very fast" encoder settings. Currently there's two tables, one for offline use (with no limitation on runtime) and one for interactive use, where it is ok as long as it runs faster than realtime (is there any specification on what system is used for testing this, or does it vary?).

For some usecases it might be useful to know how different codecs behave with settings aimed at encoding much faster than realtime (e.g. processing of large numbers of streams in parallel on a single machine, or for aiming at very weak machines), so a comparison at speeds roughly comparable to x264 veryfast/ultrafast would be nice as well. In such a comparison, the OpenH264 results would probably look slightly less bad, since it's mostly comparable to x264 ultrafast, and doesn't have any settings for slower encodes and better quality/smaller files, other than enabling loopfilter and cabac.

alvestrand commented 9 years ago

The table that considers encoding speed is the third table on the "results" page ("interactive results"). So far, I have not included openh264 here, because I wanted to get the simplest use case first.

But that use case also tries to encode at various rates, so the problems with rate control persist. Note: I don't trust anything printed by the codec itself if I can avoid it. Encode CPU time is measured by calling times() before and after the encode and looking at the "child CPU usage".

alvestrand commented 9 years ago

Correction: openh264 is already included in the third table. Example result (against x264 baseline):

http://compare-codecs.appspot.com/results/show_result.html?codec1=openh264&codec2=x264_base&criterion=rt

Typically, x264 encodes in between 6 and 9 seconds; openh264 encodes in approx 6 seconds. Closing issue, since we've already considered the encoding speed.

sijchen commented 9 years ago

Yes I saw the real-time table, thanks; as mentioned above, sometimes it may be useful to have encoding faster than real-time, which may mean saving computational resources to other processing.

How about listing the running time retrieved from times() onto the results page as a reference? just as the info of "Typically, x264 encodes in between 6 and 9 seconds; openh264 encodes in approx 6 seconds." as a side note?

alvestrand commented 9 years ago

The encode time is available in the mouse-over for each encoding. I think it's worthwhile having one page per codec describing the codec, but I'd like to keep opinion text at a minimum in the results page itself - it is very hard to auto-update text like this.

The openh264 actually performs at a speed comparable to x264_base; consider these encode times on a 10-second clip (Cactus):

openh264 (CPU time is 3rd column, following are parameters) d41d8cd98f00 -624.699000 11.38 4711dec1f2db 34.568000 11.8 -rc 2 b26cbc85d66f -624.699000 14.22 -rc 0 8769710044e3 -627.386000 14.77 -rc 1 b4cceb434567 -3997.671000 17.67 -rc -1

x264_base: 57b346db6a49 29.134960 8.87 --preset ultrafast --profile baseline --rc-lookahead 30 --tune psnr 377d87d68045 36.281970 11.5 --preset superfast --profile baseline --tune psnr cc80e52b2f1c 36.692960 19.86 --preset veryfast --profile baseline --rc-lookahead 30 --tune psnr

So on this clip, openh264's performance is somewhere between --veryfast and --superfast, but quite a bit slower than --ultrafast.

(to reproduce this display with your own compare-codecs install:

list_config_results --codec x264_base --component encode_cputime 10000 video/mpeg_video/Cactus_1920x1080_50.yuv | sort -n -k 3

Numbers may depend on which tests you've run.)

sijchen commented 9 years ago

Thanks for the info. It is good to have detailed speed info. Yes the speed number varies with different sequences.