Best settings for H265

[MarkGhanz]

Is there a better setting for h265? The default setting can easily overload the encoder and output a video of bitrate several hundred mbps if setting the bitrate to 0 (auto calculate). But H265(legacy) and the h265 option from obs-gstreamer plugin works fine. Can it be a regression of gstreamer?

Graphics Card: RX 7900 XT Mesa version: 23.0.0 gstreamer vaapi version: 1.22.0 OS: Arch Linux Kernel Version: 6.2.1 OBS Version: 29.0.2

[fzwoch]

There should have been a log with settings attached for such a report.

Defaults with neither version are tuned for OBS. You will need to know about these settings and set them to your use case. As they are two different encoders their settings may be quite different. 1080p60 seemed fine to me. Everything else you probably would have to report to GStreamer.

[fzwoch]

For my own tracking: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1991

Thanks for taking it upstream!

[Big-FG]

Ive been testing the new encoder a fair bit and here are the tips i can give

1. Use CBR as CQP can go pretty nuts on the bitrate at times while CBR is much better behaved. It seems to be a weird thing with this new encoder at the moment.
2. There is no real point in going over 3 ref frames as it just makes it more likely to overload the encoder with no visual benefit to my eyes
3. target usage 4-7 will net you the best speeds for encoding and depending on resolution /bitrate you really wont need to worry about quality reduction. The lower your resolution/bit rate the lower you set target but if you use higher resolution/bit rate you can increase the target for better speed.
4. if you insist on using CQP try to use sane QP settings so the encoder doesnt freak out as much. Min-QP 16-18 is the AMF encoders default so try following that and leave Max-QP alone. The other QP settings stick around 23-26 for H.265 as the return on investment for going much lower isnt high with H.265

[Atemu]

@Big-FG thanks for the investigation.

Note that the entire point of CQP is to have a variable bitrate but relatively constant quality given the content.
If you had a video with a few seconds of high movement and then a few seconds of a static image, those sections would require wildly different bitrates. With CBR, they'd get the same bitrate however; making the high-movement scene look pretty bad and the static image not much better.

CQP is for recordings where the exact bitrate at any given time doesn't matter much (but average bitrate does to a degree) and CBR for streaming where both current and average need to be relatively stable.

[Big-FG]

@Big-FG thanks for the investigation.

Note that the entire point of CQP is to have a variable bitrate but relatively constant quality given the content. If you had a video with a few seconds of high movement and then a few seconds of a static image, those sections would require wildly different bitrates. With CBR, they'd get the same bitrate however; making the high-movement scene look pretty bad and the static image not much better.

CQP is for recordings where the exact bitrate at any given time doesn't matter much (but average bitrate does to a degree) and CBR for streaming where both current and average need to be relatively stable.

Im aware, but what i mean by "go pretty nuts" is itll suddenly shoot to hundreds of megabits per second and overload the encoder in situations it really shouldnt be. The issue is acknowledged in the post just before mine and ive even experienced it to a lesser degree at lower resolutions.

The encoder simply behaves better with CBR at the moment with H.265 especially if youre recording 4k. I was recording 720p 60fps at CQP 24 and it was spiking in excess of 100mb/s which I dont have happen on AMF, the Old Encoder, or the old OBS patch for HEVC. Its to be expected on a brand new encoder still in the works, at least on 720p/1080p CQP doesnt seem to go crazy too often. 4k seems to really suffer from it the most unfortunately for those who record in it.

[Atemu]

Interesting. Not sure that has anything to do with this add-on as it simply uses/configures the gstreamer VAAPI encoder whose responsibility it is to implement things like CQP. That might happen one step further down actually, in mesa.

As a workaround, I believe you can set a reasonable bitrate limit.

[Big-FG]

Interesting. Not sure that has anything to do with this add-on as it simply uses/configures the gstreamer VAAPI encoder whose responsibility it is to implement things like CQP. That might happen one step further down actually, in mesa.

As a workaround, I believe you can set a reasonable bitrate limit.

i didnt say it was related to the plugin, its and upstream issue please track the progress in the linked issue above like i mentioned previously.

[fzwoch]

Closing this. No idea what the exact status is, but if there is still something wrong it sounds like an upstream issue.