Closed Changliu52 closed 3 years ago
The implementation works fine on M3-7Y30. For some reason, the bgr0 format doesn't seem to work on m3-8100Y
Hi @Changliu52,
Responding to this issue will take some time (I am busy with some deadlines at work, I have to assemble test environment with 20.04, etc).
Which example of RNHVE are you calling? realsense-nhve-h264
?
If it is H.264 (not HEVC Main10) it should work with both i965 and the default driver (newer).
Thanks for the prompt reply. I initialised the NHVE library as follows:
net_config_ = {0};
net_config_.ip = ip_char;
net_config_.port = (uint16_t)port_depth_;
hw_config_.pixel_format = "bgr0";
hw_config_.profile = FF_PROFILE_H264_CONSTRAINED_BASELINE; //FF_PROFILE_H264_MAIN, FF_PROFILE_H264_CONSTRAINED_BASELINE, FF_PROFILE_H264_BASELINE
hw_config_.encoder = NULL;
hw_config_.width = image_width_;
hw_config_.height= image_height_;
hw_config_.framerate = 30;
hw_config_.max_b_frames = 0;
hw_config_.device = NULL; //NULL as last argv argument, or device path
hw_config_.bit_rate = 1000000;
if( (streamer_color_ = nhve_init(&net_config_, &hw_config_)) == NULL )
return 1; // failure
I have a separate depth streamer NHVE initiated in nv12
format, which seem to have worked according to the console output.
The only hw_config_
difference is that pixel_format
. Not sure why the bgr0
can cause No usable encoding profile found.
error.
@Changliu52,
I confirm that in 20.04 (or rather with 20.04 FFmpeg version) there is a problem with non nv12/yuv420p/p010le pixel formats.
Looking at FFmpeg code this seems to be the commit where behavior changed, some notes are also here.
My shallow understanding after peeking into code is that smart profile auto-detection code tries to match software pixel format with compatible VAAPI profile. I am guessing here that it fails because there is no profile for bgr0/rgb0 (those were hardware pixel format conversions, under the hood it was loosing some information when converting 4:4:4 chroma of bgr0 to 4:2:0 of nv12).
Possible ways of solving this:
Since I have only looked at it, I may be wrong. It would be nice to check if it works with bgr0 on other generations than AmberLakeY (which M3-8100 is).
The funny thing is that avcodec, when queried for acceptable formats for VAAPI still reports bgr0.
Hi @bmegli, Really appreciate your detailed feedback. It looks exactly as the problem I experianced. Seems using bgr0 format is not supported by the VAAPI anymore.
Do you recommend any method to convert bgr0 data to yuv420p? The opencv Im using only offers the conversion for the other way round.
Im also looking at testing the new Lattepanda delta https://www.dfrobot.com/product-1908.html?tracking=5f61ea809f1a0. maybe this one can work?
Hi @Changliu52,
I have just checked it on KabyLake in 20.04 (FFmpeg 4.2.4) with the same result for bgr0. This confirms that it is not really AmberLake Y(m3-8100) problem but rather software layer problem.
Seems using bgr0 format is not supported by the VAAPI anymore.
Precisely speaking the change is probably on the FFmpeg level, not the VAAPI level.
Do you recommend any method to convert bgr0 data to yuv420p? The opencv Im using only offers the conversion for the other way round.
Probably it should be just CV_BGR2YUV_YV12
or less likely CV_BGR2YUV_I420
. Are you certain your OpenCV doesn't support it?
Short term - any method is ok.
Long term:
With optimized pixel format conversion (I expect OpenCV has such) you will probably not see any difference from hardware conversion (apart from a bit higher CPU usage).
Im also looking at testing the new Lattepanda delta https://www.dfrobot.com/product-1908.html?tracking=5f61ea809f1a0. maybe this one can work?
As above, this doesn't seem to be caused by hardware, rather by software (FFmpeg version).
Delta possibly may not support HEVC Main10 encoding (used for depth encoding).
Edits:
By using grep on FFmpeg releases we see that this commit was introduced between 4.0 and 4.1.
git status
On branch release/4.1
Your branch is up to date with 'origin/release/4.1'.
git log | grep "vaapi_encode: Choose profiles dynamically"
vaapi_encode: Choose profiles dynamically
# nothing like that in previous releases
By browsing 2018 August ffmpeg-devel mailing list archive we see that the commit was introduced as one of 41 smaller patches reflecting larger effort in FFmpeg VAAPI support.
The fact that it is 4.1 release means that there is no trivial way to check it (e.g. Ubuntu 18.04 is FFmpeg 3.4, Ubuntu 19.04 is already at 4.2).
The implementation works fine on M3-7Y30. For some reason, the bgr0 format doesn't seem to work on m3-8100Y
Probably it doesn't work with bgr0
starting from FFmpeg 4.1 (so both in Ubuntu 19.04 and 20.04)
@Changliu52
I have filed a bug on FFmpeg bug tracker for this.
Upvote it if you are interested in having this fixed/investigated.
@bmegli Appreciate the feedback and details. Really keen to see they could bring back the rgb0 support.
not sure how to up vote it? do I have to register an account?
do I have to register an account?
Yes, exactly.
Also try the CV convertion:
Do you recommend any method to convert bgr0 data to yuv420p? The opencv Im using only offers the conversion for the other way round.
Probably it should be just CV_BGR2YUV_YV12
From discussion in FFmpeg trac it looks like the long long-term solution will be to invoke vaapi_scale
filter for hardware color conversion + some logic to decide when such conversion is needed.
HVE master already uses vaapi_scale filter (not for color conversion, for scaling if requested by user).
NHVE and RNHVE both use older version of HVE without scaling (thorugh vaapi_scale
) support.
So the roadmap would be:
Hi @bmegli I was testing the on the 20.04 on M3-8100Y. I have set
LIBVA_DRIVER_NAME=i965
the encoder seems to still have problem to initialise. The console output is: