Closed Changliu52 closed 4 years ago
Hi @Changliu52,
Since we have one working system and the second to diagnose this should be just a matter of time.
A couple of ideas:
vainfo
and
vainfo --display drm --device /dev/dri/renderD128
If you don't have vainfo
install it with:
sudo apt-get install vainfo
Could you paste the output of
ffmpeg -encoders | grep vaapi
Could you paste the ouput of:
ldd realsense-nhve-h264
This will tell us what libraries we are using at runtime. There are obvious like avcodec
,avutil
but also indirect like libva
, libva-drm
ant lots of other coming from ffmpeg and librealsense depending how they were compiled.
We are mostly interested in the cases when some library is not used from the default system path
(/lib
, /usr/lib
).
Edits:
vainfo
if missingJust checked with:
and got working hardware encoding with NHVE out of the box so we need output from above commands to diagnose further.
Also information which version of LattePanda Alpha you have may be helpful.
panda@panda:~/realsense-network-hardware-video-encoder/build$ vainfo
libva info: VA-API version 0.39.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.39 (libva 1.7.0)
vainfo: Driver version: Intel i965 driver for Intel(R) Kabylake - 1.7.0
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Simple : VAEntrypointEncSlice
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileH264MultiviewHigh : VAEntrypointVLD
VAProfileH264MultiviewHigh : VAEntrypointEncSlice
VAProfileH264StereoHigh : VAEntrypointVLD
VAProfileH264StereoHigh : VAEntrypointEncSlice
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileVP8Version0_3 : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
panda@panda:~/realsense-network-hardware-video-encoder/build$ vainfo --display drm --device /dev/dri/renderD128
libva info: VA-API version 0.39.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.39 (libva 1.7.0)
vainfo: Driver version: Intel i965 driver for Intel(R) Kabylake - 1.7.0
vainfo: Supported profile and entrypoints
VAProfileMPEG2Simple : VAEntrypointVLD
VAProfileMPEG2Simple : VAEntrypointEncSlice
VAProfileMPEG2Main : VAEntrypointVLD
VAProfileMPEG2Main : VAEntrypointEncSlice
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileH264MultiviewHigh : VAEntrypointVLD
VAProfileH264MultiviewHigh : VAEntrypointEncSlice
VAProfileH264StereoHigh : VAEntrypointVLD
VAProfileH264StereoHigh : VAEntrypointEncSlice
VAProfileVC1Simple : VAEntrypointVLD
VAProfileVC1Main : VAEntrypointVLD
VAProfileVC1Advanced : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileJPEGBaseline : VAEntrypointEncPicture
VAProfileVP8Version0_3 : VAEntrypointVLD
VAProfileVP8Version0_3 : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
panda@panda:~/realsense-network-hardware-video-encoder/build$ ffme
No command 'ffme' found, did you mean:
Command 'ffe' from package 'ffe' (universe)
ffme: command not found
panda@panda:~/realsense-network-hardware-video-encoder/build$ ffmpeg -encoders | grep vaapi
ffmpeg version 3.4.1 Copyright (c) 2000-2017 the FFmpeg developers
built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.12) 20160609
configuration: --enable-shared
libavutil 55. 78.100 / 55. 78.100
libavcodec 57.107.100 / 57.107.100
libavformat 57. 83.100 / 57. 83.100
libavdevice 57. 10.100 / 57. 10.100
libavfilter 6.107.100 / 6.107.100
libswscale 4. 8.100 / 4. 8.100
libswresample 2. 9.100 / 2. 9.100
panda@panda:~/realsense-network-hardware-video-encoder/build$ ldd realsense-nhve-h264
linux-vdso.so.1 => (0x00007ffdd29fd000)
librealsense2.so.2.31 => /usr/lib/x86_64-linux-gnu/librealsense2.so.2.31 (0x00007f65105b5000)
libavcodec.so.58 => /usr/lib/x86_64-linux-gnu/libavcodec.so.58 (0x00007f650ee0d000)
libavutil.so.56 => /usr/lib/x86_64-linux-gnu/libavutil.so.56 (0x00007f650eaf2000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f650e770000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f650e55a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f650e190000)
libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007f650df78000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f650dc6f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f650da52000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6511a2c000)
libswresample.so.3 => /usr/lib/x86_64-linux-gnu/libswresample.so.3 (0x00007f650d833000)
libvpx.so.3 => /usr/lib/x86_64-linux-gnu/libvpx.so.3 (0x00007f650d40f000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f650d1ed000)
librsvg-2.so.2 => /usr/lib/x86_64-linux-gnu/librsvg-2.so.2 (0x00007f650cfb6000)
libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007f650cd63000)
libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f650ca52000)
libcairo.so.2 => /usr/lib/x86_64-linux-gnu/libcairo.so.2 (0x00007f650c73e000)
libzvbi.so.0 => /usr/lib/x86_64-linux-gnu/libzvbi.so.0 (0x00007f650c4b3000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f650c299000)
libsnappy.so.1 => /usr/lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007f650c091000)
libaom.so.0 => /usr/lib/x86_64-linux-gnu/libaom.so.0 (0x00007f650bb67000)
libcodec2.so.0.4 => /usr/lib/x86_64-linux-gnu/libcodec2.so.0.4 (0x00007f650b934000)
libgsm.so.1 => /usr/lib/x86_64-linux-gnu/libgsm.so.1 (0x00007f650b726000)
libmp3lame.so.0 => /usr/lib/x86_64-linux-gnu/libmp3lame.so.0 (0x00007f650b4b1000)
libopenjp2.so.7 => /usr/lib/x86_64-linux-gnu/libopenjp2.so.7 (0x00007f650b276000)
libopus.so.0 => /usr/lib/x86_64-linux-gnu/libopus.so.0 (0x00007f650b02c000)
libshine.so.3 => /usr/lib/x86_64-linux-gnu/libshine.so.3 (0x00007f650ae1f000)
libspeex.so.1 => /usr/lib/x86_64-linux-gnu/libspeex.so.1 (0x00007f650ac06000)
libtheoraenc.so.1 => /usr/lib/x86_64-linux-gnu/libtheoraenc.so.1 (0x00007f650a9c7000)
libtheoradec.so.1 => /usr/lib/x86_64-linux-gnu/libtheoradec.so.1 (0x00007f650a7ad000)
libtwolame.so.0 => /usr/lib/x86_64-linux-gnu/libtwolame.so.0 (0x00007f650a58a000)
libvorbis.so.0 => /usr/lib/x86_64-linux-gnu/libvorbis.so.0 (0x00007f650a35f000)
libvorbisenc.so.2 => /usr/lib/x86_64-linux-gnu/libvorbisenc.so.2 (0x00007f650a0b6000)
libwavpack.so.1 => /usr/lib/x86_64-linux-gnu/libwavpack.so.1 (0x00007f6509e8d000)
libwebp.so.5 => /usr/lib/x86_64-linux-gnu/libwebp.so.5 (0x00007f6509c31000)
libx264.so.155 => /usr/lib/x86_64-linux-gnu/libx264.so.155 (0x00007f6509770000)
libx265.so.179 => /usr/lib/x86_64-linux-gnu/libx265.so.179 (0x00007f65085fb000)
libxvidcore.so.4 => /usr/lib/x86_64-linux-gnu/libxvidcore.so.4 (0x00007f65082e7000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f65080e3000)
libva.so.1 => /usr/lib/x86_64-linux-gnu/libva.so.1 (0x00007f6507ec7000)
libva-drm.so.1 => /usr/lib/x86_64-linux-gnu/libva-drm.so.1 (0x00007f6507cc4000)
libva-x11.so.1 => /usr/lib/x86_64-linux-gnu/libva-x11.so.1 (0x00007f6507abe000)
libvdpau.so.1 => /usr/lib/x86_64-linux-gnu/libvdpau.so.1 (0x00007f65078ba000)
libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f6507580000)
libdrm.so.2 => /usr/lib/x86_64-linux-gnu/libdrm.so.2 (0x00007f650736e000)
libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f6511be0000)
libsoxr.so.0 => /usr/lib/x86_64-linux-gnu/libsoxr.so.0 (0x00007f6507109000)
libgdk_pixbuf-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0 (0x00007f6506ee7000)
libgio-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgio-2.0.so.0 (0x00007f6506b5f000)
libpangocairo-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0 (0x00007f6506952000)
libpango-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0 (0x00007f6506706000)
libcroco-0.6.so.3 => /usr/lib/x86_64-linux-gnu/libcroco-0.6.so.3 (0x00007f65064cb000)
libxml2.so.2 => /usr/lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f6506110000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f6505f08000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f6505c98000)
libpixman-1.so.0 => /usr/lib/x86_64-linux-gnu/libpixman-1.so.0 (0x00007f65059f0000)
libfontconfig.so.1 => /usr/lib/x86_64-linux-gnu/libfontconfig.so.1 (0x00007f65057ad000)
libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f6505503000)
libpng12.so.0 => /lib/x86_64-linux-gnu/libpng12.so.0 (0x00007f65052de000)
libxcb-shm.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-shm.so.0 (0x00007f65050da000)
libxcb-render.so.0 => /usr/lib/x86_64-linux-gnu/libxcb-render.so.0 (0x00007f6504ed0000)
libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f6504cae000)
libXrender.so.1 => /usr/lib/x86_64-linux-gnu/libXrender.so.1 (0x00007f6504aa4000)
libXext.so.6 => /usr/lib/x86_64-linux-gnu/libXext.so.6 (0x00007f6504892000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f650468a000)
libogg.so.0 => /usr/lib/x86_64-linux-gnu/libogg.so.0 (0x00007f6504481000)
libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f6504276000)
libXfixes.so.3 => /usr/lib/x86_64-linux-gnu/libXfixes.so.3 (0x00007f6504070000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f6503e4e000)
libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007f6503c4a000)
libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f6503a28000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f650380d000)
libpangoft2-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangoft2-1.0.so.0 (0x00007f65035f7000)
libthai.so.0 => /usr/lib/x86_64-linux-gnu/libthai.so.0 (0x00007f65033ee000)
libicuuc.so.55 => /usr/lib/x86_64-linux-gnu/libicuuc.so.55 (0x00007f650305a000)
libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f6502e31000)
libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f6502c2d000)
libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f6502a27000)
libharfbuzz.so.0 => /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0 (0x00007f65027c9000)
libdatrie.so.1 => /usr/lib/x86_64-linux-gnu/libdatrie.so.1 (0x00007f65025c1000)
libicudata.so.55 => /usr/lib/x86_64-linux-gnu/libicudata.so.55 (0x00007f6500b0a000)
libgraphite2.so.3 => /usr/lib/x86_64-linux-gnu/libgraphite2.so.3 (0x00007f65008e4000)
Looks like it is still using ffmpeg 3.4.1 I compiled from source?
My LattePanda is also M3-7Y30 I removed the ffmpeg 3.4.1 from the installation and recompiled the RNHVE package. Its better but gives the following output:
panda@panda:~/realsense-network-hardware-video-encoder/build$ ./realsense-nhve-h264 127.0.0.1 9766 color 640 480 30 50 /dev/dri/renderD128
libva info: VA-API version 0.39.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_39
libva info: va_openDriver() returns 0
[AVHWDeviceContext @ 0x19df000] Initialised VAAPI connection: version 0.39
[AVHWDeviceContext @ 0x19df000] VAAPI driver: Intel i965 driver for Intel(R) Kabylake - 1.7.0.
[AVHWDeviceContext @ 0x19df000] Matched driver string as known nonstandard driver "Intel i965 (Quick Sync)", setting quirks (0x2).
[h264_vaapi @ 0x1b35940] Input surface format is yuyv422.
[h264_vaapi @ 0x1b35940] No usable encoding profile found.
hve: cannot open video encoder codec
nhve: failed to initalize hardware encoder
unable to initalize, try to specify device e.g:
./realsense-nhve-h264 127.0.0.1 9766 color 640 360 30 5 /dev/dri/renderD128
./realsense-nhve-h264 127.0.0.1 9766 infrared 640 360 30 5 /dev/dri/renderD128
VAAPI configuration in the OS is ok
It seems it was compiled without vaapi support and with minimal option set (only --enable-shared
)
The output of
ffmpeg -encoders | grep vaapi
Should be:
V..... h264_vaapi H.264/AVC (VAAPI) (codec h264)
V..... hevc_vaapi H.265/HEVC (VAAPI) (codec hevc)
V..... mjpeg_vaapi MJPEG (VAAPI) (codec mjpeg)
V..... mpeg2_vaapi MPEG-2 (VAAPI) (codec mpeg2video)
V..... vp8_vaapi VP8 (VAAPI) (codec vp8)
At least passing --enable-vaapi
to configure
, maybe also some other switches would be necessary and compiled version maybe would work.
I am not sure.
I would do:
sudo ldconfig
git clone --recursive https://github.com/bmegli/realsense-network-hardware-video-encoder.git
cd realsense-network-hardware-video-encoder mkdir build cd build cmake .. make
Then I would try with infrared (infrared uses native VAAPI pixel format - nv12):
```bash
./realsense-nhve-h264 127.0.0.1 9766 infrared 640 360 30 5 /dev/dri/renderD128
Also in the build directory:
# in RNHVE build directory after make
cd network-hardware-video-encoder
cd harware-video-encoder
./hve-encode-raw-h264 10 /dev/dri/renderD128
I am not yet sure what's going on.
[h264_vaapi @ 0x1b35940] Input surface format is yuyv422.
[h264_vaapi @ 0x1b35940] No usable encoding profile found.
4.2 is enough far away from 3.4 (2 years of development) that something may have changed.
One thing I am certain:
Later on, KabyLake VAAPI hardware encoding only supports 4:2:0 chroma subsampling so maybe it is complaining in the newer version that it will not be able to use the 4:2:2 information from chroma yuyv422.
In FFmpeg this would correpond to FF_PROFILE_H264_HIGH_422 (not supported by VAAPI).
Somewhat blindly I have explicitly set the profile FF_PROFILE_H264_HIGH in the branch explicit-h264-profile
which may fix the problem.
So you may try:
#in RNHVE directory
git pull
git checkout explicit-h264-profile
cd build
make
./realsense-nhve-h264 127.0.0.1 9766 color 640 360 30 5 /dev/dri/renderD128
Later on I will check this myself (no time now)
Edit:
I checked myself in 16.04.6 with ffmpeg 4.2, and infrared streaming is working, color and depth not. Looks like HVE needs some update for 4.2.
So you may try [...]
Checked it, didn't help.
I am not planning to update HVE for FFmpeg 4.2 now, it is meant for 18.04 LTE (10 year support) with FFmpeg 3.4.
Your options are:
For option 1 and 2 you should probably first remove 4.2 ffmpeg, avcodec-dev, avutil-dev
sudo apt-get remove ffmpeg libavcodec-dev libavutil-dev
Then remove ppa (search for the correct command or remove from the ppa list)
Edit:
both master and explicit-h264-profile worked with infrared streaming.
./realsense-nhve-h264 127.0.0.1 9766 infrared 640 480 30 5 /dev/dri/renderD128
but non works with color
./realsense-nhve-h264 127.0.0.1 9766 color 640 480 30 5 /dev/dri/renderD128
with the same error as before
Looking at the output of vainfo, the problem is deeper than just FFmpeg version.
The most important part:
VAProfileVP8Version0_3 : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
In 16.04 you are missing the profile for HEVCMain10 encoding.
This is what you should have (18.04 ecosytem):
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointEncSlice
So even if you solve the problem with FFmpeg version you will not be able tu run depth encoding.
To get it working you would probably have to:
With 16.04 reaching end of support next year, 20.04 LTS coming next month, this is not worth the effort.
I suggest switching to 18.04 LTS.
I see. Thank you very much for the suggestions. It has been super useful!
Chang
@Changliu52
For reference:
intel-vaapi-driver:
Version 1.7.3 - 10.Nov.2016
- Add support for HEVC 10bit encoding on KBL
On 16.04 you have 1.7.0 version, on 18.04 there is 2.1.0 version
Dear @bmegli
I was able to compile the code with no problem in ubuntu 16.04 following https://github.com/bmegli/network-hardware-video-encoder/issues/3
But when running with
./realsense-nhve-h264 127.0.0.1 9766 color 640 480 30 50 /dev/dri/renderD128
on LattePanda Alpha, it reports the following errorI have checked that I can
ls /dev/dri/renderD128
. Not sure If there is any setup I need to go through? Im runing with realsense d435Thank you in advance.
Chang