ishmael1985
commented
5 years ago Well it isn't supported atm. It's not very difficult to add the crop feature though.
Open bcroxall opened 5 years ago
ishmael1985
commented
5 years ago Well it isn't supported atm. It's not very difficult to add the crop feature though.
bcroxall
commented
5 years ago @ishmael1985 Thanks for the quick reply. I really appreciate it. Is cropping on the roadmap? Do you have an approximate timeline? I'm going to look into implementing it myself but I have a couple questions. Would you implement cropping with caps filters like was done in earlier versions of this plugin (crop-x, crop-y, crop-w, crop-h) or would you implement settings within mfxvpp? I'm just hoping you can get me started in the right direction. Thanks.
ishmael1985
commented
5 years ago @bcroxall I don't think so, afaik. No idea about the timeline too, since I'm not with the original team anymore who worked on this project, even if I am the main author. I'm a little pressed for time these days, so I couldn't really answer that many queries at once on this project, sorry about that. I can guide you to add the feature though, but I suggest you follow the videocrop element options and add them to mfxvpp. If you look in the MSDK manual, cropping is achieved by setting the MSDK context, in the gst-mfx codebase, that setting is done in gstmfxfilter.c source - https://github.com/intel/gstreamer-media-SDK/blob/topic_linux_and_window/gst-libs/mfx/gstmfxfilter.c I recommend you work with the topic_linux_and_window branch, or better yet, my fork here - https://github.com/ishmael1985/gstreamer-media-SDK
bcroxall
commented
5 years ago @ishmael1985 Would you mind giving a quick comparison of mainline, topic_linux_and_window, and your fork? Is mainline not getting the attention it should? I'd like to understand why I should choose one over the other. Thanks.
bcroxall
commented
5 years ago @ishmael1985 I know you're busy but I have a couple questions. Thanks for helping me out if you have the time. I am wondering how to be sure I am using video memory and hardware acceleration. My pipeline is as follows with a few settings set on icamerasrc (I have a mipi camera connected to a custom board):
icamerasrc ! "video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1" ! mfxvpp ! "video/x-raw(memory:MFXSurface),format=NV12,width=1920,height=1080" ! mfxsinkelement.
As you can see, I am setting memory:MFXSurface on the filter between mfxvpp and mfxsinkelement. However, this resize from 4K to 1080 is making a huge hit on my CPU. Am I doing something incorrectly? In general, how do I ensure I am using video memory and hardware acceleration.
Another thing for comparison, if I replace the mfx plugins with vaapi plugins, the cpu usage is about 3/4 of what it was with mfx. Is this expected or reasonable?
More detail...if I set io-mode=3 (dma_import) and buffer_usage=3 (dma_export) on icamerasrc, the CPU usage is cut drastically...a great thing. However, mfxvpp must not be able to handle the memory type as the display is garbled. For reference, when using vaapi, the CPU usage was also cut drastically and the video was displayed properly.
Also, I am implementing an application in Qt. They have some nice plugins that I would like to use. They are gl plugins however (e.g. qmlglimagesink). How would you recommend integrating from icamerasrc to gl plugins using the Intel Media SDK gstreamer plugins?
I appreciate the help.
ishmael1985
commented
5 years ago @bcroxall the mainline branch is focused for Intel customer specific requirements, while my fork or the topic_linux_and_window is geared for broad market usage with full support for Windows and more features. That's why I'd recommend my fork over the mainline one here.
ishmael1985
commented
5 years ago @bcroxall that doesn't sound right, there seems to be a mistake in your pipeline in the mfxvpp output caps where you specify 4k instead of 1080p. It should have much lower CPU usage when specifying dmabuf-import, and mfxvpp should support this, but I only tested this to work with v4l2src and not icamerasrc. It's possible mainline has fixes for this which has not been ported to topic_linux_and_windows, you could check out both branches for your use case.
bcroxall
commented
5 years ago @ishmael1985 My pipeline was setup to exercise a resize by mfxvpp (4K to 1080) as a test. Your comment is correct. I had 4K coming in and 1080 going out. It works fine if io-mode is not set on icamerasrc. If I set io-mode=3, the display looks like there is a format issue. It's "blocky" and I can see motion but there is a misalignment of some kind. If I run 1080 in and out of mfxvpp, I get the same behavior depending on how io-mode is set on icamerasrc. I've tried every combination of formats and the problem persists. Does this sound like it could be a bug in mfxvpp or would you think it's more likely a problem in icamerasrc? Without knowing more, I would imagine mfxvpp would operate the same whether the buffer was system or video (dma). I've been spending a fair bit of time in the code, but if you would suggest a specific spot to look at, I'd appreciate it.
bcroxall
commented
5 years ago @ishmael1985 I was looking over other issues and discovered issue #125. My display looks very similar to the image attached to that issue. I am also using the Sony imx274 (device-name=imx274-2). @SiewHoon mentioned that this was fixed in 1.4.0-rc0, however I am using 1.4.0-rc6 and getting the same results. We are an Intel customer and it seems most appropriate to use mainline but I would have expected to see an MR4 branch in conjunction with other Intel MR4 releases for the hardware we are using. @SiewHoon do you expect to merge topic_linux_and_window with master soon and will you create an MR4 branch? Again, any suggestions to address this in the current mainline would be awesome. Thanks.
ishmael1985
commented
5 years ago @bcroxall I would love to have fixed this issue but I didn't have the hardware to test that time, now I'm doubly busy these days. As for the merge I've made this request for a very long time #49. I would want an answer from @SiewHoon for this myself. Till then maybe you could look over Siew Hoon's commits related to #125 and send a PR to my fork for review.
SiewHoon
commented
5 years ago (Only for imx274 )
Command: gst-launch-1.0 icamerasrc io-mode=3 ! "video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1, tiled=false" ! mfxvpp ! "video/x-raw,format=NV12,width=1920,height=1080" ! mfxsinkelement
The code fixed related to #125 is already fixed in mainline, but not in topic_linux_and_window branch.
bcroxall
commented
5 years ago @SiewHoon I haven't looked at the code yet concerning the tiled setting but I'd like to understand why this setting is needed (and only for imx274?). The pipeline you provided did require specifiying memory:MFXSurface on the latter caps filter but it worked. Thanks for the help.
@SiewHoon Do you expect to have an MR4 release in the near future? The other Intel packages we're using are MR4. I presume this code is on the same version scheme.
@SiewHoon @ishmael1985 Since you're both responding to this issue, I wanted to ask if you see occasional glitches in the display when using the mfx plugins. I am using weston as the compositor so it could be introducing something??? (X is not installed.) Too often for our application, I see a flash of what looks like a white horizontal line of pixels (100-200 pixels wide) in random locations on the display. When using X and vaapi plugins, I don't see these issues? I haven't tried X with the mfx plugins.
@ishmael1985 I plan to look over the fixes to #125 to at least understand them. I'll try and let you know what I find.
SiewHoon
commented
5 years ago imx274 camera sensor output is NV12 linear. For VAAPI hardware is using NV12 format, always will be NV12 tiled. It is NV12 linear vs NV12 Tiled problem. That why you see the corruption issue in gst-msdk if you did not set tiled=false on io-mode=3 (dmabuf import ).
If you got another camera sensor like imx185-li output also is NV12 linear. It will work with this fixed as well.
Another issue you mention: "I see a flash of what looks like a white horizontal line of pixels (100-200 pixels wide) in random locations on the display", please provide the picture and gst command pipeline.
bcroxall
commented
5 years ago @SiewHoon I'll have to get a capture of an image with the glitch. It happens with the pipeline you provided above but it happens more regularly while doing a 4k to 1080 resize...
gst-launch-1.0 icamerasrc device-name=imx274-2 io-mode=3 scene-mode=ull ! "video/x-raw,format=NV12,width=3840,height=2160,framerate=30/1,tiled=false" ! mfxvpp ! "video/x-raw(memory:MFXSurface),format=NV12,width=1920,height=1080" ! mfxsinkelement
bcroxall
commented
5 years ago @SiewHoon @ishmael1985 I have something more pressing than the glitch going on right now. With io-mode=3, the pipeline works well with gst-launch. If I try to run the same pipeline in the application, I'm getting a seg fault with the backtrace below. For some reason, the display pointer is null on one of the surfaces. Any ideas??? Thanks.
at /usr/src/debug/gstreamer-msdk/1.4.0-rc6+gitAUTOINC+301ffd25bb-r0/git/gst-libs/mfx/wayland/gstmfxwindow_wayland.c:243at /usr/src/debug/gstreamer-msdk/1.4.0-rc6+gitAUTOINC+301ffd25bb-r0/git/gst-libs/mfx/gstmfxwindow.c:456at /usr/src/debug/gstreamer-msdk/1.4.0-rc6+gitAUTOINC+301ffd25bb-r0/git/gst/mfx/gstmfxsink.c:110@SiewHoon @ishmael1985 I figured out my problem above. In my code, I had named my instance of icamerasrc as "source". When mfxvpp sets up its sinkpad buffer pool, it looks upstream for an element named "camerasrc" and therefore didn't query the source for its io-mode setting. I'll just not name my source but it seems there would be a better way to do this.
I cannot figure out how to crop the incoming video stream other than from the top, left corner (CropX=0, CropY=0). How is this to be done using mfxvpp? Thanks.