search

Igalia / cog

WPE launcher and webapp container
MIT License
238 stars 61 forks source link

canvas.drawImage() not working on i.MX8M mini #481

Open bakusui opened 2 years ago

bakusui commented 2 years ago

When I draw the Video tag to canvas with canvas.drawImage() in javascript, I can't draw it. Is this a WPE/Cog bug? Or am I missing a setting?

[my environment] SoC: NXP i.MX8M mini kernel: v5.10.9 WPE: v2.36.0/Cog:v0.12.3 backend: fdo, wayland (weston)

[Solution] If I remove "gst_gl" from the build options, I can draw with canvas.drawImage().

I want to keep gst_gl enabled because the drawing performance drops.

A part of "local.conf" is extracted.

IMAGE_INSTALL_append = "wpewebkit cog weston" PREFERRED_PROVIDER_virtual/wpebackend = "wpebackend-fdo" IMAGE_INSTALL_append = "packagegroup-fsl-gstreamer1.0 packagegroup-fsl-gstreamer1.0-full" PACKAGECONFIG_append_pn-wpewebkit += " webrtc mediastream opengl " PREFERRED_VERSION_wpewebkit = "2.36.0+git%"

PACKAGECONFIG_remove_pn-wpewebkit += "gst_gl"

philn commented 2 years ago

Hi, thanks for the report! Can you share gst logs and pipeline dumps please? I'm interested in the case where gst-gl is enabled. See https://github.com/Igalia/meta-webkit/wiki/Providing-useful-GStreamer-Zero-copy-issue-reports

bakusui commented 2 years ago

Hi, thank you for confirming. The requested data. Is this okay? CaseA : drawimage success case. (gst_gl removed) remove_gst_gl

CaseB : drawimage fail case. (gst_gl enabled) gst_gl

philn commented 2 years ago

When you says it doesn't work, do you mean nothing is rendered or... something else? Can you share the reproduction case? Is this happening with this demo too? https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Manipulating_video_using_canvas

bakusui commented 2 years ago

Sorry, I can't get the board to work right away. https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Manipulating_video_using_canvas Confirmed on the page above. result : No people are drawn on both canvases.

philn commented 2 years ago

Ok so the issue might be somewhere in MediaPlayerPrivateGStreamer::paint() which is not much instrumented with logs... First thing would be to see if one of the early returns there is hit or not.

I can't reproduce this issue here unfortunately...

bakusui commented 2 years ago

That's probably right I think that the following function acquires the image with the "paint" function. ImageGStreamer::ImageGStreamer(GstSample* sample) The "bufferData" variable is being treated as a pointer to the image, but it seems that this bufferData actually has no image data. When I tried filling "bufferData" with red, red was displayed.

philn commented 2 years ago

Can you check if the if (!GST_VIDEO_INFO_IS_RGB(&videoInfo) || hasExternalOESTexture) path is taken or not? If not, try to force it.

bakusui commented 2 years ago

That path is taken.

When I tried filling "bufferData" with red, red was displayed.

The test code was written in the conditions.

       for (int x = 0; x < width; x++) {
            for (int y = 0; y < height; y++) {
#if 1 //debug: fill red
                bufferData[0] = 255; // R
                bufferData[1] = 0;   // G
                bufferData[2] = 0;   // B
                bufferData[3] = 255; // A
#endif
philn commented 2 years ago

So the GL color conversion seems to fail... can you run this with GST_DEBUG=gl*:6 ?

philn commented 2 years ago

Which version of GStreamer are you using? I hope it's not been much butchered by NXP...

bakusui commented 2 years ago

I tried it.I tried it. Do you have any hints? ("Manipulating_video_using_canvas" page play.) gst.zip and gstreamer version is... ~# gst-launch-1.0 --version gst-launch-1.0 version 1.18.0 GStreamer 1.18.0 Unknown package origin

philn commented 2 years ago

Can you try to disable this? https://github.com/WebKit/WebKit/blob/main/Source/WebCore/platform/graphics/gstreamer/GLVideoSinkGStreamer.cpp#L73..L82

bakusui commented 2 years ago

comment out? build errors.　What is this test for?

error: 'imxVideoConvertG2D' was not declared in this scope; did you mean 'GstVideoConverter'?

As an alternative, I inserted the code below before the reference,

imxVideoConvertG2D = nullptr;

now building. tell me if it doesn't make sense

philn commented 2 years ago

Ah yes I missed some bits, sorry. I just wanted to avoid this element. A simple one-liner is to replace its name with identity which is a passthrough element.

bakusui commented 2 years ago

( Nullptr assignment soft still failed to "drawimage()". )

I just wanted to avoid this element. A simple one-liner is to replace its name with identity which is a passthrough element.

Sorry, I have no knowledge. How should I change the code.

philn commented 2 years ago

In https://github.com/WebKit/WebKit/blob/main/Source/WebCore/platform/graphics/gstreamer/GLVideoSinkGStreamer.cpp#L76 replace the string there with "identity". But anyways, if you tested already and it fails, it's not helping... Someone will need to properly debug this, we're not getting very far here :(

bakusui commented 2 years ago

replace the string there with "identity".

It fails.

Debug log added at ImageGStreamer::ImageGStreamer(). Case1:gst_gl enabled(fail)

[TEST]ImageGStreamer_DEBUG RGB?=2 <-GST_VIDEO_INFO_IS_RGB(&videoInfo) [TEST]ImageGStreamer_DEBUG m_frameMapped==1 <-m_frameMapped [TEST]ImageGStreamer_DEBUG m_hasAlpha/componentSwapRequired=1 / 1 <-m_hasAlpha,componentSwapRequired [TEST]ImageGStreamer_DEBUG data[0:1]=0:0 <-bufferData[] [TEST]ImageGStreamer_DEBUG data[2:3]=0:0

Case2:gst_gl disabled(success)

[TEST]ImageGStreamer_DEBUG RGB?=2 [TEST]ImageGStreamer_DEBUG m_frameMapped==1 [TEST]ImageGStreamer_DEBUG m_hasAlpha/componentSwapRequired=0 / 0 [TEST]ImageGStreamer_DEBUG data[0:1]=14:bc [TEST]ImageGStreamer_DEBUG data[2:3]=c2:ff

Something wrong with Image input? What should I check next? Where does "m_sample.get()" write from?

bakusui commented 2 years ago

Log of gstreamer,I found this 0:00:00.076271750 7378 0xaaaaeedadde0 WARN video-info video-info.c:526:gst_video_info_from_caps: invalid matrix 3 for RGB format, using RGB I thought it was a format error and inserted "capsfilter" before "imxvideoconvert_g2d". No more error logs. But drawImage failed. what should I analyze? capsfilter_added gst.log

philn commented 2 years ago

I think this boils down to bad interaction between the NXP decoder and the gst-gl conversion elements...

bakusui commented 2 years ago

Sorry, "invalid matrix 3 for RGB format, using RGB" doesn't seem to be related. "imxvideoconvert_g2d" output RGB data. Data check result.(fakesink dump)

Zero : gst-launch-1.0 filesrc location=video.mp4 ! qtdemux name=d d.video_0 ! h264parse ! vpudec ! imxvideoconvert_g2d ! glupload ! glcolorconvert ! fakesink dump=true Zero : gst-launch-1.0 filesrc location=video.mp4 ! qtdemux name=d d.video_0 ! h264parse ! vpudec ! imxvideoconvert_g2d ! glupload ! glcolorconvert ! gldownload ! fakesink dump=true Zero : gst-launch-1.0 filesrc location=video.mp4 ! qtdemux name=d d.video_0 ! h264parse ! vpudec ! imxvideoconvert_g2d ! glupload ! gldownload ! fakesink dump=true Zero : gst-launch-1.0 filesrc location=video.mp4 ! qtdemux name=d d.video_0 ! h264parse ! vpudec ! imxvideoconvert_g2d ! glupload ! fakesink dump=true RGBA : gst-launch-1.0 filesrc location=video.mp4 ! qtdemux name=d d.video_0 ! h264parse ! vpudec ! imxvideoconvert_g2d ! fakesink dump=true

The result of changing "fakesink" to "waylandsink".

NG(could not link) OK(but,yellow turns blue) //Color normalized by adding "capsfilter caps=video/x-raw,format=RGB16"　before waylandsink NG(Internal data stream error.) NG(could not link) OK

I tried "gldownload" added, but drawimage() faild.

I think the following is the problem

  1. Missing "gldownload" in pipeline
  2. Is the image reference method in "appsink" wrong?
bakusui commented 2 years ago

At Source/WebCore/platform/graphics/gstreamer/GLVideoSinkGStreamer.cpp, I set something like "glcolorconvert ! gldownload ! capsfilter caps=video/x-raw,format=RGBx ! appsink" in the pipeline and drawimage() succeeded!

Thanks.

philn commented 2 years ago

that's a hack though... the right solution would be to fix GPU downloading on WebKit side. Adding a gldownload in the sink is not great for performance.

philn commented 2 years ago

cc @magomez

magomez commented 2 years ago

I'm afraid I can't help much with this, as it seems to be an internal GStreamer issue.

When the ImageGStreamer class is created, the frame data is read into the image. This is what's done in ImageGStreamerCairo.cpp, and what you have been debugging. When gst-gl is disabled, the video frames are CPU memory buffers with BGRx format, and that case works well. When gst-gl is enabled, the video frames are RGBA textures, which what's failing.

The code on ImageGStreamerCairo relies on gst_video_frame_map() providing a pointer to the frame pixels. When using gst-gl, this means that gstreamer internally will call glReadPixels to get the pixel values from the texture and put them into a normal buffer accessible by GST_VIDEO_FRAME_PLANE_DATA. If gstreamer is failing internally to achieve this I then it's probably some gstreamer problem over the imx8 platform.

philn commented 2 years ago

Before reaching ImageGStreamerCairo we do a conversion in MediaPlayerPrivateGStreamer::paint()... I think that's what's failing.

magomez commented 2 years ago

Before reaching ImageGStreamerCairo we do a conversion in MediaPlayerPrivateGStreamer::paint()... I think that's what's failing.

Ah, I see. My reasoning still applies anyway, the buffer data should be accessible after gst_video_frame_map() in ImageGStreamerCairo, but it doesn't seem to be. Could it be a problem with the color conversion performed in paint????

From the pipeline dumps attached before, when gst-gl is enabled we're getting RGBA frames from the sink, which is expected.

Then, assuming that I'm properly interpreting the meaning of this log in ImageGStreamerCairo:

Case1:gst_gl enabled(fail) [TEST]ImageGStreamer_DEBUG RGB?=2 <-GST_VIDEO_INFO_IS_RGB(&videoInfo) [TEST]ImageGStreamer_DEBUG m_frameMapped==1 <-m_frameMapped [TEST]ImageGStreamer_DEBUG m_hasAlpha/componentSwapRequired=1 / 1 <-m_hasAlpha,componentSwapRequired [TEST]ImageGStreamer_DEBUG data[0:1]=0:0 <-bufferData[] [TEST]ImageGStreamer_DEBUG data[2:3]=0:0

m_hasAlpha is 1 and componentSwapRequired is 1, which means that the frame format at this point is RGBA. So no color conversion has been performed on the frame. If a color conversion was performed, the format should be BGRA, which would have m_hasAlpha as 1 and componentSwapRequired as 0.

What decides whether a color conversion has to be performed in paint it this if

if (!GST_VIDEO_INFO_IS_RGB(&videoInfo) || hasExternalOESTexture) {

The incoming video is RGBA, so the first part is false. And I guess there's not ExternalOESTexture support on the platform, so it's probably false as well.

I still think that this is an internal gstreamer problem on that platform. For some reason the download of the frames from the GPU is failing.

philn commented 2 years ago

I still think that this is an internal gstreamer problem on that platform. For some reason the download of the frames from the GPU is failing.

Yeah that's also my assessment. I'm not sure we can do much about this. Might be better to contact the NXP customer support.

bakusui commented 2 years ago

Thank you for your comment. (I'm sorry if I said something strange because I don't have knowledge.) Can "gst_video_frame_map()" retrieve data from hardware in any case? There seems to be corresponding code for each case in gstgldownloadelement.c. However, video-frame.c(gst_video_frame_map_id) doesn't seem to support it. The following pages also seem to use gldownload. https://developer.ridgerun.com/wiki/index.php/IMX8/Multimedia/GStreamer_Support/Hardware-accelerated_plugins In NXP's gstreamer user guide sink appears to use glimagesink. https://community.nxp.com/t5/i-MX-Processors-Knowledge-Base/i-MX-8-GStreamer-User-Guide/ta-p/1098942

philn commented 2 years ago

Usually if a frame is in GPU memory, its data can be accessed from CPU using the GST_MAP_GL flag. But here indeed it's not used... Maybe that could be done in ImageGStreamerCairo.

bakusui commented 2 years ago

ImageGStreamerCairo edited(add GST_MAP_GL,delete gldownload etc.). But crashed when playing video.

[ 637.640428] audit: type=1701 audit(1665481858.136:31): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=1744 comm="WPEWebProcess" exe="/usr/libexec/wpe-webkit-1.0/WPEWebProcess" sig=11 res=1 (cog:1720): Cog-Core-WARNING **: 09:50:58.283: https://developer.mozilla.org/en [ImageGStreamerCairo.patch.txt](https://github.com/Igalia/cog/files/9753866/ImageGStreamerCairo.patch.txt) -US/docs/Web/API/Canvas_API/Manipulating_video_using_canvas Crash!: The renderer process crashed. Reloading the page may fix intermittent failures.

philn commented 2 years ago

Can you try this branch? https://github.com/philn/WebKit/tree/gl-draw-image And provide a log file: GST_DEBUG="webkit*:8,gl*:8" GST_DEBUG_FILE=gst.log cog...

bakusui commented 1 year ago

My yocto tree and the version are quite different, so it is difficult to deal with. https://github.com/WebKit/WebKit/compare/main...philn:WebKit:gl-draw-image Is the result of applying the above diff to my tree ok? Drawimage fail log. https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Manipulating_video_using_canvas page played. gst.log.gz