search

plugorgau / bbb-render

Scripts to convert a BigBlueButton recording into a single video file
MIT License
64 stars 26 forks source link

Render Annotations #8

Closed symptog closed 3 years ago

symptog commented 3 years ago

This PR add the ability to render annotations. The annoations from the shapes.svg will be rendered as png files and added as asset to the layer.

symptog commented 3 years ago

Rendered Example: https://chemnitzer.linux-tage.de/2021/de/programm/beitrag/128#video

aukondk commented 3 years ago

Looking good. Not sure why it's not working for me. It creates new png files with the slide background at the right places but the vectors don't get drawn. The SVG data works fine if I dump the decoded string to the terminal and load it into Inkscape but cairosvg doesn't want to draw it. I'll try again with a clean environment tomorrow.

symptog commented 3 years ago

Can you test it with the full (root based) filesystem path? Maybe we need to add the full path for the images.

like ./make-xges.py --annotations /my/full/path/to/presentation presentation.xges

aukondk commented 3 years ago

I thought of that and no, it doesn't make a difference.

I've tried spinning up a VM and only installing what I need to get it running. Ubuntu 20.04 same as my local box. Still the same issue.

I've now played with cairosvg from the command line and after a lot of dead ends and red herrings I've found the issue is all the vector elements (all \ in my case) have "visibility:hidden;" in their styles.

<g id="image138-draw17" class="shape" timestamp="5590.4" undo="-1" shape="image138-line1" style="stroke:#ff0000;stroke-width:8.63309;visibility:hidden;fill:none;stroke-linecap:butt">
      <line x1="843.75" y1="354.22662" x2="938.71406" y2="358.54316"/>
    </g>

Inkscape ignores this but cairosvg does not. A quick find and replace on the shapes.svg confirms that it works if I remove that string from the lines.

Not sure why this happened with mine and not yours. Maybe different library or BBB versions.

symptog commented 3 years ago

Can you share a link to this presentation?

aukondk commented 3 years ago

https://bigpinkbutton.ferit.hr/playback/presentation/2.0/playback.html?meetingId=1dd49012cf195b0c70724c45c10cf4c565d473d2-1606472131356

Line drawing starts at about 93 minutes.

symptog commented 3 years ago

if I remove visibility:hidden; from the elements the lines are rendered correct.

I think the visibility property is required for the bbb-player as the javascript just needs to change this property to display or hide the elements.

symptog commented 3 years ago

fixed those shapes, but text shapes are still missing.

jhenstridge commented 3 years ago

I haven't had time for a full review, but just a few initial points:

  1. Do we need cairosvg here? There is a GStreamer element for loading SVG files via librsvg, which I would have thought would allow GES to load SVG files as clips directly.
  2. The use of intervaltree looks pretty neat. It could probably simplify some of the existing logic too.
  3. I haven't been relying on a requirements.txt file so far because not all of the deps can be installed by pip. The main one is gst-python, which can't be installed from pypi. I'm not opposed to adding the file, but don't want to give people the impression that they can easily set this all up in a virtualenv.
symptog commented 3 years ago

I haven't had time for a full review, but just a few initial points:

  1. Do we need cairosvg here? There is a GStreamer element for loading SVG files via librsvg, which I would have thought would allow GES to load SVG files as clips directly.

I've tried to write SVG files but ges-launch was unable to load the files. Maybe I've missed something. I'm currently using cairosvg to render the images. However the Text-Inputs are still missing and I don't know how to properly render the text within a bounding box. The player just displays the xhtml element. To render an image/svg we need correct text elements.

  1. The use of intervaltree looks pretty neat. It could probably simplify some of the existing logic too.

Please feel to add annotations to the code :)

  1. I haven't been relying on a requirements.txt file so far because not all of the deps can be installed by pip. The main one is gst-python, which can't be installed from pypi. I'm not opposed to adding the file, but don't want to give people the impression that they can easily set this all up in a virtualenv.

Yes. Maybe I or we can just add the instructions for a venv to the Readme?

jhenstridge commented 3 years ago

I've tried to write SVG files but ges-launch was unable to load the files. Maybe I've missed something.

I had a go creating a project in Pitivi and importing an SVG image. It seemed to work, so perhaps I'm not driving the API correctly. The <asset/> element looks a bit different to those produced by make-xges.py (it has a <stream-info/> child), so maybe there is an extra step needed to when loading the assets.

jhenstridge commented 3 years ago

I did a bit of work using your branch as a base and pushed it here:

https://github.com/plugorgau/bbb-render/tree/annotation-support

The main deltas are:

  1. Combine add_slides and add_slides_with_annotations into one method.
  2. Keep the "Slides" layer as just the PNG slides, even when annotations are enabled.
  3. Put the annotations into their own layer sitting above the slides layer.
  4. The annotation images now have a transparent background rather than being composited onto their slide.
  5. Drop the CairoSVG dependency in favour of saving the annotations as SVG and leave it to GES to render them.

I don't have any readily available recordings with text annotations here, but I wouldn't be surprised if they work with this branch through the librsvg renderer.

symptog commented 3 years ago

I did a bit of work using your branch as a base and pushed it here:

https://github.com/plugorgau/bbb-render/tree/annotation-support

The main deltas are:

  1. Combine add_slides and add_slides_with_annotations into one method.
  2. Keep the "Slides" layer as just the PNG slides, even when annotations are enabled.
  3. Put the annotations into their own layer sitting above the slides layer.
  4. The annotation images now have a transparent background rather than being composited onto their slide.
  5. Drop the CairoSVG dependency in favour of saving the annotations as SVG and leave it to GES to render them.

I don't have any readily available recordings with text annotations here, but I wouldn't be surprised if they work with this branch through the librsvg renderer.

There is still a bug with the rectangles and lines. Your are missing the following logic: https://github.com/plugorgau/bbb-render/pull/8/commits/16e732f3a35ba2f488cd5478de99b7b016820e80

symptog commented 3 years ago

for the shapes with the same shape-attribute: the end need to be the start from the next shape

https://github.com/plugorgau/bbb-render/pull/8/commits/16e732f3a35ba2f488cd5478de99b7b016820e80#diff-cf935f084d9ce4eb989fb7217297676b92b90fe1e2315b08a0b25e7fdf60bfdeR333

jhenstridge commented 3 years ago

Looking at the JS used to play back presentations, it sounds like the correct behaviour would be to discard all but the last

https://github.com/bigbluebutton/bigbluebutton/blob/32c741cef1c701e73bc40a3f1ec8045d41b7f836/record-and-playback/presentation/playback/presentation/2.0/lib/writing.js#L715-L717

Does c3938002 look like it handles this correctly?

symptog commented 3 years ago

please don't. we won't see the drawing process. Even in the JS-Version the drawing process is displayed. Example: https://webroom.hrz.tu-chemnitz.de/playback/presentation/2.0/playback.html?meetingId=ef153938d8b15587eff96c475d2c355c6161a4b8-1616591890688

jhenstridge commented 3 years ago

I can see the drawing process in a video I rendered using this branch. I'm doing the shape culling on a per-interval basis, so anything that code culls will have been displayed in an earlier interval.

jhenstridge commented 3 years ago

Here is how my branch renders the recording you linked:

https://user-images.githubusercontent.com/12724165/113587222-0d0ecd00-9661-11eb-8eea-bf637a95873f.mp4

It seems to show the shapes being drawn interactively the same as the web playback. The obvious thing missing from the video is the text test. While rsvg can handle SVG native <text> elements, what we've got in the recording is HTML embedded via <foreignObject>:

    <g id="image1-draw262" class="shape" timestamp="58.1" undo="-1" shape="image
1-text4" style="color:#0000ff;word-wrap:break-word;visibility:hidden;font-family
:Arial;font-size:16.12903px">
      <switch>
        <foreignObject width="484.67741" height="181.45281" x="863.55846" y="68.
53331">
          <p xmlns="http://www.w3.org/1999/xhtml" style="margin:0;padding:0">Tes
t Text 1234567</p>
        </foreignObject>
      </switch>
    </g>

It looks like fairly simple restricted set of HTML (a paragraph containing text nodes and line breaks), so maybe we can transform it into equivalent <text> elements:

https://github.com/bigbluebutton/bigbluebutton/blob/1a4dd87f7f09c526a98fb9a8568bbb47078b440a/record-and-playback/presentation/scripts/publish/presentation.rb#L327-L355

jhenstridge commented 3 years ago

So it looks like the reason BBB is using an HTML foreignObject for text is for line wrapping. There is no native support for line wrapping in SVG 1.1, and it seems neither RSVG or CairoSVG support the text wrapping syntax of SVG 1.2 Tiny or SVG 2.

It might still be worth rewriting the SVG to output unwrapped text (possibly clipped to the bounding box?). That would handle the case of simple one line text annotations, even if it misrenders more complex ones.

With that said, maybe we should push text support off into a future PR, since what we have here seems like a big improvement over what we had before.