bluesky-social / social-app

The Bluesky Social application for Web, iOS, and Android
https://bsky.app
MIT License
11.77k stars 1.5k forks source link

Incorrectly generated video thumbnail for embeds, on portrait video with MPEG rotation #6582

Open mcclure opened 1 day ago

mcclure commented 1 day ago

Steps to Reproduce

Consider this post. Now look at the following two screenshots; the first is a screenshot of the post on bsky.social; the second is an "embed" of the post, created by clicking ⋯, choosing "Embed post", and pasting the HTML elsewhere.

Attachments

Screenshot from 2024-11-21 09-57-38

Screenshot from 2024-11-21 09-58-04

As you can see, the video appears properly on bluesky. In the embed however it is given an incorrect thumbnail. A bottom 16:9 aspect ratio slice is cut out of the 9:16 aspect ratio to produce the fixed thumbnail. This produces a bad thumbnail as the "subject" of the image is in its center.

What platform(s) does this occur on?

Web (Desktop), Web (Mobile)

Device Info

I am seeing this in a web browser, the browser is Firefox 130.0.1 on Linux.

What version of the app are you using?

1.94.0

Additional Information

This video was taken on a Sony XPeria 5 III phone, held in portrait orientation. As such the video is 1080x1920, "1080p sideways".

The problem appears to be an issue with CSS, not with the thumbnail itself. If I right-click the image and open it in a new window, i get this, and it contains all the image content. You're just somehow HTML-embedding the embed so that it displays with an incorrect aspect ratio.

In my testing, embeds of images with 9:16 aspect ratios appear properly. Therefore there is something special about either videos, or this particular video, to make bsky.app produce the bad CSS.

As it happens, there is something special about this video. I can't figure out how to download the exact copy of the video on Bluesky's server, but if I get an original copy of the video and run ffmpeg -i, I find (snipping only the video track):

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '522354ab61852ad8.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.7.100
  Duration: 00:00:15.92, start: 0.000000, bitrate: 17660 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1920x1080, 17497 kb/s, 29.96 fps, 29.97 tbr, 90k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
    Side data:
      displaymatrix: rotation of -90.00 degrees

"1920x1080… displaymatrix: rotation of -90.00 degrees". So the phone, detecting it was held sideways, took the video in 1080p, then added MPEG metadata to rotate it 90 degrees to the appropriate 1080x1920 resolution. I understand this to be a common technique for mobile phone cameras, I assume all portrait videos (and probably images) taken on my phone do this.

Hypothesis: Every part of the bluesky stack, as well as the web browser and bsky android app, are correctly understanding "1920x1080 + -90deg rotation" to mean 1080x1920, except for the thing that generates the CSS for the embed code, and this one part is getting confused and creating html that crams the 1080x1920 thumbnail into a 1920x1080 box.