360 video support

[D4rk4]

If we want build concurrent video platform in 201x, I think we should include 360 video support to player.

---

Roadmap:

• [x] federate the 3D/360 metadata gathered server-side, as it's hard to parse client-side
• [x] add a pre-transcoding step gathering information about the video
• [ ] make the video transcoding keep the video in a playable form by videojs-vr

[DavidLibeau]

Suggesting we use Aframe (MIT) by Mozzilla. Or maybe lower level tech, like Three.js (MIT) ? If I understand something about the PeerTube code, I will try to implement it.

[Chocobozzz]

We use VideoJS for the player so I think we could use https://github.com/videojs/videojs-vr (it seems they use three.js).

[andreaspeters]

When the support will be coming?

[rigelk]

@DavidLibeau @Chocobozzz after some tests using the sample videos in videojs/videojs-vr, I can tell it works fine.

BUT. But we need to add a way to detect if a video is a 360° video, and pass it to the player to activate the plugin. Otherwise it messes a lot with regular videos.

Hopefully now that we've meddled a bit with FFmpeg that shouldn't impress us, right?

There is an a Google RFC (not even W3C or IETF, but they don't care…) for Metadata to describe spherical data in MP4 videos.

This includes the introduction of a new spherical video header box, svhd, which we could supposedly test for the presence of to detect if a video is a VR 360 one.

Now, even YouTube has guidelines to upload 360 videos (and they recommend a tool only available on Windows and Mac :roll_eyes: ), and they apparently require that kind of added metadata to allow them to detect the 360 videos.

[DavidLibeau]

Yep, Google RFC is something to handle in PeerTube, but I think it could be cool to add a way to let the user manually say that it is an equirectangular video, and to add it in the API/Activity Pub. Maybe, the simplest way might be to specify a fixed tag?

In the same time, as a University work, I actually am building a WebVr client, as seems there : https://framagit.org/DavidLibeau/peertube-vr-player (but this project is not in conflict with this feature, as the idea is to propose a way to navigate in PeerTube without leaving the headset).

[rigelk]

Soon enough we will have our own RFCs :smirk:

@DavidLibeau that's impressive!

[andreaspeters]

Well, u already using ffprobe to get out metadata to the videos. It's not a bad idea to add a new function to get the spherical: equirectangular information of a video. And by the way, the recommended tool is working quite well under linux. :-) Just start it with python2 gui.py.

[rigelk]

@andreaspeters could you write a POC getting that metadata using ffmpeg/ffprobe?

[rigelk]

Okay, so after playing a bit with the CLI I got a gist of what the output might look like for a video complying with Google's standard.

ffprobe -v quiet -show_streams <filename> -of json | jq '.. |."side_data_list"? | select(. != null)'

[
  {
    "side_data_type": "Stereo 3D",
    "type": "side by side",
    "inverted": 0
  },
  {
    "side_data_type": "Spherical Mapping",
    "projection": "equirectangular",
    "yaw": 0,
    "pitch": 0,
    "roll": 0
  }
]

[andreaspeters]

ok, I was to slow. :-) I just wrote:

var ffmpeg = require('fluent-ffmpeg');

ffmpeg.ffprobe('test.mp4',function(err, metadata) {
  console.log(require('util').inspect(metadata, false, null));
});

And got as result a nice json:

   [ { index: 0,
       codec_name: 'h264',
       codec_long_name: 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10',
       profile: 'High',
       codec_type: 'video',
       codec_time_base: '1/120',
       codec_tag_string: 'avc1',
       codec_tag: '0x31637661',
       width: 2176,
       height: 1088,
       coded_width: 2176,
       coded_height: 1088,
       has_b_frames: 2,
       sample_aspect_ratio: 'N/A',
       display_aspect_ratio: 'N/A',
       pix_fmt: 'yuv420p',
       level: 50,
       color_range: 'unknown',
       color_space: 'unknown',
       color_transfer: 'unknown',
       color_primaries: 'unknown',
       chroma_location: 'left',
       field_order: 'unknown',
       timecode: 'N/A',
       refs: 1,
       is_avc: 'true',
       nal_length_size: 4,
       id: 'N/A',
       r_frame_rate: '60/1',
       avg_frame_rate: '60/1',
       time_base: '1/15360',
       start_pts: 0,
       start_time: 0,
       duration_ts: 1317120,
       duration: 85.75,
       bit_rate: 6336491,
       max_bit_rate: 'N/A',
       bits_per_raw_sample: 8,
       nb_frames: 5145,
       nb_read_frames: 'N/A',
       nb_read_packets: 'N/A',
       side_data_type: 'Spherical Mapping',
       projection: 'equirectangular',
       yaw: 0,
       pitch: 0,
       roll: 0,
       tags: { language: 'und', handler_name: 'VideoHandler' },
       disposition: 
        { default: 1,
          dub: 0,
          original: 0,
          comment: 0,
          lyrics: 0,
          karaoke: 0,
          forced: 0,
          hearing_impaired: 0,
          visual_impaired: 0,
          clean_effects: 0,
          attached_pic: 0,
          timed_thumbnails: 0 } },
     { index: 1,
       codec_name: 'aac',
       codec_long_name: 'AAC (Advanced Audio Coding)',
       profile: 'LC',
       codec_type: 'audio',
       codec_time_base: '1/44100',
       codec_tag_string: 'mp4a',
       codec_tag: '0x6134706d',
       sample_fmt: 'fltp',
       sample_rate: 44100,
       channels: 2,
       channel_layout: 'stereo',
       bits_per_sample: 0,
       id: 'N/A',
       r_frame_rate: '0/0',
       avg_frame_rate: '0/0',
       time_base: '1/44100',
       start_pts: 0,
       start_time: 0,
       duration_ts: 3805169,
       duration: 86.285011,
       bit_rate: 128545,
       max_bit_rate: 128545,
       bits_per_raw_sample: 'N/A',
       nb_frames: 3717,
       nb_read_frames: 'N/A',
       nb_read_packets: 'N/A',
       tags: { language: 'und', handler_name: 'SoundHandler' },
       disposition: 
        { default: 1,
          dub: 0,
          original: 0,
          comment: 0,
          lyrics: 0,
          karaoke: 0,
          forced: 0,
          hearing_impaired: 0,
          visual_impaired: 0,
          clean_effects: 0,
          attached_pic: 0,
          timed_thumbnails: 0 } } ],
  format: 
   { filename: 'test.mp4',
     nb_streams: 2,
     nb_programs: 0,
     format_name: 'mov,mp4,m4a,3gp,3g2,mj2',
     format_long_name: 'QuickTime / MOV',
     start_time: 0,
     duration: 86.309,
     size: 69446824,
     bit_rate: 6437041,
     probe_score: 100,
     tags: 
      { major_brand: 'isom',
        minor_version: '512',
        compatible_brands: 'isomiso2avc1mp41',
        encoder: 'Lavf58.12.100' } },
  chapters: [] }

So... If you could create a tag with a 360degree version of peertube, then I will build a dockerimage with it to try it on my peertube instance. I already have a 360degree video there. :-) I'm not a nodejs guy, sorry. :-(

[rigelk]

Ah, strangely your side data are not contained in a list as was the case for mine.

[andreaspeters]

interesting! I used the google tool to add the side data! My way is to take the videos from the camera, transform them to a format that peertube is expected, and then add the side data info again. Because these data are missing after the transform.

[rigelk]

It's probably missing because FFmpeg only cares about that kind of side data when using -strict unofficial, as per https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/movenc.c#L2069-L2077

[andreaspeters]

ah cool, good to know thanks. :-) so if peertube trancode, is also should use these flag.

[rigelk]

Just to let you know, I have implemented the ground work necessary to play 360 videos using videojs-vr in MR!17, but am blocked by the following:

It is now in working state, except that the transcoding produces videos that are not compatible with the plugin somehow and have glitches. But playing the original video after the upload and before the transcoding ends works fine.

Either the transcoding messes the file, or this is a problem with the videojs-vr plugin. In any case, I would need help on that one.

[andreaspeters]

I'm using this flags to transcode offline:

ffmpeg -i video.mp4 -y -vcodec libx264 -threads 8 -framerate 60 -movflags faststart -strict unofficial "video_trans.mp4"

In peertube I deactivate the transcoding. Its because of my to slowly s3 storage. After the transcoding peertube got a error and the video was deleted. But thats a other case! So... Maybe the peertube transcoding missing the strict parameter?

[rigelk]

@andreaspeters no.

Plus, before even caring about the transcoding within PeerTube, I just tested transcoding the official example video of videojs-vr manually with roughly the same options you mentioned, and uploading it. Uploading the original video works and plays in PeerTube with videojs-vr. Uploading the transcoded video doesn't.

[andreaspeters]

did u had a look with ffprobe whats the difference between these files?

[andreaspeters]

@rigelk ok I think u are angry because of my answer. Sorry for that. :-) Is there a way that u send me the video after the transcode, therefore I can try to investigate?

[rigelk]

@andreaspeters no no no, I'm not angry :slightly_smiling_face:

You can access the gist of my tests here. Here is a paste of the README if put there:

---

Basic test

First and foremost, download the original video here. I'm refering to it as eagle-360-in.mp4 thereafter.

To shorten the test, I'm just transcoding 2 seconds via -t 2:

ffmpeg -i eagle-360-in.mp4 -strict unofficial -c:a copy -c:v libx264 -t 2 eagle-360-out.mp4

After that, you can check the presence (or rather, absence) of the 360 metadata on the resulting video:

ffprobe -v quiet -show_streams eagle-360-out.mp4 -of json | jq '.. |."side_data_list"? | select(. != null)'

But trying with the original video shows that it lacks metadata too, so this clearly isn't a determining factor to check if a video can or cannot be played by videojs-vr.

Adding metadata

Use spatial-media to detect and inject metadata:

$ python spatialmedia eagle-360.mp4                        spatial-media/git/master 
Processing: /home/rigelk/Vidéos/eagle-360-in.mp4
Loaded file...
    Track 0
    Track 1
$ python spatialmedia -i --stereo=left-right eagle-360-in.mp4 eagle-360-in-with-metadata.mp4
Processing: /home/rigelk/Vidéos/eagle-360.mp4
Saved file settings
    Track 0
        Spherical = true
        Stitched = true
        StitchingSoftware = Spherical Metadata Tool
        ProjectionType = equirectangular
        StereoMode = left-right
    Track 1

Checking for further differences

At that point, metadata is useful for us to enable/disable the videojs-vr plugin and not mess with normal videos, but it clearly isn't the determining factor to check if a video can be played by the plugin.

So what is?

[andreaspeters]

Thanks for your answer. Good to know that u was not angry. :-) Well, like u wrote, the original video don't have the spherical: equirectangula meta information inside and vlc also don't recognise it as 360 degree video. I have a 360 camera and all (of course) of that videos have this meta information inside. The question is, is the original video you use a reliable source for testing? Of course, I have no idea how about other 360 cameras, and so long there is no standard its not possible to be sure that every video will work. But maybe its a good way to start. In worst case, the only thing can happen is, that a 360degree video is not playable or looks like a normal video. Am I right?

[rigelk]

Well, the original video is one of the example files given by the videojs-vr plugin itself… And it does work. So it sounds reliable - at least that way we can exchange with the videojs-vr folks if need be.

You can see all their video samples here.

[andreaspeters]

Ok, I clone the videojs-vr repo with the fluid.html example. For me, its only working if I change the projection from auto to equirectangular (browser is Firefox under Arch Linux). When I'm using one of my 360degree videos, its working also with AUTO parameter. With working I mean, the video is playing and I can move the camera position. For me it looks like, the example videos are not 360degree. But with the projection parameter we can force the player to ignore it. That means, with a video that have the spherical: equirectangula metadata and AUTO as projection parameter, should work the plugin very well.

[rigelk]

@andreaspeters could you try modifying my MR to see if your changes work?

[andreaspeters]

I will try, if u give me time until monday. :-)

[rigelk]

@andreaspeters of course :) I'm asking because I don't have so much time in the following two weeks, so take your time

[andreaspeters]

well, I changed the code and try to test it. but just get this error:

aventer.biz:443] 2018-08-20 13:55:41.271 info: Begin migrations.
[aventer.biz:443] 2018-08-20 13:55:41.280 info: Executing 0245-video-360.js migration script.
[aventer.biz:443] 2018-08-20 13:55:51.339 error: Cannot execute migration 0245. {
  "err": {
    "stack": "TimeoutError: ResourceRequest timed out\n    at ResourceRequest._fireTimeout (/home/node/peertube/node_modules/generic-pool/lib/ResourceRequest.js:62:17)\n    at Timeout.bound (/home/node/peertube/node_modules/generic-pool/lib/ResourceRequest.js:8:15)\n    at ontimeout (timers.js:424:11)\n    at tryOnTimeout (timers.js:288:5)\n    at listOnTimeout (timers.js:251:5)\n    at Timer.processTimers (timers.js:211:10)",
    "message": "ResourceRequest timed out",
    "name": "TimeoutError"
  }
}
npm ERR! code ELIFECYCLE
npm ERR! errno 255
npm ERR! peertube@1.0.0-beta.10.pre.3 start: `node dist/server`
npm ERR! Exit status 255
npm ERR! 
npm ERR! Failed at the peertube@1.0.0-beta.10.pre.3 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/node/.npm/_logs/2018-08-20T13_55_51_373Z-debug.log

Dont know what nodejs want from me. :thinking:

[rigelk]

@andreaspeters it's the migration which fails. Since I had to modify the video table, I wrote migration 245: 0245-video-360.js. You should check what's wrong with it and clashes with your current migration number and database.

[rigelk]

@andreaspeters are you still stuck?

[andreaspeters]

@rigelk Sorry, I'm on travelling. :-) Will go forward when I'm back at weekend.

[DavidLibeau]

Any updates (or not) on this? :smile:

[Utopiah]

Any updates (or not) on this?

Is "make the video transcoding keep the video in a playable form by videojs-vr" blocking if so what more precisely?

[Chocobozzz]

Hello,

We don't have time to implement 360 video support. But you can create a plugin to support it. If you need specific hooks, please create dedicated issues.

See the documentation for more information: https://docs.joinpeertube.org/#/contribute-plugins

[Utopiah]

Thanks for clarifying. If anybody starts a plug-in please link here so that we might help each other.

[despens]

Hello folks, I was wondering if there was any development talent available for completing work on supporting 360 video, either as a core peertube functionality or as a plugin.

Background: I work for a non-profit organization that commissioned artists to create 360 videos that would play in a mobile app. The company that created the app disappeared (I know, it happens a lot). We're running our own peertube instance and would like to showcase these videos there, esp. after facing artist videos being removed from mainstream platforms.

So I'd be interested to learn what a possible timeline and budget requirements for adding 360 video support to peertube could be, we might be interested in sponsoring the development.

[Utopiah]

Here is a 10 line example :

<html>
  <head>
    <script src="https://aframe.io/releases/1.2.0/aframe.min.js"></script>
  </head>
  <body>
  <a-scene>
    <a-assets>
      <video id="video" autoplay loop="true" 
            src="https://video.benetou.fr/download/videos/44851d25-b851-4307-b580-b4166466d20f-960.mp4">
      </video>
    </a-assets> 
    <a-videosphere src="#video"></a-videosphere> 
  </a-scene>
  </body>
</html>

that you can try on Glitch https://peertube3630.glitch.me then remix.

This is very basic and doesn't rely on the player https://docs.joinpeertube.org/contribute-architecture?id=client-code but it could be a quick start by providing the video URL as a parameter. It requires some relatively easy adjustments in order to work well on mobile (which doesn't accept autoplay). Videos with a tag, e.g 360, could then via a plugin play with this basic integration instead. Integrating the player for peer to peer and adaptive streaming support (rather than now getting the direct download source link from https://video.benetou.fr/videos/watch/44851d25-b851-4307-b580-b4166466d20f ) is more demanding but is the "proper" way IMHO.

[despens]

Thanks for the example via the a-frame player, @Utopiah! Yes it would be preferable of course to have the 360° player well-integrated into peertube, with support for the most common stereoscopic video formats and access modes (like Google cardboard etc). Probably the complete a-frame framework would be overkill in this case.

Is there an idea what steps are missing from today's perspective to support the complete cycle including uploading, transcoding, publishing, and accessing?

I'll work on convening a group of conservators in 2022 to discuss some ideas. Everyone, please get in touch with me if you'd like to join. (My email is public in my GitHub profile.)

[rigelk]

My initial stab at this feature was lacking a formal and well defined list of formats/standards to support, as well as test 360 source files to validate the feature. In particular, I had trouble transcoding the file without losing the 360 nature of the file.

Just providing the list, the corresponding source files, and the valid ffmpeg options would go a long way to making the issue tracking progress.

[Utopiah]

Is there an idea what steps are missing from today's perspective to support the complete cycle including uploading, transcoding, publishing, and accessing?

IMHO only the player. If you check my example I uploaded the video to an unmodified PeerTube instance and was able to play it, as-is.

losing the 360 nature of the file.

I don't think that exists. There might be meta-data and I imagine some standards for that but I doubt it's widely supported. That's also usually why most if not all players provide a manual option to change the format e.g 360 vs 180 or stereoscoptic vs mono or top/down versus side by side in stereoscopic. That's why I mentioned the tag as a an option to manually to do and outside of the video itself.

[rigelk]

@Utopiah did you test your solution on a video that passed through transcoding by PeerTube?

[DavidLibeau]

I don't think that exists.

The specification exists : https://github.com/google/spatial-media. But I think we don't know really how to implement it with FFMPEG in the PeerTube processes.

The Minecraft ReplayMod is recording 360 videos with the metadata (https://github.com/ReplayMod/ReplayMod). Worth checking how they do. (Example of video: https://www.youtube.com/watch?v=swEtPJIX5IY)

[Utopiah]

@DavidLibeau FWIW specifications aren't standards. Maybe the MPEG group has standards but 1 implementation for 1 vendor, even though useful, might show for a bias. Also not sure how pragmatic that is as Google doesn't make commercially available cameras anymore so arguably I'd look at what e.g Theta is outputting instead.

@rigelk I have a Theta V and a PeerTube 4 instance running so if it's useful I can run some tests. Point being though that a player like AFrame that I shared takes a video then project it on a sphere. It doesn't care much for anything else so you could take any format or container, even with the wrong ratio, and it would still project it.

[fenarinarsa]

Dropping a note here. I'm really interested in having VR support in Peertube because, basically, the number of online video services that support VR videos is near zero. YouTube does it but... there is no practical web VR player and the app is the worst I've ever seen. Actually the best VR support so far is from a famous adult video site. This is really a shame since the VR headset market keeps expanding, thanks to (or because of) Meta that sell their cheap Oculus Quest.

Now I see that the only kind of video you're talking about is 360 videos. This is not the most used VR format (far from it). The format you see the most now is stereoscopic VR180 video, in 4k+. It's easier to shoot and more immersive. There are now whole channels of those videos on YT because there are more and more VR180 capable cameras out there, and Canon just released an amazing VR180 stereo lens for its latest cameras, so be sure there will be more in the future. But as I said, currently the easiest way to watch videos is to download them, which is the least practical way of browsing videos online.

Unfortunately I'm not a frontend dev so I wouldn't be able to help (at least not in a short timeframe). I just wanted to point out the current VR usage.

Here's an example of YT channel with a lot of VR180 3D videos: https://www.youtube.com/channel/UCDsaUsbzJgNJ-zLJp3kluxA

[despens]

Thank you for expressing support, @fenarinarsa. I think the term "360° video" is a sloppy catch all term for all types of stereoscopic video. If this project would move along it would make sense to survey what exact formats and features stakeholders would want to prioritize. For instance in the field of art you're very often dealing with 3D rendered environments which are not coming out of a camera but out of game engines or animation software; active surfaces in space that act as links in between different videos are also popular. But as a start it would also be valuable to gain basic support and importing/transcoding functions.

[fenarinarsa]

Importing and transcoding is not very complex - basically it's 2:1 ratio videos and the content use a spatial projection (most often equirectangular). They can be transcoded exactly like regular videos. Highest resolutions might be needed, and better compression settings - the last few weeks I tried to import some videos from YouTube and Peertube's compression is really not as good as YT's (even from h264 to h264). But it's just a matter of settings.

Of course some metadata should be added (like: VR true/false, projection type, 180/360, etc). This is no big deal either.

But supporting VR playback means a VR compatible player, for flat screen and VR headsets. And that's the thing that would take the most time to develop. I know Firefox has a builtin VR support, I don't know for other browsers.

[Utopiah]

Mozilla decided to focus VR support only in Mozilla Hubs, their social network. WebXR support in Firefox on standalone is not maintained anymore (see the https://github.com/MozillaReality/FirefoxReality ) and the lead developer on Firefox itself for WebXR left for another company. Consequently I wouldn't make decision based on that, unfortunately.

For the most popular standalone device, the Quest 2 by Meta, the browser is chromium based. I believe it does support 360 playback natively but I can check with a Quest 1, if need be.

Honestly I think complete support of all formats is a great goal but is a mistake to make it a requirement. I believe starting with monocular 360 is sufficient to make it useful, get feedback from both content producer and viewers and then grow support to more format. I don't have statistics of usage but as most 360 consumer cameras are monocular it means at least for a non professional audience who might produce a lot of content it already becomes interesting.

In terms of playback support I did share an earlier example of a framework based solution, namely AFrameVR. It is based on threejs and actively community maintained https://github.com/aframevr/aframe/

Edit: Igalia is reviving FirefoxReality https://github.com/Igalia/wolvic

[fenarinarsa]

And I, on the other hand, think supporting only one format is a huge mistake too :) 360 is basically a subset of 180 and the videos are exactly the same format (2:1 equirectangular most of time). If you can support one, you can support the other. Also as I said above, Canon released his 180 lens last year and it's a game changer, since this lens is going to be compatible with their following cameras - and indeed, their latest camera released this month is also VR compatible. You don't have statistics and I don't have any either, the only thing I see is that most of groups of VR video enthusiasts has been switch to VR180 for some time now. That's why some cheap and correct cameras like the Insta360 Evo (no longer produced) are in high demand.

So if someday peertube adds support for a VR format (be it 180 or 360), you can be certain that in the following hours you'll get a lot of requests to support the other one. Best to add every format metadata in the backend in one shot :)

[Utopiah]

I'm not advocating to support only 1 format but rather to start with one, the most popular one, rather than wait until 2, or all formats are supported because I worry this will never be the case. That being said obviously if someone were to implement this and prefer one format over another or could trivially support both or even all format, that'd be amazing. The example I shared before https://github.com/Chocobozzz/PeerTube/issues/119#issuecomment-998039130 was for 360 because my content is 360 and AFrame, the framework used https://aframe.io/docs/1.2.0/primitives/a-videosphere.html supports these. I'm not advocating to limit to this.

[sawine]

Hello,

Simply bumping this to say I would also really appreciate this implementation to PeerTube!

[Utopiah]

Please try https://github.com/jerferbe/peertube-plugin-videojs-xr and let us, me and a colleague, what you think of it.