search

google / spatial-media

Specifications and tools for 360º video and spatial audio.
Other
1.85k stars 428 forks source link

How to inject proper metadata to make video players interpret video as equirectangular 360° video? #188

Open Boscop opened 6 years ago

Boscop commented 6 years ago

I record 360° videos with OBS (from my opengl application that renders in equirectangular mode) but the videos just have normal video metadata (and OBS doesn't know that they are equirectangular 360°). It works when I stream to youtube (when I tell youtube to interpret the incoming video as 360°), but it doesn't work when I open the mp4 with VLC3 (which has 360° video support) because of the lacking metadata (I think). How can I inject the right metadata so that any video player such as VLC 3 interprets the video as equirectangular 360° video?

Thanks :)

campbellwmorgan commented 6 years ago

I'd also be really interested in this especially as youtube have removed the tickbox, so it's not possible to stream live to youtube in 360 without camera 360 metadata.

Boscop commented 6 years ago

Really? When did youtube remove the checkbox? So now it's not possible anymore to stream as 360° with OBS when capturing from a window that renders visuals equirectangularly? Or does OBS support injecting the 360° metadata?

Boscop commented 6 years ago

@campbellwmorgan

I just found an answer here: https://support.google.com/youtube/answer/6178631 Under "prepare for upload" it links to this: https://github.com/google/spatial-media/releases/tag/v2.0 I just tried this tool on a random video I had (not equirectangular, but just to test) and it works, now VLC 3 shows it interpreted as equirectangular 360° video: image

Now, what would be the equivalent command line arguments to inject the same metadata manually? It seems to have appended this at the end of the video file:

<?xml version="1.0"?>
<rdf:SphericalVideo xmlns:GSpherical="http://ns.google.com/videos/1.0/spherical/">
  <GSpherical:Spherical>true</GSpherical:Spherical>
  <GSpherical:Stitched>true</GSpherical:Stitched>
  <GSpherical:StitchingSoftware>Spherical Metadata Tool</GSpherical:StitchingSoftware>
  <GSpherical:ProjectionType>equirectangular</GSpherical:ProjectionType>
</rdf:SphericalVideo>

And the question remains, how to configure OBS to stream to youtube as 360°? Are you sure they removed the checkbox? It's still in the help docs: https://support.google.com/youtube/answer/6396222?hl=en

campbellwmorgan commented 6 years ago

@Boscop I've spent all yesterday looking into this and it is quite tricky. I'm debating whether it's worth me writing some software to handle this.

A few points:

  1. You linked to the v2 draft spec, but as far as I can tell from looking into the source code, the python tool in this library only implements the v1 spec which does not have support for cubemap
  2. The python code (and the spec) manipulates the boxes / atoms in the mpeg4 container (editing the video file at a binary level) to add the fields specified in addition to the xml and this would be required on the livestream.
  3. As far as I can tell, rtmp media streams use the FLV container which is slightly different to the mpeg4 container and there is nothing in the spec for that.
  4. I think that adding in this metadata will have to happen at quite a low level in the OBS Studio api (probably the C api as I think the lua/python api seems to be more for image rendering - please correct me if I'm wrong about that)
  5. Pretty sure youtube have removed the checkbox - try https://www.youtube.com/live_dashboard. I think this happened a few days ago
Boscop commented 6 years ago

@campbellwmorgan Someone else had the same issue and asked on the forum here: https://obsproject.com/forum/threads/youtube-live-360-metadata.82785/

But no answer so far :/ Maybe we should contact the OBS developer on IRC?

Boscop commented 6 years ago

I created an issue for OBS: https://obsproject.com/mantis/view.php?id=1213

Squishynoize commented 6 years ago

@Boscop Yo they never removed the checkbox. You just have to start the stream from the live control panel or that box won't show up! Names Squishy.Formally Unreality Journeys U.S Have a good one!

Boscop commented 6 years ago

@Squishynoize Thanks! How do you mean "from the live control panel", instead of from where? (I thought there is only one stream control panel.) Does it mean the checkbox is only available when the stream is already public when I'm starting to stream? In my tests I wanted to keep it private to confirm that everything is working.. (I only tested the streaming a few times to check if it will work later (when my software* is ready).)

Btw, I'm a big fan of your work :) (You may have noticed that I referenced your Youtube channel in that OBS issue.) I also like George Clanton and your collaborations with him..

* I'm currently working on a synaesthetic music+VJ software for live performances/improvisation/live-remixing (with 360° music-synced visuals inspired by vaporwave & techniques of the demoscene) that renders the virtual 3D scene (showing the post-processed webcam feed in a virtual screen in the scene) in equirectangular projection so that it can be used to record & stream full 360° music performances (in combination with OBS). The music/audio part of the software is mostly finished, I'm currently streamlining the workflow (so that it also works for multi-hour performances) and then integrating the visual components, and then I'll record 360° performances for Youtube.. (I hope they won't remove the checkbox before OBS has metadata support for this..)

utacc commented 4 years ago

Hey folks, did any of this ever go anywhere? To date, I've seen no support in OBS for spherical video, nor any way to inject the necessary metadata...?