search

Neos-Metaverse / NeosPublic

A public issue/wiki only repository for the NeosVR project
193 stars 9 forks source link

360 youtube videos not loading as 360 #2546

Open Kodufan opened 3 years ago

Kodufan commented 3 years ago

Describe the bug

When attempting to load a 360 youtube video, the resulting loaded video will appear as if the view is locked and rendered from the default 2d viewpoint when loading the video on desktop youtube.

Relevant issues

N/A

To Reproduce

Paste this video into Neos as a 360 video. The video will fail to load the complete 360 scene and instead only display one fixed 2D viewpoint. If I were to guess, this is likely a change youtube or youtube DL has made to requesting and loading 360 videos so they may be played on a single 2D display, and will need to be changed so that Neos loads the entire, unwrapped 360 scene,

Expected behavior

The video to load the entire 360 video.

Log Files

N/A

Screenshots / Video

https://imgur.com/a/80vkbzL This should be an unwrapped 360 video, meaning heavy distortion to make the entire 360 view fit in a flat plane. Instead, it is the default viewpoint rendered in 2D as you'd see on youtube,

Bug information (please complete the following information):

Reporters:

Kodufan#7558

Frooxius commented 3 years ago

I've looked into this, but it seems that it's unfortunately what YouTube actually serves in the streams - a pre-rendered 2D preview - you'll get the same behavior if you try to load the YouTube URL directly in VLC.

It seems that the actual 360 is available only in the higher resolution streams, but those do not contain the audio stream and would need to be muxed on Neos' side, which is a bit complicated right now.

The 360 stream also isn't in a format that Neos currently supports anyways: https://github.com/Neos-Metaverse/NeosPublic/issues/1442

I'd recommend uploading 360 videos rendered in equirectangular directly to Neos if you want reliable playback, this kind of thing isn't officially supported by YouTube and is prone to breakage or compatibility problems.

Until we have a way to determine that some streams are 360 and should be picked over the 2D ones through YoutubeDL and have video playback that allows muxing separate audio and video streams, we unfortunately can't do much on our end.