search

morsh / unity-drones-multiplayer

Enable multiple players to connect to the same enrironment (UNET), fly a drone, and watch others fly their drones
MIT License
36 stars 26 forks source link

Integrate 3dtoolkit into desert scene #2

Open anderm opened 6 years ago

bengreenier commented 6 years ago

I made a ton of progress on this today see https://github.com/bengreenier/unity-drones-multiplayer/tree/3dtoolkit-desert

morsh commented 6 years ago

Guys, looks like a lot of work, when you guys have come back online, let's have a sync over those things :) @bengreenier, @anderm

bengreenier commented 6 years ago

Sure! Since syncs might be tricky I'll try to keep a better log of what i'm doing here. 😸

So far, the goal is to enable the desert scene to stream, and support multi-peers. This uses our pending 3dtoolkit multi-peer work. This requires some changes to the mental model, as the server and the desert experiences effectively merge. This is because when we're streaming, the server is running the core experience on behalf of all users, and is just streaming video and accepting input from a bunch of "dumb" clients. per #4 we're going to modify one of our existing clients to support MR and input from an xbox controller - so that will be our "dumb" client.

At this point, I have the core 3dtoolkit plugin behaving correctly in the desert scene, and my focus will now shift to ensuring the streaming video works as expected (not always as trivial as it sounds, unfortunately). Then I'll be making some changes to enable managing the lifecycle of all the drones in the desert as 3dtoolkit connections get created/removed. I'll try to keep all my work prefab managed, so we can avoid scene merges and make it easier to enable this scenario going forward.

bengreenier commented 6 years ago

f9ba598936fa26b0cf14261383bba13cfca53923 supports streaming! it needs a flip fix (the content is upside down in a mono feed) but it's functional and mouse controls enable movement for a single client.

image

bengreenier commented 6 years ago

note: this required me to change your project default resolution to our supported streaming resolution (currently there isn't flexibility here).

image

bengreenier commented 6 years ago

https://github.com/bengreenier/unity-drones-multiplayer/commit/9213e07acf0d18f9682073b97f845d17c3505438 flips the render, so we get the following client rendering now:

image

bengreenier commented 6 years ago

Tried to modify webrtc to run at 120fps (or really, any arbitrary fps) but am stuck there. It's quite complicated to update all the webrtc plumbing. I believe I modified all the transmission framerates to run at 120, but i am only receiving ~50fps still. I'm investigating receiving framerate logic to see if that's also a blocker, but i can't really verify that my transmission is running at 120 without a receiver so i'm unable to verify where the issue really lies.

This will be very tricky to unblock, I'm not confident i'll have this working by Wednesday.

tylergibson commented 6 years ago

webrtc currently caps out at 60fps. There are a ton of places, especially in the video jitter engine that aren't really plumbed to handle greater than 60fps transmission. If we really need >60fps, then we would be better off using a custom data channel and sending video frames through it, bypassing the video engine entirely. You also need to ensure Unity has vsync disabled and the host system has vsync disabled.

We also need to make sure in our DirectX samples that we disable vsync on present

bengreenier commented 6 years ago

thanks @tylergibson, that makes sense. The work mentioned above was to investigate the cost of modifying those places in webrtc that do cap out at 60fps. As you correctly assumed, and we determined, that work is non-trivial ☹️