search

bevyengine / bevy

A refreshingly simple data-driven game engine built in Rust
https://bevyengine.org
Apache License 2.0
35.12k stars 3.45k forks source link

iOS Support #87

Closed cart closed 2 years ago

cart commented 4 years ago

It should be possible to run Bevy Apps on iOS

MichaelHills commented 4 years ago

It looks like the window is fixed at 1280x720, happened to notice that when logging touch event x/y coordinates (adding virtual joysticks to my game). Haven't looked into where it's coming from, I suspect winit perhaps.

CleanCut commented 4 years ago

Hmm, I don't see anything in winit, but this part of bevy_window seems suspicious:

https://github.com/bevyengine/bevy/blob/612c2552a50e8009715e825504b9e8b74b56821e/crates/bevy_window/src/window.rs#L87-L99

MichaelHills commented 4 years ago

@CleanCut haha suspicious indeed. I'll check my code tonight, maybe I configured the window wrong.

MichaelHills commented 4 years ago

Thanks, was just my config. My code was like this, when I moved WindowDescriptor to be before the default plugins my settings actually took effect. I still can't get it to just "fullscreen" cleanly and map to the edges of the screen. Now things are a bit vertically stretched and the window appears to extend past the right and bottom of my phone. Will keep investigating.

     App::build()
         .add_default_plugins()
         .add_resource(WindowDescriptor {

Edit: The window is the correct size, and the touch events x/y confirm it. Maybe something wrong with the viewport... not sure.

Edit 2: I can fix it with this... the window size is 1334 by 750 though.

wgpu_render_pass.set_viewport(0.0, 0.0, 1112.0, 563.0, 0.0, 1.0);
MichaelHills commented 4 years ago

Argh nevermind, I had typo'd my ios vs not-ios cfgs and I had #[cfg(target = "ios")] not #[cfg(target_os = "ios")] and so it was doing the opposite of what I wanted.

It meant I was passing 1600x1000 to iOS, which still somehow created a window of size 1334x750 as per windows.get_primary().unwrap(); but when I logged winit's ScaleFactorChanged event it logged

ScaleFactorChanged(2, w=1600 h=1000)

so that's how I knew I messed up. Anyway after fixing this issue, I now have another problem. I have defaulted my game in XCode to landscape-only. When I do fullscreen, it's choosing a portrait mode window size of 750x1334 even though it's already in landscope mode.

MichaelHills commented 4 years ago

Finally figured it out. I think there's an issue in winit where if you start the iOS app in landscape, the view frame doesn't match the window bounds (view in portrait, window in landscape), so in layoutSubviews just need to detect this and update the frame. Now if you start in either portrait or landscape, and rotate the phone, everything works fine.

diff --git a/src/platform_impl/ios/view.rs b/src/platform_impl/ios/view.rs
index 5481b8ef..da195a0a 100644
--- a/src/platform_impl/ios/view.rs
+++ b/src/platform_impl/ios/view.rs
@@ -134,6 +134,18 @@ unsafe fn get_view_class(root_view_class: &'static Class) -> &'static Class {
                     height: screen_frame.size.height as f64,
                 }
                 .to_physical(scale_factor.into());
+
+                // If the app is started in landscape, the view frame and window bounds can be mismatched.
+                // The view frame will be in portrait and the window bounds in landscape. So apply the
+                // window bounds to the view frame to make it consistent.
+                let view_frame: CGRect = msg_send![object, frame];
+                let window_bounds: CGRect = msg_send![window, bounds];
+                if view_frame.size.width != window_bounds.size.width
+                    || view_frame.size.height != window_bounds.size.height
+                {
+                    let () = msg_send!(object, setFrame: window_bounds);
+                }
+
                 app_state::handle_nonuser_event(EventWrapper::StaticEvent(Event::WindowEvent {
                     window_id: RootWindowId(window.into()),
                     event: WindowEvent::Resized(size),
MichaelHills commented 4 years ago

@cart I'm assuming if I get this landed https://github.com/rust-windowing/winit/pull/1703 that I'll need you to also update cart-tmp-winit? I couldn't find where that repo was.

cart commented 4 years ago

Yeah if we need to consume upstream master winit, then we can update cart-tmp-winit. Currently theres no public repo for it, but I can definitely put one up as soon as we need it.

I would like to eventually move back upstream though. Maybe we can just ask them to cut a new version :smile:

MichaelHills commented 3 years ago

Audio might be a bit of a hurdle. Rodio uses cpal uses coreaudio-rs uses coreaudio-sys and the latter don't yet work on iOS.

Relevant PRs to make it work on iOS but based on the first one it looks like it's never been in a working state?

Over at cpal someone started but abandoned an OpenAL backend https://github.com/RustAudio/cpal/issues/224

Apple provides an OpenAL implementation but apparently it's been deprecated in macOS Catalina, so maybe iOS is next.

The OpenAL framework is deprecated and remains present for compatibility purposes. Transition to AVAudioEngine for spatial audio functionality.

Perhaps an AVAudioEngine backend to cpal would be the way forward?

Edit: What am I talking about, finishing off the CoreAudio stuff is probably the quickest route.

simlay commented 3 years ago

On the subtopic of Audio:

Relevant PRs to make it work on iOS but based on the first one it looks like it's never been in a working state?

I think the README on cpal is/has been incorrect. I stumbled into coreaudio-rs/sys and thought adding those PRs at it seemed like low hanging fruit. I ended up spending months slowly adding better objective-c support to rust bindgen. I think it's one PR merge short of being pretty useful.

This means that the bindings for coreaudio-sys will look a bit different for iOS targets than they do macOS if you want to use the objective-c features of bindgen (I think you might have to). One annoying thing about coreaudio-rs is that it rexports which is then used in cpal. So it seems that it might need a bit of work.

Funny enough, I actually don't know much about the CoreAudio framework or the internals of cpal well enough to actually implement this without a lot of learning. Given enough time, I'll probably get to it but if someone wants to take the reins, I'm all for it.

MichaelHills commented 3 years ago

Thanks for the hard work on iOS @simlay I've seen your PRs all over the place. :) Including in cmake-rs unfortunately got reverted :( https://github.com/alexcrichton/cmake-rs/issues/95 I tried to spark some discussion but didn't go anywhere. I think for Bevy at least, once naga is ready we can drop shaderc-rs and then I think the dependency on cmake-rs goes away (for now).

Back to CoreAudio, I imagine the differences wouldn't be that major between macOS and iOS. I think in cpal we could get away with some target_os switches where macOS and iOS diverge?

I might have a go, but I'm not sure where's the best place to start. If I were to try, do I need the 3 of your branches? rust-bindgen, coreaudio-rs, coreaudio-sys? Why did you need to make changes to rust-bindgen for iOS CoreAudio?

simlay commented 3 years ago

Including in cmake-rs unfortunately got reverted :( alexcrichton/cmake-rs#95 I tried to spark some discussion but didn't go anywhere.

I'd like to eventually work on this again. It's valuable for a bunch of iOS projects.

I might have a go, but I'm not sure where's the best place to start.

Yeah! @MichaelHills, that would be awesome! There are few rust iOS folks and I'd like there to be more! You can find me on discord in the Bevy server (is that what they're called?) along with most of the other rust discords as simlay#3120.

If I were to try, do I need the 3 of your branches? rust-bindgen, coreaudio-rs, coreaudio-sys?

Right now, I think so. I'm all for a better work flow.

Why did you need to make changes to rust-bindgen for iOS CoreAudio?

CoreAudio for iOS has a bunch of objective-c in the headers and so bindgen (which uses clang to get the AST from the headers), and so just to get it to compile it required adding -x objective-c as a clang parameter to bindgen, this resulted in an issue with the objective-c generics not generating working rust.

After that, I slowly added to make all the bindings more ergonomic:

It's possible I might have gone a bit far down the rabbit hole but these are still cool and useful features if you ask me.

I've spent so much time trying to get the objective-c Nullability stuff working but I've not had a ton of luck. :/.

I'm not going to claim to actually know enough CoreAudio to tell you that these features really help you all that much. For all I know, you could blacklist all of the objective-c and you'd be left with some bindings that are quicker to generate.

I don't know your knowledge level for iOS development stuff but I think making an example iOS project for coreaudio-sys or coreaudio-rs using xcodegen would be useful. I did something like this for uikit-sys. There's also some useful things in a blogpost I wrote about using uikit-sys as well.

Eventually, using cargo-dinghy in CI would also be a nice touch.

Anyway, I hope this is useful and gives some perspective.

MichaelHills commented 3 years ago

Success! I managed to play a wav directly with Rodeo. The API has changed a bit in Rodeo/CPAL latest versions compared to where bevy is at, so I don't know if something is broken with that integration now on the newer versions. (Short story is that the cpal stream seems to not be retained and gets dropped from memory and so the mixer weak reference is null.)

Here's what I did

This might take a while to clean up the hacks. i.e. no mic input, and no device enumeration atm. There is a bunch of shared code with the macos coreaudio. The existing enumeration is macOS-specific. It might be hard to untangle this stuff.

I guess I'm initially thinking splitting up the cpal code from mod.rs / enumeration.rs to something like:

Then use target_os switch to pull in the correct implementation.

MichaelHills commented 3 years ago

I applied the audio_output.rs patch from here https://github.com/bevyengine/bevy/pull/310 and now I can finally play audio through Bevy and iOS. :)

MichaelHills commented 3 years ago

An update on where things are at:

Once touch and audio are working, I think we can call it as Bevy having iOS support? There might be bugs here and there and we can fix them as we go.

cart commented 3 years ago

334 seems to have stalled. I posted my touch support diffs earlier in this thread if you scroll up, that's what I'm currently using. If someone wants to look at that PR and look at my diffs and take over touch support that'd be helpful.

Lol I can't find the diffs you're talking about because apparently my brain doesn't work :smile: I think I'll address my own comments in that pr, maybe apply your diff, then merge it. Alternatively i could address my comments and you could create a follow up pr (that way you get proper attribution).

Once touch and audio are working, I think we can call it as Bevy having iOS support? There might be bugs here and there and we can fix them as we go.

Yup that sounds good to me! I'm so happy that this came together so quickly :heart:

MichaelHills commented 3 years ago

Lol I can't find the diffs you're talking about because apparently my brain doesn't work 😄 I think I'll address my own comments in that pr, maybe apply your diff, then merge it. Alternatively i could address my comments and you could create a follow up pr (that way you get proper attribution).

I had implemented my own touch support in parallel to that PR because I didn't know about that PR. My comment is buried further up this page https://github.com/bevyengine/bevy/issues/87#issuecomment-683432609 (had to click load hidden items).

In my version I had the following and avoided using Input since it didn't map well. However, in my game code I never used the just_*. I just used active_touches to power my on-screen virtual joysticks and local system state to keep track of which touch id was which joystick. Perhaps the just_* stuff would be useful for UI buttons and things like that.

#[derive(Default)]
pub struct TouchInputState {
    pub active_touches: HashMap<u64, ActiveTouch>,
    pub just_pressed: HashSet<u64>,
    pub just_released: HashSet<u64>,
    pub just_cancelled: HashSet<u64>,
}

I also didn't implement TouchMotion event. I wasn't super clear on what's the correct API. Doing both TouchInputState and TouchMotion event lines up the closest with existing mouse implementation. Naming-wise TouchInputState would be better as TouchInput to line up with Input.

Probably best to just clean up and merge that other one and then I can follow up with anything useful from my diff. e.g. having dx() and dy() helpers was useful for calculating distance moved from when the touch was initiated. Might want to call it delta and return a Vec2 instead or something, not sure.

MichaelHills commented 3 years ago

Just an FYI, my personal situation has changed and I'll be strapped for time to continue developing my game / Bevy iOS support for the time being. I intend to finish off the audio support but will hand over a WIP cpal branch if I need to. Happy to keep helping out with testing though!

Once all the other pieces land, I still plan to improve the iOS example project with the following:

CleanCut commented 3 years ago

@MichaelHills Thank you for all your hard work. I love being around great contributors like you who push projects forward in meaningful ways. I hope you are able to return to your game in the future!

cart commented 3 years ago

@MichaelHills I agree with @CleanCut. You've done a bunch of great work already. Don't feel pressured to put in more work than you want to!

dvtkrlbs commented 3 years ago

The new version of rodio is now released. I am working on PR. (this helps with audio not working on macOS and ios)

naithar commented 3 years ago

I've made some changes to original @MichaelHills touch support diff here: In my fork I plan to make a PR if #334 wouldn't be moving anywhere if that's okay :) I'll also run some more test just to be sure that everything is working as intended.

MichaelHills commented 3 years ago

Update on audio:

dvtkrlbs commented 3 years ago

Another problem is it takes a long time for a new version of rodio to release on crates.io. So even if these issues gets fixed you wont be able use them for a long time (rodio makes new releases for every 4 months or so) so you need to maintain your own bevy-audio (since crates.io crates cant use git dependencies).

MichaelHills commented 3 years ago

Another problem is it takes a long time for a new version of rodio to release on crates.io. So even if these issues gets fixed you wont be able use them for a long time (rodio makes new releases for every 4 months or so) so you need to maintain your own bevy-audio (since crates.io crates cant use git dependencies).

I haven't needed to modify Rodio. Does that mean we're ok? Or at worst can just use [patch.crates-io] to change the cpal version if there's a significant version bump.

MichaelHills commented 3 years ago

I've made some changes to original @MichaelHills touch support diff here: In my fork I plan to make a PR if #334 wouldn't be moving anywhere if that's okay :) I'll also run some more test just to be sure that everything is working as intended.

@naithar Looks like no movement on #334, want to put up yours? Or give some diffs so that #334 can be landed with original attribution, and then put up another PR for enhancements?

naithar commented 3 years ago

@MichaelHills yeah, seems like it. I'll rebase and make a PR soon.

naithar commented 3 years ago

Well, I found our problem! You need to disable XCode "Remove text metadata from PNG". I disabled everything though just to be sure... I guess the image loading library can't handle whatever XCode does to it.

this shouldn't be required anymore after https://github.com/bevyengine/bevy/pull/693. Listing assets folder in Copy bundle resources is enough for assets to work. Tested with iOS device and macOS .app file.

Also there is no need for a workaround for set_current_dir as PR seems to be solving this too.

francesca64 commented 3 years ago

Is there any interest in using cargo-mobile to simplify generating Xcode projects / building / running on device / etc?

cart commented 3 years ago

Medium-to-long term I might be interested in adopting higher level abstractions. Short term I'd rather keep it simple, encourage people to become familiar with the "native" mobile tooling, and maintain control over "official" templates. We will likely create an official bevy_template repo in the near future.

alice-i-cecile commented 2 years ago

This now works!

Rust-Ninja-Sabi commented 1 year ago

Is there any tutorial about Bevy and ios today?