WIP visionOS support

[alexhaugland]

An extremely bare bones port to visionOS. It works enough to pair, view apps and connect to a stream in the simulator. Mouse input and video seems to work fine.

Notable changes:

• Switched to using openssl3 instead of openssl1 since there was a readily available build for visionOS/visionOS-sim (https://github.com/krzyzanowskim/OpenSSL)
• Updated the certificate generation code to use outdated parameters compatible with iOS/visionOS' outdated certificate importing code instead of the new openssl3 defaults
• Refactored a few of the wrapper core data classes to swift so they could be @Observable in the ui
• Changed an assortment of build settings
• Built a basic SwiftUI interface
• Basic ViewModel behind the interface that could theoretically be used on all platforms
• Added a main function to call the SDL init stuff
• Added swift-collections
• Probably some other things

Known things to do:

• [x] Figure out how to use a version of openssl that is maintained and not a random fork of a fork I made (https://github.com/alexhaugland/OpenSSL)
• [x] Have a sane data flow in the UI
• [X] Implement the rest of the UI
• [X] Fix resizing the window (it is delayed by a frame or two due to where I'm updating the render layer's size)
• [x] Investigate using SwiftData everywhere and cleaning up the save/load code a bit
• [ ] Figuring out where to put the within-stream UI to allow for keyboards / special keys / closing the stream / etc
• [X] Testing it on a real device
• [ ] Actually import the c++ header so the audio/video codec definitions are readable instead of copypasting them
• [x] Fix the view when stats are enabled - it is real broken
• [ ] Deal with the five hundred warnings related to libraries not built for exactly the right target

Any and all constructive advice / criticism / suggestions are welcome!

[Geczy]

im confused, im already using moonlight with sunshine on my windows 11 pc and it works great. what does this introduce for vision os?

[alexhaugland]

im confused, im already using moonlight with sunshine on my windows 11 pc and it works great. what does this introduce for vision os?

the biggest thing for me is window management - if you're using the ipad app compatibility mode you're pretty restricted in how you can resize the window

there's some secondary issues around UX that can be improved as well - the swipe to go back gesture is difficult / impossible to do with your eyes, the settings menu target is hard to hit, the individual settings are a bit crowded, etc

[adam-fox]

Just played with this on my Vision Pro. Super excited to see where this goes. Thanks for your work. I was able to load up my Sunshine server just fine but am unable to access the settings menu in Moonlight to tweak anything.

[alexhaugland]

Just played with this on my Vision Pro. Super excited to see where this goes. Thanks for your work. I was able to load up my Sunshine server just fine but am unable to access the settings menu in Moonlight to tweak anything.

The settings menu does not exist yet - I'll implement it eventually but PRs are welcome :D

[adam-fox]

Ah! Well, that would make sense as to why it isn't working for me then. I'm no developer or I'd jump on helping out. Is there any chance you know the variables I could manually edit in Xcode to change the resolution? Thanks again.

[alexhaugland]

the bottom of MainViewModel holds the hard-coded settings

[Geczy]

ah got it. is this also what it takes to remove the black bars on the top and bottom when casting?

[alexhaugland]

ah got it. is this also what it takes to remove the black bars on the top and bottom when casting?

yeah pretty much

[Geczy]

have a testflight or anything? i can share it around and have beta testers

[alexhaugland]

have a testflight or anything? i can share it around and have beta testers

i might set one up once the settings menu works but no promises

[robbiet480]

This is a fantastic work that will also make it easier to rebuild the UI of the other platforms with SwiftUI, something I was investigating doing a few months ago but gave up on quickly.

The only issue I'm seeing with your fork is that my gaming PC is being discovered 3 times (so 3 sidebar entries). This may be because it has 3 IPs (IPv4, IPv6 and Tailscale which both AVP and my gaming PC have enabled)?

[alexhaugland]

This is a fantastic work that will also make it easier to rebuild the UI of the other platforms with SwiftUI, something I was investigating doing a few months ago but gave up on quickly.

The only issue I'm seeing with your fork is that my gaming PC is being discovered 3 times (so 3 sidebar entries). This may be because it has 3 IPs (IPv4, IPv6 and Tailscale which both AVP and my gaming PC have enabled)?

Yeah, that's a known bug. It's probably related to how it compares TemporaryHosts for equality.

[adam-fox]

Just want to say that I tried this on my Vision Pro and it works extremely well. The only change that I made was that I hacked in an option for 90 Hz streaming. Thank you so much for doing this.

oh, nice! @alexhaugland is making some amazing progress. all i’m waiting for is custom resolutions, custom bit rates and 90hz and this would be 100% usable for me as is. can’t wait to be able game on a 100” ultra wide display.

[VeniceNerd]

have a testflight or anything? i can share it around and have beta testers

i might set one up once the settings menu works but no promises

Oh yes, please!!! Would love to test this out via TestFlight!!!!

[austinarchibald]

Thanks for this work! Trying to build and deploy with xcode. Do I need to change the bundle identifier? I think it's having issues installing/upgrading because I already have iPad version from App Store. Do I need to delete that first before installing onto my device, or can I have both so I can compare performance?

[Geczy]

my build is failing too in xcode

[matte-schwartz]

working fine here... make sure to follow all the steps in the instructions and ensure you used --recursive when cloning the visionOS repo and branch, or after switching to the visionOS branch and updating the submodules if you forgot --recursive while cloning.

[Geczy]

nvm, got it to build by deleting the old xcode version, and using only xcode beta 3

[austinarchibald]

Working for me now, thanks, probably didn't clone properly the first time. On visionos branch on github, I clicked code > Open with xcode, closed xcode, opened Terminal, cd'd to moonlight-ios dir, then git submodule update --init --recursive to finish those linked dirs, opened in xcode, Targets > Moonlight Vision, Signing & Capabilities, Team: My Name, then changed Bundle identifier, built and launched on my vision pro within seconds.

Will continue to update this as updates are pushed, thanks!

[alexhaugland]

The settings screen kind of works now - I'm waiting on review for a testflight link.

[alexhaugland]

@MetricExpansion I merged your changes (minus some log lines) into the branch. Thanks!

[VeniceNerd]

Will you post the link to the TestFlight here once it is approved? I’m dying to try this out!

---

From: alexhaugland @.> Sent: Sunday, February 11, 2024 2:06:59 PM To: alexhaugland/moonlight-ios @.> Cc: VeniceNerd @.>; Comment @.> Subject: Re: [alexhaugland/moonlight-ios] WIP visionOS support (PR #1)

@MetricExpansionhttps://github.com/MetricExpansion I merged your changes (minus some log lines) into the branch. Thanks!

— Reply to this email directly, view it on GitHubhttps://github.com/alexhaugland/moonlight-ios/pull/1#issuecomment-1937739421, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AF7W2M4TIKIBWHS5IR7KWBLYTC67HAVCNFSM6AAAAABCOFTDQGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZXG4ZTSNBSGE. You are receiving this because you commented.Message ID: @.***>

[mstoreytr]

Thank you so much for all of the hard work on this. After the recent updates, things are looking really amazing! Definitely a big step up from using the iPad app.

iPad App (20ft away & max size):

VisionOS App (20ft away & max size):

[austinarchibald]

Latest build is looking awesome! Got flawless 3ms stream at 4k/90 HDR, with a screen that filled my entire wall. Thanks @alexhaugland and @MetricExpansion!

[alexhaugland]

Here's a testflight link for the latest build: https://testflight.apple.com/join/4eE59dyH

[Geczy]

already working so good, great job on this. are you using a real avp to test btw?

bug report: the bitrate setting doesn't save, i set it low, connect, disconnect, and its back to 200

[alexhaugland]

already working so good, great job on this. are you using a real avp to test btw?

bug report: the bitrate setting doesn't save, i set it low, connect, disconnect, and its back to 200

yeah, I've got a physical device

technically the bitrate setting would save but it would be blown up when you reopened the settings menu :D - I've uploaded a fixed build, lmk if that works for you

[mddfe]

Looks great so far! When using the magic trackpad (or a mouse with proper assistive touch settings) it seems like right click does not work, with or without mouse support enabled.

[MetricExpansion]

bug report: the bitrate setting doesn't save, i set it low, connect, disconnect, and its back to 200

@Geczy Most likely, the setting is actually getting saved and applied. It's just that the Settings UI is doesn't reload the setting when you go to look at it. Try setting the bitrate, framerate, or resolution super low and it will be noticeable.

@alexhaugland the problem is the use of @State to store the index of the resolution, framerate, and bitrate settings. You should probably either

• change the tag values to actually just be settings values themselves and bind directly to the settings model (as is already being done for the other settings), or
• change them to each to be a Binding where the get and set closures convert the index to/from the setting value the settings model.

Either way, you can also then dispense with the @State variables and the use of the onChanged modifier. I've attached a patch with the first solution.

moonlight-ios-c796c95-Fix settings roundtrip for resolution, framerate, and bitrate.patch

[alexhaugland]

bug report: the bitrate setting doesn't save, i set it low, connect, disconnect, and its back to 200

@Geczy Most likely, the setting is actually getting saved and applied. It's just that the Settings UI is doesn't reload the setting when you go to look at it. Try setting the bitrate, framerate, or resolution super low and it will be noticeable.

@alexhaugland the problem is the use of @State to store the index of the resolution, framerate, and bitrate settings. You should probably either

• change the tag values to actually just be settings values themselves and bind directly to the settings model (as is already being done for the other settings), or
• change them to each to be a Binding where the get and set closures convert the index to/from the setting value the settings model.

Either way, you can also then dispense with the @State variables and the use of the onChanged modifier. I've attached a patch with the first solution.

moonlight-ios-c796c95-Fix settings roundtrip for resolution, framerate, and bitrate.patch

That's a lot cleaner, thanks. I've applied the patch and am uploading another build.

[MetricExpansion]

Here's a patch that adds more resolution options, to cover many more aspect ratios. Might still be good to have a custom option at some point.

moonlight-ios-afabe36-Add more resolution options.patch

[alexhaugland]

Here's a patch that adds more resolution options, to cover many more aspect ratios. Might still be good to have a custom option at some point.

moonlight-ios-afabe36-Add more resolution options.patch

I'm not sure how to best do it so I've been avoiding it. I don't love the way the current apps have a popup when you select it. The way the visionOS settings app works is pretty nice for a mixture of default and custom options on a subpage of each option, but I'm not sure if there's a quick and easy way to do it.

[MetricExpansion]

Here's one way to do it, if you like it. Just two text fields, but also a menu to select from pre-defined options.

moonlight-ios-e8d5c43-Custom resolution options.patch

[alexhaugland]

I don't entirely love it - most people probably want to pick 1080p/4k so that should be as easy as possible - but I won't let that stop the people from having custom resolutions for now 😁

[MetricExpansion]

That's totally fair. Ideally this would be like a few common resolutions plus a "Custom" option, which then reveals the text fields. Not too hard to implement, but a little more fuss than I was feeling like going through.

[MetricExpansion]

Another concept, maybe more in line with what you're thinking? Put the resolution options into a submenu instead of a popup menu, and add an ornament to enter in the custom resolution. Also remove some options since they can now be entered manually. Patch is attached if you like it.

moonlight-ios-7359549-Custom resolution options in submenu.patch

[alexhaugland]

Yeah, I think that's a lot closer!

[anthonyjutchenko]

Thank you for this. This is infinitely better than running the iPad app on the AVP. I do notice a slight amount of input lag/delay, which is undetectable when I stream my PC to my Steam Deck or other devices. I wonder if it’s a software or hardware limitation.

[alexhaugland]

Thank you for this. This is infinitely better than running the iPad app on the AVP. I do notice a slight amount of input lag/delay, which is undetectable when I stream my PC to my Steam Deck or other devices. I wonder if it’s a software or hardware limitation.

I've noticed the same thing - it got a bit better when I switched to a dualsense controller but I would not be surprised if we're losing at least one frame of latency due to the compositor. What refresh rate are you using?

[Geczy]

just curious, does vision os dev have hot module reloading like a js environment? or do you have to rebuild each time and relaunch

[alexhaugland]

just curious, does vision os dev have hot module reloading like a js environment? or do you have to rebuild each time and relaunch

You've gotta rebuild/relaunch but it's relatively quick. SwiftUI has a preview functionality for iterating on just UI that has hot reloading in the editor.

[anthonyjutchenko]

Thank you for this. This is infinitely better than running the iPad app on the AVP. I do notice a slight amount of input lag/delay, which is undetectable when I stream my PC to my Steam Deck or other devices. I wonder if it’s a software or hardware limitation.

I've noticed the same thing - it got a bit better when I switched to a dualsense controller but I would not be surprised if we're losing at least one frame of latency due to the compositor. What refresh rate are you using?

I’m capping the frame rate at 90 frames per second. I’ve been testing with an Xbox controller and keyboard/mouse, both connected to my hardwired PC. Streaming over WiFi 6e. I would estimate an added 20ms of input lag, compared to my Steam Deck OLED and LG OLED C3.

[adam-fox]

@alexhaugland @MetricExpansion you both are amazing. A little over a week and we've got a full featured AVP app. Thank you so much!

[adam-fox]

@anthonyjutchenko noticed the same but in my testing, this is also present in the iPad compatible version on the AVP. are you seeing the same?

[Geczy]

i cnnect to a desktop, and take the headset off. any way to keep the desktop connected? is it intentional that it disconnects and closes the window? i want to put the headset back on and resume my desktop mirroring unless i explicitly close it. this is the way most apps work from what i can tell

[anthonyjutchenko]

@anthonyjutchenko noticed the same but in my testing, this is also present in the iPad compatible version on the AVP. are you seeing the same?

Yes, the same amount of input lag is present with the iPad app.

[austinarchibald]

The TestFlight version behaves differently for me than the one I build and install via xcode. Same version (latest). I deleted the xcode version on my Vision Pro to just use Testflight for OTA updates, and it would not recognize my local IP. Says no machine found, see if Sunshine is running correctly (it is). Then when I enter my public ip using the [::ffff:x.x.x.x] workaround (what you have to do on all ios versions), it does recognize the machine, Sunshine receives a pin request, I enter the pin code, Sunshine web gui says success, but on Moonlight, the pin instructions disappear, and it still says Start Pairing for yet another pin request, then same behavior endlessly. I cannot pair my machine whether lan or wan. I went back to installing the xcode version, and I can enter in my local ip, and pairing is successful on first try.

Anyone else seeing this with the Testflight version?

[alexhaugland]

i cnnect to a desktop, and take the headset off. any way to keep the desktop connected? is it intentional that it disconnects and closes the window? i want to put the headset back on and resume my desktop mirroring unless i explicitly close it. this is the way most apps work from what i can tell

AIUI it must disconnect when you remove the headset since the app is being put to sleep but in theory it could reconnect when the app is resumed - PRs welcome :D

[alexhaugland]

The TestFlight version behaves differently for me than the one I build and install via xcode. Same version (latest). I deleted the xcode version on my Vision Pro to just use Testflight for OTA updates, and it would not recognize my local IP. Says no machine found, see if Sunshine is running correctly (it is). Then when I enter my public ip using the [::ffff:x.x.x.x] workaround (what you have to do on all ios versions), it does recognize the machine, Sunshine receives a pin request, I enter the pin code, Sunshine web gui says success, but on Moonlight, the pin instructions disappear, and it still says Start Pairing for yet another pin request, then same behavior endlessly. I cannot pair my machine whether lan or wan. I went back to installing the xcode version, and I can enter in my local ip, and pairing is successful on first try.

Anyone else seeing this with the Testflight version?

does the app store ipad version work with the same public IP?

[adam-fox]

@anthonyjutchenko noticed the same but in my testing, this is also present in the iPad compatible version on the AVP. are you seeing the same?

Yes, the same amount of input lag is present with the iPad app.

I actually think my issue isn't so much related to input lag as it is this: https://github.com/moonlight-stream/moonlight-ios/issues/607. Looking at the statistics overlay, everything should be good but the stream is stuttering constantly. Happening on both this and the iPad version on the AVP. Also occurring when testing on an iPad Mini, though. So, seems to be an issue with the iPadOS version of Moonlight itself.