Perceivable latency

[franktinsley]

For me, perceivable latency really kills the experience so it's my number one issue. I'm a humble C# Unity dev but if someone can point me in some kind of direction on where I might start I will do anything I can to try to help get latency down. I would love to see latency reduced to the point where we can all play Mario at least as well as we can on Mario Maker for Wii U (which manages to have remarkably low latency even through a wireless gamepad that's also receiving a video stream!)

[franktinsley]

So I think I'm going to focus all my efforts on making PRs for Provenance's UI and in exchange hope someone with skills can try to reduce latency. I have a feeling, since Provenance is apparently based on OpenEMU, that OpenEMU is where latency will really have to be solved.

[kreeger]

By perceivable latency, are you referring to button-press latency on the Apple TV with a third-party controller? Or just emulation latency across the whole application?

[fdotg]

I have noticeable latency on my appletv and the nimbus controller.

[dougnukem]

Have you tried turning down the D-Pad/Analog threshold from 50% to 0% that seemed to fix it for me. I was really only noticing latency in the D-Pad or Joystick, normal button presses seemed fine.

[tommolling]

I have a latency too when using the Nimbus controller on appletv (feels like 100-200ms. ). It does not seem to be limited to the d-pad since i experience it a lot in Super Mario when jumping using "a/b"-buttons.

[kreeger]

For me and my Nimbus controller on the Apple TV, it's not D-pad latency that I've noticed — it's button presses as well, across all games and emulation cores. Pairing the same controller with my iPad or iPhone doesn't produce any noticeable latency at all — so I'm seeing the same as @tmolling.

I'm happy to run this tethered to my Apple TV to help diagnose this. I tried to throw in some logging myself this past weekend to figure out where the disconnect is, but I didn't get very far. Logging happens right away when the button presses are sent to the various core-specific gamepad controllers, but once the emulation code receives the button press, it seems to be a 100-200ms delay before the press is registered on screen.

And this latency doesn't happen in the selection menu, nor does it happen with any other game. I wonder if it's an Apple TV processing timing issue? (I'm shooting from the hip now.)

[XeresRazor]

The lag may very well be TV related, most modern televisions have a delay due to the image processing they perform, something that doesn't occur on an iOS device. You might try checking the settings for your TV to see if it has a game mode that will reduce this delay.

[tommolling]

Dont think it's TV-related as I tried it on my main Television as well as hooking up the AppleTV on a computer screen with HDMI. Same result.

[jasarien]

@kreeger mentions that it doesn't happen on iOS, only tvOS? Then that would suggest some issue with the AppleTV itself and not Provenance?

[hambly]

I did some diagnostics today with regards to MFI controller input lag. I used both a Mad Catz Ctrl i Micro, and a Steel Series Stratus with both and Apple TV and iPhone 6s+. The input latency is specific to Provenance/Emulators on the Apple TV only.

1) Provenance Apple TV - both controllers have the same input lag, it's just bad enough to ruin the gameplay. Someone above mentioned 200ms, and that probably makes sense from what I was experiencing. Tested on a couple SNES games. 2) Native Apple TV Apps - no noticeable lag with either controller. Tested playing Transistor with both controllers. 3) Provenance on iPhone - no lag, tested on the same SNES games

[dougnukem]

Testing this and trying to put in some time elapsed logs, it's starting to feel like it might just be the Steel Series Stratus controller buttons themselves.

I tested on iOS and on tvos and the lag felt the same to me (felt like 200ms or so on SNES Super Mario world when I hit the A or B buttons).

The D-Pad doesn't seem to have any latency once I dropped the D-Pad threshold to 0%.

What it feels like is that the Steel Series A, B, X, Y buttons are hard to register for a press you have to fully depress them and then it springs back up and it feels like it's a lot less smooth than a PS4 controller (or even a standard NES controller).

I'm starting to think it's not really an input lag issue with MFI Controller on TVos but just sluggish mechanical buttons.

[dougnukem]

hmm I'm not quite sure though I just played Rayman and everything feels really responsive.

[jasarien]

Interested to know if anyone here experiencing latency has their Apple TV directly connected to the TV via HDMI, or has it running through a receiver or Xbox One's HDMI-In or similar?

I have mine running through my Xbox One's HDMI-In and I get some latency from that. Without the Xbox in the middle it feels fine.

[marlinmayo27]

My Apple TV is directly connected to my TV via HDMI and I have latency issues with both the D-pad and regular button presses. Very noticeable in Mario games. I just got the Nimbus controller today and only tested it out for about 5 minutes.

[hambly]

I'm connected directly from AppleTV to the TV's HDMI-In, and experiencing the latency with Provenance-TV only. Native apple tv games are OK, as is Provenance-iPhone.

I'll dig into the controller interface code and see if I can do some helpful profiling if I get a chance over the next few days.

[kreeger]

My experiences exactly are the same as @hambly's — I'm plugging right into my Vizio TV's HDMI port here, and I don't notice the same lag on Provenance-iPhone, nor any lag on other Apple TV games.

[kaikai1234]

as all mfi controllers have pressure sensitive buttons, maybe having deadzone settings for all buttons might help.

[jasarien]

There aren't any artificial dead zones added to other buttons. They use the '-pressed' method from the game controller framework to decide when they are pressed. The documentation states that this takes into account any kind of hardware deadzone from the manufacturers side.

[mattstocum]

I'm experiencing latency too, but similar to what I see in some games. Alto's Adventure has very similar latency with button presses. Canabalt, on the other hand, has zero perceivable latency. I'm wondering if the heuristic tvOS is using to determine button presses is just bad.

[jasarien]

I'm beginning to wonder if the paradigm Provenance is using for controller input is wrong...

All of the emulators used in Provenance work on the basis of polling for user input. Each iteration of the emulation loop, the emulator checks its controller representations to see which buttons are pressed at that point in time.

Meanwhile, provenance is working with an event-listening model for the MFi controllers, assigning a handler to each button and updating the controller state once a button is pressed.

I think this might lead to some sync issues between the controller and the emulator, where the input is out of sync with what is being displayed on screen.

it might be difficult to explain, but I'll try anyway:

Consider Mario on the NES as an example. The emulation loop executes and renders a frame of the game with an enemy close to Mario that would prompt the player to jump in order to avoid the enemy. Upon seeing this frame on the screen, the player would press the jump button. Here's where things get complicated. I think the emulation loop executes faster than the event processing loop for the app, and by extension, the game controller framework. So by the time the event handler for the button has been called and updated the controller state for the emulator, the emulator has already rendered a few more frames, before seeing that the button was pressed, and making Mario jump.

I'll experiment with this theory to see if I'm right, and if I am, then I'll need to rewrite the controller handling so that the emulator can query the input state directly from the controller itself.

[rreinhardt]

Hi @jasarien

Another way to reproduce something that would be indicative of your theory is to rapidly press the jump button, say under a unlimited coin [?] box... If you do it fast enough you can witness certain jumps being 'dropped' which wouldn't happen if the game was in sync with the controller.

[whitverj]

Just wanted to leave a "Me too!" for latency on the Apple TV using a SteelSeries Nimbus. I've turned all the processing options off on my TV and enabled gaming mode, so I'm sure that's not the cause. It's not just the D-pad, it's all the buttons, and the delay is just enough that it's nigh-impossible to successfully play a game.

[kaikai1234]

@jasarien correct me if im wrong but your theory does not explain why some people here are claiming they are getting no lag when using a controller on the iPhone but getting lag on apple tv. anyone here who has a iPhone 6 and a Lightning Digital AV Adapter should try it on the same tv as their apple tv so we can eliminate tv lag as the culprit. i dont have a iphone but have a ipad 2 and that has a lot more lag then my apple tv.

[jasarien]

I don't know, maybe the different Sync rates have an effect?

[jasarien]

Should be fixed by 9bdd377f0530ac64b2c420fcb9ea97715762c576

Test it out and let me know if it feels better.

[kreeger]

Thanks for your work on this, @jasarien — I'm gonna check this out when I get home today and I'll update my comment about the changes.

Edit: the lag has been greatly reduced. There's still the tiniest little bit of noticeable lag, compared to the same setup on an iPad or an iPhone, but games are much more playable now. If the lag was 100ms before, it's like 40-50ms now.

[drewbaumann]

I still seem to still be experiencing lag (enough to hinder gameplay) on the Apple TV. I am using a steel series stratus and experimented on the NES and SNES emulator.

[kaikai1234]

i see no difference in latency,in fact i seem to be seeing a lot more with latest update, I'm going install a old version and check if im correct. jasarien i think you should try out a game like mario world, to see how bad it is.

[dougnukem]

I'm still seeing it as well.

Is there some way we could measure or print log the latency, I tried logging at the button press but I couldn't figure out where the internal SNES emulator actually handled the press state.

[hambly]

Tested on two apple tv's with two controllers (Stratus & Micro CTRLi) with a variety of games, definitely still enough latency with the controllers to make certain games unplayable (such as Super Mario World or Contra).

[whitverj]

I'm wondering if there's a simple way to just have a dot or something appear on screen when it registers an A,B,X,Y button press, just so we can nail down if the latency is from the time we press the button until Provenance registers it, or if it's downstream from that.

[kaikai1234]

ok did a full restart on the apple tv the horrible lag i was getting before is gone, the new version is better but there is still noticeable lag.

[bvleur]

Have to confirm the above. With tvOS 9.0.1 and the SteelSeries Nimbus controller and a build with 9bdd377f0530ac64b2c420fcb9ea97715762c576 included I see much less lag, but there still is some.

[mattstocum]

Still seeing latency here too on Super Mario World.

[tommolling]

Same here. No real change on NES/SMB. Latency is still pretty much the same and some button presses are not even registered. i.e. when holding A to run and then pressing B to jump, the jump sometimes is not made.

[jasarien]

Dunno what to say. It worked fine for me when I tested it, latency was much improved. I wouldn't have committed the changes had it made it worse. And there's so many mixed reports here that I can't draw any useful conclusions. Some people say it's better, some say it's the same, others say it's worse. Some say rebooting their AppleTV fixes some latency issues. Have you tried that? Have you tried re-pairing your controller to eliminate any bluetooth issues?

[drcreek]

I've just connected a brand new controller to the latest build. Freshly rebooted. Controller is very quick and responsive when controlling the UI. But in games. It's laggy in a way that is just enough to suck all the joy out of it. The Genesis and Master System emulators seems faster than SNES. but both lag. Just the SNES lags a little more.

[chriscthompson]

Ok I've been watching this thread for a while and I have to chime in...

I'm using release version 1.2.2 with a Nimbus Controller on a 32 GB  TV 4 and I have next to zero or an imperceptible lag if there is any at all. My son also has the exact same experience on his with the identical setup.

No lag in SNES Super Mario World or Genesis Sonic & Knuckles

On Nov 21, 2015, at 11:02 AM, drcreek notifications@github.com wrote:

I've just connected a brand new controller to the latest build. Freshly rebooted. Controller is very quick and responsive when controlling the UI. But in games. It's laggy in a way that is just enough to suck all the joy out of it. The Genesis and Master System emulators seems faster than SNES. but both lag. Just the SNES lags a little more.

— Reply to this email directly or view it on GitHub.

[jasarien]

@chriscthompson 1.2.2 is a few versions behind the current. Could you try with the latest build?

[chriscthompson]

Yes. I can update tonight and let you know my feedback.

On Nov 21, 2015, at 11:10 AM, James Addyman notifications@github.com wrote:

@chriscthompson 1.2.2 is a few versions behind the current. Could you try with the latest build?

— Reply to this email directly or view it on GitHub.

[tommolling]

Looks like this is gonna be a neverending story :P

well, i tested again: apple tv rebooted, provenance 1.2.6, Nimbus re-paired, controller is very close to the aTV, but it still lags.

Here the result: https://vid.me/30Z7

tbn. i've tested with the aTV remote as controller, and the lag is identical. so the nimbus can be excluded as root cause.

[jasarien]

@tmolling Does your TV have a "game-mode" and is it enabled?

[tommolling]

yes, gaming-mode is turned on. tested on a samsung tv, which does not feature a gaming mode, but same lag.

edit: and as mentioned above by someone, all other games like rayman, edge, jetpack or canabalt on the aTV as well as menu inputs work just fine

[drcreek]

Just to add. It's not my TV at least. Game mode on. I have the same lag as in the video. But the controller is almost instant on Crossy road. And that is a twitchers game.

Skickat från min iPhone

21 nov. 2015 kl. 22:02 skrev tmolling notifications@github.com:

yes, gaming-mode is turned on. tested on a samsung tv, which does not feature a gaming mode, but same lag.

— Reply to this email directly or view it on GitHub.

[franktinsley]

With latency being a huge problem for OpenEMU is there some reason Provenence should be expected to improve on it?

Also a very simple way to test a setup and verify that the latency is coming from Provenence and no where else is to simply test other games that use mfi controllers on the same Apple TV.

[kreeger]

@franktinsley The issue seems to be specific to the combination of Apple TV + Provenance + MFi controllers. We've tested for controller latency on other Apple TV games (Badland, Canabalt, etc.) and there doesn't seem to be any latency with those.

Likewise, using the same MFi controller with the same build of Provenance on an iPad or iPhone doesn't yield any latency there, either. I wouldn't strike OpenEMU off the list as being a problem, but latency does seem to be just fine on iOS. It's just on tvOS that seems to be an issue.

[drcreek]

Also. I have noticed the lag is much much lower with the Apple Remote.

MFi controller. It's not playable for most games.

[jasarien]

There is way too much conflicting information in this issue. Some people say they experience latency, others do not. Everyone has different configs and setups, different hardware. No-one has done a controlled experiment with a Test ROM designed to measure latency, no-one has wired up an LED to the controller button and taken High-FPS video (and I mean not iPhone camera footage) to capture when the LED lights and the Test ROM shows the button press.

I appreciate you may feel like you have latency issues, but with so many variables (bluetooth, different models of TV, different controllers, different connections, different games), it's impossible to know empirically where that latency may be coming from, whether it's Provenance, the individual ROMs, the hardware, the wireless connection. etc.

Edit: I'm not saying Provenance is perfect, and I'm not saying Provenance isn't the cause of the latency. Provenance will get better over time, I'm aware that some people may be experiencing latency - Continued posting in this thread is a distraction and doesn't help.

[jasarien]

I'm unlocking the comments here, since I've had some reports that iOS 9.1 has improved the latency issues some were experiencing with MFi controllers on tvOS. If you could test it and report back, I'd be interested to hear.

[kreeger]

Gladly. Do I need to recompile against the tvOS 9.1 SDK first?