Lag and controllers becoming unresponsive

sniejadlik commented 6 years ago

Thanks again for getting this all working. I have finally freed up time to start tinkering with this and have noticed some interesting issues. The first time I fired it up the entire thing went incredibly easy. My joycons connected flawlessly, Unity read them and they were super responsive, I was loving it and testing out the different buttons and delighted to see the rumble button firing correctly.

I then set about creating a UWP friendly version of the joycon class, I finally got that in a good place, went to test and nothing. I noticed my controllers were only showing paired in the settings and would not connect. removed the entire package and reimported it, taking the code back to before. I removed them and tried again, nothing, checked for driver updates, restarted bluetooth and computer and they finally connected but when I ran the code they were responding about 3-4 seconds behind and the rumble and joysticks were no longer registering.

I am currently charging them back up in case it is a battery level issue but was wondering if you have ever encountered lag and no joystick/rumble response? It's very confusing as it worked flawlessly last night and I have reverted the entire build back to your latest without any of my modifications.

sniejadlik commented 6 years ago

NM, I realized that in trying to update this I switched over to .net4.6

BTW we were demoing right next to each other at ARIA this past week, I wish I had made the connection when I saw your company logo.

wormyrocks commented 6 years ago

Oh fantastic! I wasn't at ARIA but you probably saw my boss.

I haven't tested this exhaustively with multiple Joy-Cons yet - that feature is basically experimental right now. I have had the experience of lag and a lack of rumble response with multiple Joy-Cons connected simultaneously, but for the purposes of our company it is not terribly important as we only really need to support one at a time. If you end up gaining any insight, don't hesitate to let me know and I can try to fix it!

Did changing the .net version solve the problem permanently?

sniejadlik commented 6 years ago

Oh nice! Yes it was your boss then, I think he was in the same presenting block as me. Love what you guys are doing. Anyways yes, switching back to the previous .net version fixed the lag. I am currently in the process of reworking the code to deploy to UWP as that is the only way to deploy to the Hololens. So in doing that I was getting an error that made me think it needed to be switched. Turns out that isn't the case. I spent yesterday learning about repackaging dlls and getting everything converted and ported over and I think I am down to one last hurdle. Converting .net Threading into uwp ASync Tasks. I think. That seems to be the one last thing preventing the application from reading the values. Hoping to spend some time over the next few days and solve that. I will definitely ping you as I work through it.

Right now for test purposes one joycon will be plenty but overall two is the dream. Magic AR wands with haptic feedback....drool.

sniejadlik commented 6 years ago

So in trying to work through the conversion issues to UWP I have run into some hiccups. The Thread you are using to poll the Controller on begin, is that continuously running or is that just to establish initial values? I am trying to figure out the best solution to work with that because there are some issues I am running into. The biggest being that if I switch to ASync Tasks as the main way to create a thread then I need to switch to .net 4.6 which then makes for this huge lag on the controllers in Unity. Do you have any insight into the controller settings as to why switching to 4.6 would suddenly cause such a lag?

I know you haven't touched this is a while so I am just trying to work through all this stuff on my own but figured I would ask in case you knew where I should start looking to resolve this.

Ideally I would love to just avoid the need to have threading since its not uwp compatible. Do you think a Coroutine would work as a proof of concept even if it caused some frame rate issues?

wormyrocks commented 6 years ago

Hey - it's great to have someone who knows what they're doing on this, especially as this project is my first time touching Unity or C#.

My understanding of the lag coming from the controller (though this may be flawed) is that once the Joy-Cons begin pumping out 6-axis data, they expect to push a packet every 15 ms, and I constructed my threading system to read HID packets at basically exactly that rate. If packets are not read as soon as they arrive, they build up in a FIFO, probably either in the Joy-Con wireless stack, or in the accelerometer chip itself. That is why if the threading is messed up, the accel/gyro data updates nicely for about 10 seconds, and then gradually begins to lag behind the actual motion. You are pulling data that is a few seconds old out of a couple kilobyte buffer on the Joy-Con. My first iteration of the library polled HID data once per frame, but even with a consistent frame rate this would occur. That is what you are experiencing - totally smooth motion, but several seconds behind, correct? Otherwise it may be something else.

wormyrocks commented 6 years ago

I have some disabled-by-default code which I used to make sure packets came in every 15 milliseconds, and they are timestamped in such a way that you can make sure they are consecutive. (report_buf[0] ticks up by 1 every time a 15ms packet comes in). Try here: https://github.com/Looking-Glass/JoyconLib/blob/master/Assets/JoyconLib_scripts/Joycon.cs#L365 though you may have to contend with my questionable DebugPrint method.

sniejadlik commented 6 years ago

Awesome, it's funny because this is my first project attempting to work with hardware and Unity. The real hiccup is in trying to deploy it to the Hololens which has it's own limitations. So that thread you are setting up in Begin is what you are using to continuously poll the device right? So it's absolutely necessary to have that separate from the main thread? It's good to know that the conversion to 4.6 could just be a matter of timing which sounds like it could be adjusted. I will look into testing it out with 4.6 just to see if I can get the ASync logic working even with a lag.

Thanks again!

wormyrocks commented 6 years ago

Happy to help. Yes, the separate thread is what is used to poll the device, and I was only able to avoid lag with that threading. It's also worth looking at these two other Joy-Con libraries: https://github.com/riking/joycon https://github.com/mfosse/JoyCon-Driver though, I'm not up to date on their level of support for motion controls.

sniejadlik commented 6 years ago

I'm slowly working through this. Definitely a tricky set of issues and lots of new stuff I have never dealt with before. Thankfully found out the 4.6 is not the lag issue. There was a debug print format issue that was causing a backup and causing the delay. Also the DebugType.THREADING wasnt printing to anything. I was getting some weird errors in 2017.1 so upgraded to .3 and so far less errors. I now have the DLL building out to x86 without any errors and so I am working through the polling and update functions trying to figure out the best way to get them working having no knowledge on async but learning as I go. Hopefully this is actually possible and I am not just spinning my wheels endlessly without a solution. I don't know if any of this is actually making a difference but I feel like it's progress.

CTCaer commented 6 years ago

I'm not good with what you are doing, but here are my 2 cents on this:

The joy-con are never congested. They just sent packets. (x30 mode) The BT protocol either takes a packet or a corrupted packet which rejects it.

The best way to parse these without lag, is to not use a sleep timer and expect a certain time between the packets. You need to use a blocking hidread thread which parses a packet the moment it comes through the hid library.

So, if the calculations are not very expensive (e.g. always use multiplications, instead of divisions with float and doubles, when using constants, by using a converted denominator: 1/denominator, or by doing the maths with bit-field operations), you will never drop or queue any packet.

And of course by disabling debug prints which are the most expensive in the parse thread.

wormyrocks commented 6 years ago

@CTCaer Hey, thanks for your help. When I designed the read thread I made sure to double check the timestamps on the packets against the actual time the packets were received to make sure I wasn't dropping/queueing the packets (i.e., one every 15 ms, and every timestamp is consecutive). That is what the debug print in the parse thread is used to confirm. Of course it is disabled by default for normal use. And yes, it is a blocking read, which is why it needs its own thread.

@sniejadlik Yes, once I got the hidread thread working well I turned off the relevant debug prints. Hopefully you figured out you can change what debug prints show up in the output by changing the debug type here. If you want a better idea of how the code is structured (with code by a better Unity developer) I would also recommend you look through the code on Unity-Wiimote, on which this is largely based (and uses the same HIDApi).

Are you experiencing these latest issues simply porting the code to UWP, or when attempting to support multiple Joy-Cons?

sniejadlik commented 6 years ago

@wormyrocks Yes figured out the debug typing and that all works well. The controllers are perfectly responsive in Unity even on 4.6. My issues are simply in trying to port over to UWP. I am definitely missing something with the initial setup as I just cannot see the controllers at all. I thought I had the async threading working right last night as it finally compiled but the controller value still stayed at 0. I am going to take another look tonight. Quick question for you, do the controllers get identified before the Poll method or during Poll method? I had assumed during but can't seem to get it to find them and was wondering if I have something else set wrong.

wormyrocks commented 6 years ago

Controllers turn on in the Attach() function, it's also what sets their LEDs.

On Jan 29, 2018 11:11 PM, "sniejadlik" notifications@github.com wrote:

@wormyrocks https://github.com/wormyrocks Yes figured out the debug typing and that all works well. The controllers are perfectly responsive in Unity even on 4.6. My issues are simply in trying to port over to UWP. I am definitely missing something with the initial setup as I just cannot see the controllers at all. I thought I had the async threading working right last night as it finally compiled but the controller value still stayed at

I am going to take another look tonight. Quick question for you, do the controllers get identified before the Poll method or during Poll method? I had assumed during but can't seem to get it to find them and was wondering if I have something else set wrong.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Looking-Glass/JoyconLib/issues/4#issuecomment-361275445, or mute the thread https://github.com/notifications/unsubscribe-auth/AKknxMnRVY0tclKSrNIqxzrbiYyhiLRyks5tPd-zgaJpZM4Rl4ih .

gb2111 commented 6 years ago

@sniejadlik i have version working on win forms if that will help you. still i have lag when 2 controllers are tracked. with one all works perfectly.

edit: even if I connect to only one controller, i still have lag. also I compared raw and processed data (for button) and they are identical and they both lag so it is not about processing.

@wormyrocks you don't have this problem in unity ? i am guessing you tried use 2 devices :)

edit2: i tried run it on 2 processes with same result. when I turn off left device, then even right one in separate process (exe) is stopping processing for a while and then resumes but with full speed.

edit3: ok. i solved. not want to be ignorant but calling 2 times ReceiveRaw() makes Joycon Left and Right workign smoothly. i someone explain why and how to address it properly will be very happy.

sniejadlik commented 6 years ago

@gb2111 I am unfamiliar with win forms So I am not sure if it's helpful.

@wormyrocks I am having issues just getting the device to be recognized in the attach method. I remember that when I messed around with these in the past the hololens recognized them as different devices than my pc so I am wondering if somehow the product id values are off. I am a complete novice when it comes to device connections so am just sort of trying anything I can. I am right now looking at the windows 10 HID capabilities just to see if I can identify the joycons as devices. (which I know have worked in the past as just generic) and wondering if that is something you have tried with these?

wormyrocks commented 6 years ago

You can always check the Vendor ID and Product ID in Windows on the HoloLens, and make sure that the product ID is 0x057e and the product ID is 0x2006/0x2007. I am using the HIDApi C#/Unity bindings from the UnityWiimote package; there should be a straightforward way to include that library with anything you end up building. @gb2111 Can you share your fix for this issue? I may take another look at this.

gb2111 commented 6 years ago

i will make also project on gitgub if you don't mind. i think it does not really matter if you use it with win forms or wpf.

gb2111 commented 6 years ago

here is project ported to desktop app. in case of any problem let me know and I will see if can help.

wormyrocks commented 6 years ago

Whoa, nice work! I'll look into why that ReceiveRaw business might be the case. Have you tried with more than two Joy-Cons; i.e., do you suppose that ReceiveRaw must be called once per Joy-Con, or is twice sufficient? @CTCaer Do you have any idea why doing two consecutive blocking reads might alleviate the lag issue with multiple Joy-Cons, even though each Joy-Con has its individual thread and ought to be running independently?

CTCaer commented 6 years ago

@wormyrocks Hmm, well actually doing this cuts the work to half. You parse 1 packet for each joy-con every 30ms, instead of 1 every 16ms.

Maybe the parsing takes more time than 15ms and for some reason the next packet read gets congested (strange though because hidapi doesn't have a queue and just drops the packet if it isn't read).

I will test tomorrow and tell if I find anything specific.

gb2111 commented 6 years ago

hi, sorry for offtop but perhaps you or someone you know would be interested in looking on this lib for GearVR?

I think I decoded all buttons and 3 variables related to rotation. Only problem is I cannot decode them into something that makes sense line quaternion. Just not mine area of skills. I made UI that makes easier to see groups of bits as they change color according to frequency they change.

GearVR is really great and it would be wonderful to have ability to integrate it in our projects. Greg

gb2111 commented 6 years ago

@wormyrocks i have only 2 joycons, also my project will not show more then 2 right now.

CTCaer commented 6 years ago

@wormyrocks @gb2111 Ok I just tested https://github.com/gb2111/JoyconLib-4-CS and both

int a = ReceiveRaw();
a = ReceiveRaw();

and

int a = ReceiveRaw();
//a = ReceiveRaw();

behave the same for me and most importantly, they both show "zero" lag.

*By "zero", I mean no noticeable lag.

Is it possible that this whole lag situation is coming from slow math/drawing performance? I have an i7-5820K, and the program seems very fast to me. Maybe on a slower CPU, the cube drawing slows down? @gb2111 Can you try the following? Comment the 2nd ReceiveRaw to go back to instant/15ms packet reading:

int a = ReceiveRaw();
//a = ReceiveRaw();

Test 1: Then start by commenting the math and drawing:

var j = joyconManager.j[0];
label1.Text = j.ToString();

//cube1.InitializeCube();
//cube1.RotateX = (float)(j.GetVector().eulerAngles.Y * 180.0f / Math.PI);
//cube1.RotateY = (float)(j.GetVector().eulerAngles.Z * 180.0f / Math.PI);
//cube1.RotateZ = (float)(j.GetVector().eulerAngles.X * 180.0f / Math.PI);

//picture1.Image = cube1.DrawCube(drawOrigin);

var j = joyconManager.j[1];

label2.Text = j.ToString();

//cube2.InitializeCube();
//cube2.RotateX = (float)(j.GetVector().eulerAngles.Y * 180.0f / Math.PI);
//cube2.RotateY = (float)(j.GetVector().eulerAngles.Z * 180.0f / Math.PI);
//cube2.RotateZ = (float)(j.GetVector().eulerAngles.X * 180.0f / Math.PI);

//picture2.Image = cube2.DrawCube(drawOrigin);

Test 2: then only the cube drawing:

var j = joyconManager.j[0];
label1.Text = j.ToString();

cube1.InitializeCube();
cube1.RotateX = (float)(j.GetVector().eulerAngles.Y * 180.0f / Math.PI);
cube1.RotateY = (float)(j.GetVector().eulerAngles.Z * 180.0f / Math.PI);
cube1.RotateZ = (float)(j.GetVector().eulerAngles.X * 180.0f / Math.PI);

//picture1.Image = cube1.DrawCube(drawOrigin);

var j = joyconManager.j[1];

label2.Text = j.ToString();

cube2.InitializeCube();
cube2.RotateX = (float)(j.GetVector().eulerAngles.Y * 180.0f / Math.PI);
cube2.RotateY = (float)(j.GetVector().eulerAngles.Z * 180.0f / Math.PI);
cube2.RotateZ = (float)(j.GetVector().eulerAngles.X * 180.0f / Math.PI);

//picture2.Image = cube2.DrawCube(drawOrigin);

Test 3: and finally only the maths:

var j = joyconManager.j[0];
label1.Text = j.ToString();

cube1.InitializeCube();
//cube1.RotateX = (float)(j.GetVector().eulerAngles.Y * 180.0f / Math.PI);
//cube1.RotateY = (float)(j.GetVector().eulerAngles.Z * 180.0f / Math.PI);
//cube1.RotateZ = (float)(j.GetVector().eulerAngles.X * 180.0f / Math.PI);

picture1.Image = cube1.DrawCube(drawOrigin);

var j = joyconManager.j[1];

label2.Text = j.ToString();

cube2.InitializeCube();
//cube2.RotateX = (float)(j.GetVector().eulerAngles.Y * 180.0f / Math.PI);
//cube2.RotateY = (float)(j.GetVector().eulerAngles.Z * 180.0f / Math.PI);
//cube2.RotateZ = (float)(j.GetVector().eulerAngles.X * 180.0f / Math.PI);

picture2.Image = cube2.DrawCube(drawOrigin);

This way you can check if the math or/and drawing is slowing you down.

Lastly, I want to add that the BT adapter plays a big role here. If I go a little far away from my crappy adapter, I also start noticing lag (which actually the lag is because of dropped corrupted packets or packets that do not reach the adapter).

wormyrocks commented 6 years ago

@CTCaer Are you testing with multiple Joy-Cons? Both @gb2111 and I have found that calling ReceiveRaw() twice only alleviates the lag situation if more than one Joy-Con is connected (in both Unity and Windows desktop app).

CTCaer commented 6 years ago

@wormyrocks Of course. I don't know why, but with 1 Joy-Con or 2 Joy-Con and 1 ReceiveRaw or 2 ReceiveRaw, the output is the same for me. The cuboid and the text coordinates had the same speed to all tests. Also, with 1 ReceiveRaw was more detailed, as it should have been.

Here's my Windows build machine: Windows 10 fully updated. i7-5820K, 16GB DDR4 2400 AMD R9 280x Visual Studio 2015.3

I've done no change to compilation and framework properties (it didn't asked me to change sth in the project).

I will write a timer and check in ms, because this is very strange

CTCaer commented 6 years ago

Here's a sample my latency: (Order is by columns. Not that it matters)

16 | 30 | 33 | 31 | 30 | 30 | 31
16 | 16 | 14 | 16 | 23 | 16 | 18
30 | 30 | 31 | 29 | 25 | 28 | 28
17 | 15 | 14 | 15 | 13 | 16 | 15
29 | 32 | 32 | 31 | 31 | 29 | 30
34 | 29 | 31 | 31 | 32 | 33 | 32
13 | 16 | 39 | 15 | 18 | 14 | 15
30 | 29 | 15 | 32 | 28 | 31 | 30
14 | 18 | 13 | 15 | 14 | 15 | 15
31 | 29 | 32 | 30 | 32 | 30 | 32
31 | 30 | 30 | 32 | 30 | 32 | 30
16 | 18 | 16 | 15 | 15 | 16 | 14
29 | 27 | 32 | 30 | 30 | 30 | 31
16 | 18 | 12 | 16 | 15 | 15 | 16
30 | 29 | 31 | 31 | 31 | 31 | 31
32 | 30 | 30 | 31 | 31 | 29 | 31
19 | 16 | 17 | 16 | 17 | 16 | 15
26 | 30 | 30 | 29 | 28 | 30 | 31
18 | 16 | 16 | 16 | 16 | 16 | 15
29 | 30 | 29 | 31 | 31 | 29 | 31  

30 | 30 | 30 | 31 | 31 | 30 | 29
27 | 19 | 26 | 15 | 16 | 16 | 18
19 | 27 | 18 | 29 | 34 | 30 | 27
15 | 16 | 15 | 15 | 14 | 15 | 16
31 | 32 | 32 | 31 | 29 | 30 | 30
31 | 29 | 30 | 31 | 31 | 32 | 30
18 | 15 | 15 | 16 | 19 | 36 | 16
29 | 31 | 30 | 30 | 26 | 10 | 30
14 | 15 | 16 | 16 | 16 | 15 | 16
31 | 32 | 31 | 32 | 32 | 30 | 29
31 | 31 | 29 | 31 | 35 | 32 | 33
16 | 16 | 19 | 16 | 11 | 16 | 14
30 | 30 | 26 | 29 | 30 | 29 | 31
18 | 16 | 16 | 16 | 16 | 16 | 16
29 | 30 | 30 | 30 | 29 | 30 | 29
30 | 31 | 33 | 31 | 33 | 31 | 32
17 | 15 | 16 | 16 | 15 | 15 | 14
29 | 30 | 30 | 30 | 30 | 31 | 30
14 | 15 | 14 | 16 | 16 | 15 | 15
31 | 32 | 32 | 29 | 28 | 32 | 30

31 | 31 |  
19 | 15 | 
28 | 32 |  
15 | 15 |   
31 | 31 | 
31 | 30 |  
15 | 15 | 
31 | 30 | 
15 | 15 | 
30 | 32 | 
31 | 30 |  
19 | 15 | 
27 | 30 |  
15 | 16 |   
30 | 30 | 
31 |  
20 |  
25 |  
17 |   
30 |

So average latency was 24ms. Best values were around 15ms, which match the report frequency from joy-cons and worst value was 39ms.

So no extreme conditions.

The results were made by connecting 2 joy-cons and showing 2 independent cubes rotating.
A single ReceiveRaw() was used.
The timer measured everything from reading packets, parsing them, doing maths and drawing the result on screen. (So, both joyconManager.Update() and UpdateInfo())
Lastly, these results are consistent with every run.

I'll now write 1 more timer. 1st will measure joyconManager.Update() and 2nd UpdateInfo(). I will post my findings

gb2111 commented 6 years ago

i have quite new i7 so I don't believe it can be anything about cpu. especially when only this app is running.

so @CTCaer you have same result on my version with 1 or 2 call to ReceiveRaw and @wormyrocks you have same situation as me?

that mighe be something with bluetooth stack. the only difference between calling ReceiveRaw once and two times is that when calling hid_read 2 times for one call to hid_write.

CTCaer commented 6 years ago

I just saw the code better. The results are updated through a timer. Reducing the timer interval to 1 I see that the whole thing is faster. Additionally, I still think that using a timer is a bad choice for polling. Your actual timer should be hidapi. It read a packet? Start doing stuff.

Anyway tested the cube calculation/drawing and only takes 0-2ms.

So the problem seems to be on 2 sides: The timerUpdate. The library side of things. What I'm gonna do now, is what I said above. I will try to call joyconManager.Update(); UpdateInfo();

from inside Joyconlib when a packet is received and avoid the whole timer thing. This way, I will find the real latency. And also minimize the latency from the timer.

wormyrocks commented 6 years ago

@CTCaer At least in the original library, I don't believe I'm using a timer for polling. In the Poll() function, I read a packet from HID, and then read the subsequent packet immediately as soon as it is available. It is a blocking infinite poll loop that runs in its own thread, as you suggested. In the main Update() loop, which at least in Unity runs once per frame, every packet is dequeued and all three sets of accelerometer/gyroscope data are plugged into the sensor fusion equation with 5ms assumed dt, and actual dt in between packets used. Is the delay you refer to here? That should only happen when a packet is missed and I just put that in so that the program would not freeze and poll indefinitely fast when a connection is questionable. My current suspicion is that some flag in the HID pipeline is put into a blocking state when multiple HID devices are called independently and by calling ReceiveRaw() twice, I am "accidentally" resetting it such that data comes through without congestion. Do you think that is possible?

CTCaer commented 6 years ago

@wormyrocks Indeed, I'm talking about @gb2111 implementation.

The timerUpdate_Tick() (by using 1ms interval) is called every 10-21ms. So, 9-20ms. This is normal though, because Microsoft states that winforms timer has an accuracy of 55ms.

This needs to be changed to System.Timers.Timer to have a 10-15ms accuracy or to a better one. The best way is to create an interface that JoyconLib can reach and call the MainForm update function.

What is left from the frontend is the latency from actually calling joyconManager.Update() and UpdateInfo(). The first one has 0ms latency and the second 0-2ms for me. So it's OK.

Have in mind that the numbers are now lower, because I implemented my check stopwatch counter better.

Now on the best case scenario, we should have a latency of 15 backend + [10-21]frontend = 25-36ms But for you and @gb2111 this is not the case.

So, for now, I'm trying to find the backend latency.

Edit: No it's not possible that ReceiveRaw fixes a congestion in hid. The reads are blocking. That means the 2nd ReceiveRaw is only called when the 1st receives a packet. Because this works for you (parsing every 30ms), means that something around there is the problem. Edit2: Also, having a blocking hid read, ReceiveRaw will stay there forever if the controller has not lost connection but has a very bad signal.

CTCaer commented 6 years ago

@wormyrocks Joycon.update() is 0ms Joycon.ReceiveRaw() is 15ms (As expected. And if I lower the signal, it goes up because of dropped/unreachable packets from BT adapter). Joycon.Poll() is 15ms with 1 ReceiveRaw() or 30ms with 2 ReceiveRaw()

JoyconManager.update() is 0ms.

So your lib latency is flawless.

@gb2111 You need to implement another timer or another way to update the form. Try to use an interface and link it with the library so you know when the library got a packet. Your timer now has around 10-21ms latency total on my machine. Your math/drawing latency is excellent 0-2ms.

@wormyrocks @gb2111 So, in the end, I don't know why you have these latency. Joycon4CS works great for me. Only around 10-30ms latency total. I had them now for half an hour and everything responds fast and within the same latency when I opened the program.

With this, I think both your signal quality must be very bad. Try to have the Joy-Con near the adapter (10cm). If this doesn't change the situation, check your machine, your DPC latency and your build enviroment/runtimes.

wormyrocks commented 6 years ago

@CTCaer I figured out what is causing the congestion on my machines; it is my SendRumble() call, which sends an 0x10 report before every call to ReceiveRaw. If I remove that call, it works with more than one Joy-Con with no congestion and only one call to ReceiveRaw(); if I retain that call, it congests after about 10 seconds on my PC. (laptop, integrated Bluetooth adapter).

It looks like when I send the 0x10 "rumble only" command, the Joy-Con queues up a response, but it also is spitting out packets continuously when it is in 0x30 output mode. So it delivers two packets per ReceiveRaw() call, but only one of them is actually read.

I have no idea why this problem only manifests when we use more than one Joy-Con, and only on a Windows PC. I will try with three Joy-Cons to try and get a stronger sense of what is going on.

Update; with three Joy-Cons, it works just the same as two. Perhaps calling ReceiveRaw() twice is the best practice, and the delay introduced when using only one Joy-Con is somehow just imperceptible.

BTW @gb2111 I don't have a Gear VR controller, but you are welcome to look at my ProcessIMU() function and the article I link to in the code if you want to try and process the accel/gyro coming out from the Gear VR. I am happy to help if you have questions.

gb2111 commented 6 years ago

@CTCaer i don't think there is anything with timer. if you will look on text output you will notice "Button Up (raw)". That is basically output from ReceiweRaw not processed by timer and if called only once we can see same lag on button press no matter if from ReceiveRaw or processed by timer. Also there is no (visible) lag between raw and timer. They both lag or not.

Regareding daydream my problem is that I have no clue what those data mean. I identified 3 values that change consistently with rotation of device. so probably no filter is required. but have no clue what thye are :)

CTCaer commented 6 years ago

@wormyrocks First, how the Joy-Con work, based on captured packets and ROM/Firmware disassembly experience: If you enable x30 input report, doesn't matter what you send, the Joy-Con will always send an x30 report every 15ms. Even if you send a x01 it will send a x21 back and a x30 on it's schedule (-+1ms). It never sends 2 x30 right away, without a 15ms time window.

How the BT adapter works: Most adapters queue input/ouput non-corrupted packets. The queue is there, because sometimes it can't parse the parse data fast enough to output it to HID protocol or output it to BT channel.

So a latency can exist there. But this latency, cannot reach hundreds of ms.

On the other hand, this latency can become way worse when the signal is not good enough. It's actually virtual latency. Because, it doesn't receive any packets, the gap looks like a latency to our eyes.

So, because you only changed the code to not encode rumble when it is not used, this suggests that the encoding functions used are quite cpu expensive. Additionally, the expected rumble "keep alive" behavior is to send a 00014040 rumble packet every 120ms or just 8 x30 packets received and not every packet received!

But, also when you send real rumble packets, the expected behavior is that parsing x30 and sending x10 does not lag.

So what you need to do, is to keep the calculation for testing reasons and try to simplify and use an inexpensive calculation. And then, remove encoding for muted rumble. (For example, Joy-Con and Switch use around 8 look up tables to do the maths needed).

The most important that I forgot to say, even in dekuNukem's github (because it actually considers how Switch is handling Joy-Con traffic): When Switch plays HD rumble, it sends rumble packets every 15ms instead of 5ms which the bnvib are defaulted. The Joy-Con support even lower than 5ms x10 packets, but the fastest rate Switch sends, even if only one Joy-Con is used, is 15ms. Switch does this to not congest the BT channel. The problem is that Switch uses look up tables to change the original values, smooth them and receive the same or even better results from the LRAs and I can't seem to make a lot of sense from them.

Lastly, as you can understand, using 2 ReceiveRaw(), you cut the work in half by having a packet ready for parsing/proceeding every 30ms. But also, you send a rumble packet only every 30ms. (Poll() repeats: Send rumble --0ms> Read packet --0ms> Read packet --15ms>)

Summary: Your library lags, because you send x10 keep alive packets every 15ms instead of 120ms and also your rumble encoding math is too expensive. How to fix:

Send rumble "keep alive" (00014040) x10 packets only every 120ms or 8 x30 received. Will fix all problems completely (when you don't "play" HD rumble).
When hd rumble is really used, cut the packets sent to 15ms. 15ms will still lag your library so you need to also do the following:
Try to find inexpensive HD rumble encoding math.

Always remember, that the BT adapter plays a big role on all of these. In my case, I don't see a lag bigger than 30ms, and I'm amazed to know that my crappy usb BT adapter can handle this much traffic.

I don't have time to proof read my comment and check if everything makes sense, if you don't understand something, or I wrote something wrong, tell me.

@gb2111 Well actually I'm talking about parsing mostly. It doesn't matter what you do on your front end, the JoyconLib receives packets every 15ms. It's threaded so it does not matter how fast or slow your timer is. And the parsing is happening only when you call the j.ToString() and JoyconManager.Update().

These 2 functions are called only when your timerUpdate fires off. The problem with that timer is that it has a low accuracy. Even though you have it set to 16ms, it fires off randomly between 9-20ms (sometimes 30ms) in my case. That means that sometimes you lose some packet parsing, because 15ms passed and it was replaced by a new one.

Summary: We now know that the actual big lag comes from the library (Poll()). That's why, as you said, you can see almost the same lag from Raw and from the timer. But your frontend has some additional lag. Not important (but unneeded in my opinion).

If the library is corrected, you don't need to do anything additional. Maybe lower the timer interval to 15ms, to match the actual x30 report rate. (It's every 15ms (66.6Hz), not 16ms (60Hz)).

If you want to go the extra mile and make a flawless frontend, synchronize your parsing with JoyconLib: Remove the timer and add an interface so JoyconLib can signal your frontend that a packet was received. When the signal is received, call joyconManager.Update() and UpdateInfo() to parse it and process it right away.

wormyrocks commented 6 years ago

@CTCaer Thanks a lot. Yes, I need to update the library to reflect the recent new discoveries about HD rumble, and not constantly sending rumble packets should be very useful to kill congestion. @gb2111 I just remembered that my housemate has a Gear VR, so I will look at your library sometime this week and try to ID the data coming out.

gb2111 commented 6 years ago

@wormyrocks many thanks for that !!!

@CTCaer for testing purposes I am updating one button in this line in ReceiveRaw call. It is not processed in Update.

I could still see delay in button press in same way as orientation. That means for me is 100% evidence that lag problem has nothing with processing timer but is purely about data being accessed from bluetooth/file.

CTCaer commented 6 years ago

@gb2111 That's what I said in my last comment. The lag comes from Poll() in JoyconLib.

Your frontend does not have lag exactly. What I mean is that the timer is a little inconsistent on triggering the values/UI update. Some times it updates faster (9ms) than the interval (16ms) other times slower (30ms). Normally it should update every 16ms but this is not the case. As I told you, this is because of the low timer accuracy.

If you want to look at real numbers based on your PC replace MainForm.cs with one of these: Measure real trigger interval of timerUpdate based on High Precision Stopwatch timer: https://mega.nz/#!4B0w2brS!ZHvC8FRB0uCFnwk0EwtB581mfW4dds1Q15TGvb9Eiwc

Measure execution time for joyconManager.Update() and UpdateInfo(): https://mega.nz/#!gZVUBSBC!frCXSNQTYzpM4N9HZyszvQkoTY-6UmWXhlSvOxl4gZs

The number that you'll see come only from your frontend

Gambitboy commented 5 years ago

First off, I appreciate the work put into this project, it's really cool. I was just wondering if anyone was able to fix the latency issue? I'm currently trying to use this library with 3 joycons in a game project.

The rumble functionality and the sensitive of the axis are basically what I'm after.

I've got the 3 joycons connected and they are working but joycon 2 and 3 have noticeable input lag. Its a split second delay on anything I do, for both axis and button input. It also fells like joycon 2 and 3 suffer from the same amount of input lag. As if they are equally slower than joycon 1.

Once the library is setup and connects the controllers, it blocks all usage of the controllers through anything else. So I'm unable to use the controllers through unity's Input system. I was thinking of using the library just for the rumble and then use Unity Input for everything else since it doesn't lag through Input and i'm willing to sacrifice the sensitivity, but I can't seem to do that.

Any help would be great, thanks.

qpskcn1 commented 4 years ago

I'm having the same issue. When the second joycon gets connected, no matter which one, it will become laggy or even unresponsive. But the first connected joycon remains normal. Has anyone resolved this problem? Calling receiveRaw() twice did help reduce the lag. But the second joycon lag is still unacceptable. Any help would be appreciated! Thanks!

Looking-Glass / JoyconLib

Lag and controllers becoming unresponsive #4