Joystick Emulation

[Flandan]

With the idea of hooking up the output weights to the a joystick directly I took a quick look at Joystick emulation options and came across this: http://vjoystick.sourceforge.net/joomla256.02/index.php/forum/4-Help/1044-getting-started-with-python

Specifically the second last post, by trafiken. It looks like a pretty nice and lightweight solution, I believe it only requires the sdk and driver for the virtual device. If I get the time I'll give an implementation a go myself. Loving this project so far.

[Flandan]

Here is a solution: https://gist.github.com/Flandan/fdadd7046afee83822fcff003ab47087

it just requires the virtual device driver from here: http://vjoystick.sourceforge.net/site/index.php/download-a-install

Cant wait to see it in action! hopefully Ill manage to do some of my own training soon too.

Edit: Right, so having finally downloaded gta to test this it seems the game doesn't actually support joysticks at all. Instead they seem to have gone for some kind of hardcoded gamepad support. I may have found a workaround though: http://www.x360ce.com

So now we have to use our python interface to control the vJoy driver to create a virtual joystick. Then remap that joystick to a virtual gamepad so that gta5 can see it. I'm starting to think there must be a better way... I also have not got this convoluted mess to actually work in game yet.

On another note, switching over to Arma 3, I'm having plenty of luck. The vJoy virtual joystick works great when hooked up to the predicted weights. So there is something in this. I am just building up a data set now for further training and tests.

[krishvishal]

Sorry for my noobness, @Flandan What GPU are you using?

[Flandan]

No worries. Im using a gtx 1060, although Im recording and using a much lower resolution dataset than Sentdex (we dont all have a titanX lying around :D). Right now Im recording 80x30 and having decent results in arma. I'll get a youtube video up soon to show where I am at.

[Sentdex]

Bummer to hear the joy isn't supported on GTA V ffs. I seem to recall they don't support much at all. You can't use real driving wheels, nor flight control sticks IIRC. I hadn't considered that with this task. Definitely share some videos from arma with us!

We definitely need something to mimic the xbox360 specifically. I've seen some C++ stuff to do it, but no clue how to tie it into python, no clue what I am doing in c++

[Flandan]

C++ is my main language so thats something I can look into. I havent really done much with it in python but I am pretty sure this is what ctypes is for. If you come across any libraries feel free to point me to them, it would be very useful for me to learn how to do this.

[krishvishal]

https://www.reddit.com/r/GrandTheftAutoV_PC/comments/33btq3/psa_you_can_eventually_get_a_joystick_working_on/ is this what you are talking about @Sentdex ?

This is a for the guide of the same program mentioned above, adding for future reference. https://steamcommunity.com/sharedfiles/filedetails/?id=573253647

[Flandan]

Thats the method I am using at the moment. It would be nice to push the controls directly to the driver though, without the mess in between. Its possible x360ce can be used as a driver api though (it is open source) so I will take a deeper look. Its not exactly its intended use case though.

[krishvishal]

@Flandan But why do we need to do that? I'm asking this clarify. If the program can enable us to use joystick then why bother about direct interface? we still can record the joystick inputs right?

[Sentdex]

@BanditBoi Sort of. Basically, there are two ending solutions to this problem. One is to use a joystick emulation library along with some sort of solution like that reddit post.

The other option is to use a pre-existing xbox360 emulation library that can be hooked up to any joystick already, and the "joystick" that we'd hook it up to is just our python program.

I am not 100% certain that we must LOG the joy inputs, certainly worth a try to see, but I think we can actually take the network output, trained from keyboard presses, and map those %'s to joy input. From my looking at the data, it appears to me just that would be a great upgrade.

[Flandan]

Exactly, really the issue is that we want something everyone that wants to work on the project can install with ease. Ideally something directly available through python or its package managers so its quick to get up and running, without too much hassle. Unfortunately that doesn't seem to exist so I'm looking for the next easiest thing to use. It is possible there is something obvious we are missing though.

[Flandan]

I am not sure If this is the best place to post this but here is where I am at so far: https://www.youtube.com/watch?v=3fiPVOuyk84 I altered the original alexnet to target an analogue output and used the average of the past 20 frames of keyboard input as analogue input. I think I will change this to the future 20 frames in my next training test as that actually makes a lot more sense.

Interestingly I only trained this model on the asphalt road but it seems to do better on the dirt road. I think this is mainly because it confuses the road barriers as road markings, as a result its pretty consistently crashing into them.

[Flandan]

Thought I would post the changes I made to alexnet to enable analogue input/output for anyone looking into finer control.

https://gist.github.com/Flandan/9f9e4ce0c46b67d17f5d74423eef264c

The training data is in similar form to the original but now with 2 analogue control values X-axis and Y-axis (steering and throttle respectively). I also changed the image data shape to be height major as I think that's wrong in the original but I'll address that in another issue when I can. For the time being you can swap height and width for minimal trouble integrating with existing code.

I managed to get good steering behaviour and good throttle behaviour independently by training targeting only one output. For example it works well when I train the model to steer and control throttle manually, it follows the road, overtakes cars etc. But, with both together, it seems to lose generalisation. I have tried adding layers and widening layers which improves things a bit but not to the point where either is reliable.

One thing I would like to have control over is which variable it prioritises in training (steering). But I haven't found a good way to do this yet, I'm still playing with the loss function but I'm not fully sure what form its inputs come in yet.

I also still haven't tried gathering data with an actual analogue controller. And, my data is completely unbalanced which I don't think is ideal but I haven't come up with a good solution for that yet.

Any ideas would be appreciated

Edit: Just thought I had better add that the module @bayangan1991 has created, PYXInput is working great with gta and makes it very easy to push output to a virtual controller.

[Sentdex]

Any chance you ran into: "Unable to load vJoy SDK DLL. Ensure that vJoyInterface.dll is present"

I downloaded the SDK, and moved vJoyInterface.dll to system32, but it still throws the error. Used 64bit version, tried a restart, not sure what else could be the issue. Apparently you can tie in vjoy and x360ce for xbox 360 emulation.

edit: Well at least with pure vjoy, you can just set a constant I see after checking that posted code above: CONST_DLL_VJOY = "path/to/vJoyInterface.dll"

edit 2: alright all set with the constant, and now I've got full xbox 360 emulation in GTA V via Python.

You can use vjoy, then tie in x360ce, and boom you can do joy input via python in gta v. Going to begin working on this more in depth. 💯

[Flandan]

Interesting, are you using the PYXInput module @Sentdex? I switched to using that as I found it to be a much cleaner, less hacky solution once we ironed it out (I probably should have made that more clear in my last post). The module installs the vXboxInterface.dll when installed via pip. Here are the steps I took to use PYXInput:

• pip install PYXInput (it is now a publicly available package)
• Head to the vxbox section on the vJoy website
• Follow the bus installation steps for scpvbus-x64 (this is the driver)
• Test the module in an interactive python session:

  >>> import pyxinput
  >>> pyxinput.test_virtual()

  Successful output (this may be slightly different but as long as there are no errors you're good):

  Connecting Controller:
  This ID: 1
  Available: [2, 3, 4]
  Setting TriggerR and AxisLx:
  0.0
  0.1
  0.2
  0.3
  0.4
  0.5
  0.6
  0.7
  0.8
  0.9
  1.0
  Done, disconnecting controller.
  Available: [1, 2, 3, 4]

  @bayangan1991 included a version of scpvbus-x64 in the PYXInput repo also but I cant recall if I have tried installing that version directly. I am having no problems after following the method I have described above though, and its been very robust so far. More examples of use are given in the repo. I may do a pull request to make other things like triggers and buttons obvious in the readme. Although just browsing the virtual_controller.py (line 90) makes things pretty clear.

It also works across all games, without the need for adding x360ce files everywhere.

[Sentdex]

I am going to close this for now. I ended up setting it all up myself via vjoy + x360ce, but I might look into pyxinput. If you want to see my implementation, it's live.

I am leaving the pyxinput post up since it looks like that's an easier implementation for OTHER people to take part in, mine's a little sloppier for actually getting things setup, but I need to test pyxinput first.

[jae1379]

Interesting, are you using the PYXInput module @Sentdex? I switched to using that as I found it to be a much cleaner, less hacky solution once we ironed it out (I probably should have made that more clear in my last post). The module installs the vXboxInterface.dll when installed via pip. Here are the steps I took to use PYXInput:

• pip install PYXInput (it is now a publicly available package)
• Head to the vxbox section on the vJoy website
• Follow the bus installation steps for scpvbus-x64 (this is the driver)
• Test the module in an interactive python session:

>>> import pyxinput
>>> pyxinput.test_virtual()

Successful output (this may be slightly different but as long as there are no errors you're good):

Connecting Controller:
This ID: 1
Available: [2, 3, 4]
Setting TriggerR and AxisLx:
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Done, disconnecting controller.
Available: [1, 2, 3, 4]

@bayangan1991 included a version of scpvbus-x64 in the PYXInput repo also but I cant recall if I have tried installing that version directly. I am having no problems after following the method I have described above though, and its been very robust so far. More examples of use are given in the repo. I may do a pull request to make other things like triggers and buttons obvious in the readme. Although just browsing the virtual_controller.py (line 90) makes things pretty clear.

It also works across all games, without the need for adding x360ce files everywhere.

Hey @Flandan thanks so much for the direction. I'm playing around with pyxinput and got it all properly installed and running. However, I'm having a bit of trouble with the pyxinput since i'm a fairly new to coding. Could you perhaps provide a quick example script for: Plugging in XBOX vjoy controller 1 --> wait on standby with no active inputs --> press X button --> release X Button --> wait on standby with no active inputs --> disconnect controller 1? I am having trouble even plugging in the controller part :-/ Thank you very much