matlo / GIMX

The GIMX software.
GNU General Public License v3.0
590 stars 105 forks source link

Add interface for computer vision #687

Closed KiwiLostInMelb closed 3 years ago

KiwiLostInMelb commented 3 years ago

I will write this feature - just an issue to track it for the eventual merge :)

Input using shared memory - I.e. another input type. Should be able to used simultaneously with mouse, kb and joystick.

I am writing this for a friend who has lost the use of his fingers on the right hand side and wants to be able to play COD again.

I will write a computer vision tracking system that simulates a relative type input with buttons. I.e. right axis on controller (hand up/down, left/right) plus buttons used for fire (on hand down).

This will be interfaced via shared memory (windows/linux) and will require another app to put through the inputs. Shared memory (shm) as it is a lot faster than tcp and easy to interface from python.

Reader will be threaded (10 mS probably) and poll for changes on the shared memory arrays. Intend to use int8 arrays for axis (-127 to +127, x and y) and one 8bit uint for button inputs.

I am developing this on linux for the Jetson Nano.

The hand tracking side is already completed as I tried to interface into the zen as a mouse but the mouse mapping in zen is crap and I couldn't get enough resolution out of it.

Any thoughts guys? Happy to include extra features if I know what they are before I integrate this.

Malto, thank you for this great app!!! Please add me to the list of devs and assign this one to me :)

KiwiLostInMelb commented 3 years ago

Hey Devs, Q on the best way to implement this. I am thinking another mouse type in linux only mkb.c. I.e. shared mem only on linux. But another mouse type that closely emulates the hid mouse type? Maybe a DEVTYPE_SHM_MOUSE?

Looking for ideas on the simplest way of implementing this with minimal impact on the rest of the code base.

Any thoughts or direction suggestions?

KiwiLostInMelb commented 3 years ago

Completed for Linux and physical input only. See: https://github.com/matlo/gimxinput/pull/3

matlo commented 3 years ago

Sorry for the slow reply. I was wondering if you considered using the network api, which relies on UDP packets. UDP should be fast and reliable on localhost.

https://gimx.fr/wiki/index.php?title=Network_API https://github.com/matlo/gimx-network-client/

KiwiLostInMelb commented 3 years ago

Hi Malto,

I implemented as shared memory for linux physical inputs. I am building and testing on a Jetson nano running headless so shm worked well and also very fast - I also wanted some method of synchronization - I.e. when the data is read/used in the poll loop.

Implementing as a mouse means I can also map using all the usual inputs for the left hand controller.

I could write this for xinput as well (does anyone use xinput on linux???). There is a pull request open for you for the physical side (shared input repository). If I can have wiki access I can put in a help page if you would like.

Not sure how windows does shm memory so a bit more research to implement for windows at the moment.

FYI - Next step is to create a ESP32 USB device as the Jetson Nano has kernel bugs that stop USB device mode working correctly (issue open with NVidia) and I have tons of ESP32's available. This should be pretty straight forward as well. Yes, I probably should buy the right micro but I have a heap of ESP32's from a failed past project with no home.

Thanks for getting back to me! I really appreciate the app and all the time it must have taken to develop.

Cheers,

Piet

On Thu, 14 Oct 2021 at 06:24, Mathieu Laurendeau @.***> wrote:

Sorry for the slow reply. I was wondering if you considered using the network api, which relies on UDP packets. UDP should be fast and reliable on localhost.

https://gimx.fr/wiki/index.php?title=Network_API https://github.com/matlo/gimx-network-client/

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/matlo/GIMX/issues/687#issuecomment-942644225, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGJZGLAN2THK2D7ELNJH5DUGXMFZANCNFSM5FWCPO4Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

KiwiLostInMelb commented 3 years ago

Grrr. Sorry I meant STM32 micro not ESP32. No morning coffee yet.

On Thu, 14 Oct 2021 at 06:42, Piet Van Nes @.***> wrote:

Hi Malto,

I implemented as shared memory for linux physical inputs. I am building and testing on a Jetson nano running headless so shm worked well and also very fast - I also wanted some method of synchronization - I.e. when the data is read/used in the poll loop.

Implementing as a mouse means I can also map using all the usual inputs for the left hand controller.

I could write this for xinput as well (does anyone use xinput on linux???). There is a pull request open for you for the physical side (shared input repository). If I can have wiki access I can put in a help page if you would like.

Not sure how windows does shm memory so a bit more research to implement for windows at the moment.

FYI - Next step is to create a ESP32 USB device as the Jetson Nano has kernel bugs that stop USB device mode working correctly (issue open with NVidia) and I have tons of ESP32's available. This should be pretty straight forward as well. Yes, I probably should buy the right micro but I have a heap of ESP32's from a failed past project with no home.

Thanks for getting back to me! I really appreciate the app and all the time it must have taken to develop.

Cheers,

Piet

On Thu, 14 Oct 2021 at 06:24, Mathieu Laurendeau @.***> wrote:

Sorry for the slow reply. I was wondering if you considered using the network api, which relies on UDP packets. UDP should be fast and reliable on localhost.

https://gimx.fr/wiki/index.php?title=Network_API https://github.com/matlo/gimx-network-client/

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/matlo/GIMX/issues/687#issuecomment-942644225, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGJZGLAN2THK2D7ELNJH5DUGXMFZANCNFSM5FWCPO4Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.