Request for help implementing HID protocol in BrailleTouch project

[discapacidad5]

Hello parquerrr,

I hope this message finds you well. I came across your BDIOHID program, which handles converting old displays to the HID protocol. I believe your expertise could be of great help to my open-source project, BrailleTouch. Our goal is to create affordable Braille displays for blind and deaf-blind individuals.

I'm reaching out to ask for your assistance in implementing the HID protocol in our project. We are currently stuck and unable to get the HID Braille to receive information from the screen reader and continue programming. As someone with experience in this area, your support could be invaluable to our project.

I believe that an open-source Braille display could be incredibly useful and have many benefits for those who are blind and deaf-blind. Your contribution to this project could make a significant impact and help us move forward in achieving our goals.

Please take a moment to review our project on GitHub at https://github.com/brailletouch. If you're interested in helping, please let me know.

Thank you for your time and consideration. I look forward to hearing back from you.

Best regards,

Cergio Monasterio

[parkrrrr]

I'm happy to help!

For background, I was one of the principal engineers for Window-Eyes from 1998 to 2016, I worked on some other AT stuff in 2017, and I worked on Narrator and UI Automation from 2018 to 2020. I've also done HID Braille work from the client side, but I don't know how much I can say about that so I won't. I can't promise that I'll be able to solve all of your problems, but I'll do my best. (And if I can't, I might still know some people who can.)

I'm not sure whether I'm looking in the right place, but I'm not seeing any code in your repos.

If you have any specific questions, feel free to ask. If you would like to communicate privately, feel free to contact me at the same username at gmail.

[discapacidad5]

Thank you so much for responding so quickly! I truly did not expect such a speedy response and I am grateful for your willingness to support my project.

Allow me to introduce myself - I am a student who is learning programming. I am also familiar with visual disabilities as I interact with individuals who are blind or visually impaired on a daily basis. This inspired me to develop a project that can benefit this community.

Regarding the project, I have written some code that receives a message from an Android application and sends it to servomotors to form Braille characters. However, I want to modify it using electronic magnets or solenoids to create Braille cells. I am confident in my ability to handle this part of the project, but I am encountering a problem with the ESP32 device I chose to use. Although it has Bluetooth capabilities, it does not have support for HID over USB. I believe the forthcoming Arduino R4, which includes Bluetooth, will be a better fit for my project.

What I need your help with is a library for Arduino that allows it to communicate with a screen reader and deliver the essential components of a Braille display. This includes the option to choose between an 8 or 6 cell display, the 8 points of the Braille keyboard and its space bar, navigation keys for forward and backward text scrolling, and the two rocker switches. Other keys would be great, but these are the most essential. Ideally, the library would be dynamic and allow the user to select which keys to use and create the report based on that. This would enable me to test the display with different Braille configurations and determine which is most suitable for the user.

As we will be working with virtual cells and a single physical cell, we will need to experiment to determine how many virtual cells are usable for the person. I would like to conduct trials with individuals who are deafblind, which is why it's crucial to have a way to receive text from a screen reader.

I think a library would be a great solution, similar to those available for keyboards. The user would incorporate the library and then decide which keys to send and the library would generate the report and send the pulses. If you could create this library for me, the rest of the project is ready to go. I have the Braille cell and need to modify it to fit the user's needs, work on the touch sensor, and create the first prototype.

I am providing a link to the simple code that I have developed so far. I have been working on this project by myself, so I am grateful for any help you can provide. I am not confident in my ability to work on the HID protocol, so I am hoping you can assist me in enabling an ESP32 device to communicate with a screen reader via Bluetooth using the HID Braille protocol.

Lastly, I want to mention that Spanish is my native language, and I am using the help of an AI language model to communicate with you in English. Thank you again for your time and willingness to help with this important project.

[discapacidad5]

This is the code that I have. It receives text via Bluetooth from an app called "brailletouch_testing.apk" and converts it into Braille using four servo motors that form the eight Braille dots. However, I am planning to modify it because the servo motors have gears that wear out over time, lose stability and do not maintain their position, and they are also quite noisy. Therefore, I plan to use an adaptation made with Arduino relay modules and magnets to create magnetic levitation in the Braille dots, eliminating the need for gears that wear out. This will result in a faster response and completely silent operation. Arduino relays and mini magnets are readily available worldwide and are also very affordable. https://github.com/brailletouch/Brailletouch/blob/main/Electronica%20%26%20firmware/braille_bluetooth.ino

[parkrrrr]

I don't know if you've tried to read the code to bdio, but one of the things it does is dynamically generate the report descriptor for whatever display is attached to its USB Host port, so we might be able to reuse some of that code.

I believe that there are some newer ESP32 variants like the S2 with built-in USB support, but I'm not sure what the extent of that support is. Are you dead set on the ESP32 platform, or would you be willing to consider something like one of the many RP2040-based boards on the market?

I know, for example, that the Adafruit QT Py RP2040 is capable of operating as a HID keyboard, and I've personally used one to create a HID keyboard with a custom report descriptor. That specific board doesn't have many GPIO pins, but there are other boards in the same family that would likely work. None of them have built-in Bluetooth support, but there is an add-on ESP32-based board that can be used for BLE peripheral support.

[discapacidad5]

Actually, I'm using the ESP32 for the following reasons: it has built-in Bluetooth and supports low-power BLE. I know that BLE allows for configuring keyboards and mice as HID devices, so I assume it will also allow for the HID Braille protocol.

The second reason is that, as I mentioned, I'm still learning programming and I don't know how to flash anything from anywhere else other than the Arduino IDE. If I were to work with something else, I would have to learn how to flash and program it.

The third reason is that I have an ESP32 at home that I started testing with, but that's the least important reason because I can buy another device if necessary. However, I'm open to your suggestions and support. If another device is needed, I would love to be able to use it via Bluetooth, as it is especially useful for mobile phones. Nowadays, most people don't have a computer but they do have a cellphone.

[discapacidad5]

Basically, the most important thing to keep in mind is that we are developing the first open-source braille display that is truly accessible to anyone, anywhere in the world, including Africa and so-called third world countries. Therefore, we must ensure that all the components that we incorporate can be easily obtained anywhere in the world. My goal is to develop a product that does not exceed $100 in cost. I know this is a challenging task, but I believe it is possible. The idea is to develop each of the pieces and create manuals so that anyone can learn how to make them and assemble their own braille display, or give the instructions to someone with the necessary knowledge to make the pieces. It is important that the pieces are easy to manufacture, which is why I am using Arduino modules. I will use the Arduino relay to create the braille cells, so that the person only needs to purchase the relays, magnets, and 3D printed pieces that form the mechanism. Currently, 3D printing can be done in most parts of the world, making it accessible. Additionally, anyone who wants to produce printed parts at an affordable price is welcome to do so. The goal is to ensure that the total cost of producing the braille display does not exceed $100, making it affordable for people with disabilities, and comparable in cost to a typical smartphone.

[discapacidad5]

In the following text, I will provide you with a more detailed explanation of how I expect my device to function so that you have a general idea of what I am manufacturing:

"The novelty in my design is the use of virtual cells. While some people have attempted to create an economical Braille display using only one Braille cell, this is not functional as it does not allow the person with a disability to orient themselves on the page. It is like taking a sheet of paper, poking a hole in it, placing it over a book and attempting to read by only seeing one letter through the hole. It is very difficult to know where that letter is located - at the beginning or end of a paragraph, centered, or not - when the book is below the paper and you do not know its position. This is why Braille displays with only one Braille cell are not functional. Unfortunately, Braille cells are the most expensive component, and creating a solenoid Braille cell for 40 characters would result in a display that is heavy, consumes a lot of energy, and is therefore expensive and impossible to power with a high energy consumption. This is why Braille displays are so expensive, with some costing up to $6,000.

The novelty in my project is that it uses a single Braille cell that is touched with one finger to read, while the other hand moves over virtual cells, providing orientation to the text being read. This is similar to a touchscreen where you move your finger over the screen to get the sense of the orientation. As you move from one virtual cell to another, a small vibrating motor simulates the sensation of moving between cells. This is similar to the sensation you get when you slide your finger across the keys of a cell phone and feel a small vibration when you move from one letter to the next. This generates a feeling of movement between different cells, and as you move across the cells, the corresponding Braille dots appear on the physical Braille cell under the other finger. This allows you to read faster or slower, depending on your reading ability. If you don't read a letter correctly, you can move your finger back to read it again, and you can also tell if the text is centered, aligned to the left or right, providing a complete sense of what you are reading.

The touch cell is simply a touch screen or proximity sensor or tactile sensors that consume very little energy and weigh almost nothing. Therefore, it is perfectly feasible to make a single-cell device using any other system because only eight points are being used. This would result in low power consumption, making it possible to create a pocket-sized device without any problems."

[parkrrrr]

I'll see what I can do with the ESP32, then. I have a few here that I can experiment with. I haven't previously done anything with BLE HID, but I'll give it a go.

[discapacidad5]

Well, don't make it too complicated for yourself. If you can't work with the ESP32 and there's another microcontroller that you can use instead and that works, go ahead and use that other microcontroller. Then explain to me how I should configure the other controller and how I can upload modifications to my code, or provide me with a URL where I can learn how to do it. The important thing is that you manage to create that library, which is the most complicated part, because there are no examples anywhere for working with HID Braille that I can use for help. However, microcontrollers typically have support pages and instruction manuals that I can use for assistance. It might take a little longer, but I can manage. On the other hand, I have nowhere to turn for help with HID Braille. So, if you can't use the ESP32, feel free to use any other microcontroller that is easy to find and economical. The important thing is that you can make that microcontroller-HID and screen reader connection, and in the name of all the deafblind people in our community, we will be very grateful for your invaluable support.