Give NVDA the ability to automatically detect braille displays.

[nvaccessAuto]

Reported by jkenn337 on 2010-12-11 15:50 When a braille display is connected to the computer and brltty is installed, NVDA should scan for and automatically detect and start using the connected USB or bluetooth display. This way the user does not have to mess about with choosing their display and things. Orca and the mac already do this. This enhancement should be across all operating systems. Blocking #1555, #3648

[nvaccessAuto]

Comment 1 by ragb on 2013-09-02 10:33 Hi,

Recently, I have been encountering some users with difficulties understanding what braille display driver they should use, either because they don't really know the display they are using, or don't know the manufacture or something else.

So, I'll research something about automatically detecting USB / bluetooth displays (brltty may be a good start)

Some questions I can think of:

1. Do we want probing/active scanning, e.g. user presses a button and NVDA scans available devices, presenting possible connectable ones, or connects to the first found?
2. Do we want autodetect / passive scan, e.g. NVDA listens to devices being plugged and checks if it is a braille display or not?
3. Do we want both? :)

Both options are great of course, but implies more complexity and changes to the braille subsystem and display drivers.

426 has some interesting discussion about this.

[nvaccessAuto]

Comment 2 by jteh on 2013-09-03 01:05 I think both probing and detection would be nice. Probing would be used for serial displays and should warn the user about possible side effects.

Bluetooth is a tricky one. You can't passively detect Bluetooth displays. However, probing Bluetooth is safer than serial because Bluetooth ports are paired to a specific device. Still, it seems ugly to constantly probe for any paired Bluetooth displays.

[nvaccessAuto]

Comment 3 by ragb (in reply to comment 2) on 2013-09-05 10:24 Replying to jteh:

I think both probing and detection would be nice. Probing would be used for serial displays and should warn the user about possible side effects.

Agreed.

Bluetooth is a tricky one. You can't passively detect Bluetooth displays. However, probing Bluetooth is safer than serial because Bluetooth ports are paired to a specific device. Still, it seems ugly to constantly probe for any paired Bluetooth displays.

Unless we can find a way to by notified about bluetooth displays becoming available (that is without constant pooling) I guess probing is the best we can get.

As most drivers already implement some sort of automatic connection, I believe probing won't be very hard to do. Basic design (based on already discussed stuff on #426 and others):

1. Display drivers have a probe method and an associated canProbe property.
2. probe concret implementations should return a list of (port, description) pairs of any displays found (connection was successful). Description here is driver-dependent, but it should contain display name and number of cells, if possible. Note: unless one has two displays or a display can be used in to different communication ports, this list may always contain one element. 3 . When probing is requested (by the user) NVDA checks all displays for which canProbe is true and shows to the user all displays found for various drivers. (if just one is found we can simply choose to it, I guess).

Regarding the interface: I believe a "probe" button on the braille settings dialog, next to the display selection, which pops a "probing" dialog would be the least confusing solution for now. This dialog can alos be accessible from the wellcome screen. Before probing starts the user should be warned of any possible dangers probing may have, such as contacting serial ports and so on.

As regarding autodetection, I'll have to research a bit more about windows plug and play and so on to be able to propose something detailed.

[nvaccessAuto]

Comment 4 by ragb on 2013-09-24 10:29 Hi,

Should probing be used only for serial ports or we try also to connect to other available ports (USB and bluetooth, usually)?. With autodetection implemented, probing those may be useless, but we may not be able to implement auto detection for all drivers.

[nvaccessAuto]

Comment 5 by jteh on 2013-09-24 10:34 Probing all ports is fine.

[nvaccessAuto]

Comment 6 by leonarddr on 2013-09-24 11:11 If this functionality is added, i think it should be optional, as non-braille users don't care about it at all. Furthermore, it will probably create a lag on start-up.

[nvaccessAuto]

Comment 7 by ragb on 2013-09-24 11:37 Hi,

At first I was convinced that the best implementation was to have probing at the driver level. Here I think of two alternatives: Drivers have a probe method which, at the driver specific implementation, tries to connect to its ports. On the other hand, one could have a portsToProbe method which returns tuples of (port, priority) so we can probe "faster" ports firstly (USB mainly) for all drivers, before probing bluetooth and serial ports.

However, a much simpler alternative is to implement probing at the braille subsystem module, taking advantage of the (possibly implemented) getPossiblePorts} method in braille display driver classes. The probing procedure would be as follows:

1. Retrieve all available braille display drivers
2. For each driver, try to connect to all possible ports, except automatic (this is because automatic may contain some of the other ports, so we avoid duplication) 2.1. If the driver has no getPossiblePorts method, just try to initialize it and see if one can connect.
3. When a successful connection is made, report it to the user along with braille display type and cell count, asking if he wants to use this display. If so, end probing and set the found display in configuration.
4. If no connection was possible, or the user rejected all found displays, just return to braille settings dialog, or wherever probing was started from.

In theory, this solution makes all drivers "probing-ready", but we loose priorities and driver specific probing methods, which, at second thought, are not so so important due to probing being a rare procedure, I believe.

Comments?

[nvaccessAuto]

Comment 8 by ragb (in reply to comment 6) on 2013-09-24 11:42 Replying to leonarddr:

If this functionality is added, i think it should be optional, as non-braille users don't care about it at all. Furthermore, it will probably create a lag on start-up.

This won't be called at startup at all. It must be always requested by the user.

[nvaccessAuto]

Comment 9 by jteh on 2013-09-24 12:46 I don't think there's any need to implement a portsToProbe method, as getPossiblePorts tends to be ordered such that more common/faster ports are first anyway.

There are two problems with doing as you suggest:

1. Probing could be very slow (particularly if it touches Bluetooth ports), so this probably needs to be done in a background thread so the user can cancel it. You'll run into cross-thread issues if you initialise a display in a background thread and then use it from the main thread. I guess you could initialise it in the background thread just for the purposes of probing, terminate and then re-initialise in the main thread, but that's pretty ugly.
2. It doesn't give us a nice way to perhaps do background polling for Bluetooth displays. I guess we could solve this in the same way, though.

[nvaccessAuto]

Comment 10 by ragb (in reply to comment 9) on 2013-09-24 14:01 Replying to jteh:

I don't think there's any need to implement a portsToProbe method, as getPossiblePorts tends to be ordered such that more common/faster ports are first anyway.

My idea was more of trying faster ports from all drivers first, and then slower ones from all drivers, so the priorities thing. But honestly it is kind of over-complicating.

There are two problems with doing as you suggest:

1. Probing could be very slow (particularly if it touches Bluetooth ports), so this probably needs to be done in a background thread so the user can cancel it.

I forgot to mention this. Probably we spawn an IndeterminatedProgressDialog and do the probing in a background thread, allowing the user to cancel it.

You'll run into cross-thread issues if you initialise a display in a background thread and then use it from the main thread. I guess you could initialise it in the background thread just for the purposes of probing, terminate and then re-initialise in the main thread, but that's pretty ugly.

It is ugly indeed, but honestly I can't find a cleaner solution :$.

1. It doesn't give us a nice way to perhaps do background polling for Bluetooth displays. I guess we could solve this in the same way, though.

Maybe... We could restrict probing for bluetooth ports and having a background thread polling them from time to time, when no braille display is selected.

[nvaccessAuto]

Comment 11 by ragb (in reply to comment 9) on 2013-09-24 17:02 Remember something else:

Replying to jteh:

1. It doesn't give us a nice way to perhaps do background polling for Bluetooth displays. I guess we could solve this in the same way, though.

We could do bluetooth pooling this way. Moreover ,we could also do USB pooling in the background, avoiding dealing with plug and play, USB ids, etc (has I believe OS X and brltty in linux are doing). It's lesse eficient but way simpler.

[nvaccessAuto]

Comment 12 by jteh (in reply to comment 11) on 2013-09-24 21:57 Replying to ragb:

Moreover ,we could also do USB pooling in the background, avoiding dealing with plug and play, USB ids, etc (has I believe OS X and brltty in linux are doing).

Pretty sure BRLtty uses udev to handle plug and play for USB.

It's certainly simpler, but it has the disadvantage that it can only be done periodically to avoid performance issues. If I plug in a USB display, I would expect it to be detected immediately, rather than having to wait. With Bluetooth, you don't expect this so much because Bluetooth is wireless.

[nvaccessAuto]

Comment 13 by jteh on 2013-09-25 05:42 I'm starting to warm to this idea a bit more. It's at least a starting solution which we can improve upon later if necessary.

For auto detection, we can periodically try to create all displays with the default port in a background thread. Windows notifies about device connection via the WM_DEVICECHANGE message. Eventually, we could detect this and trigger a search instead of waiting.

The only problem is that there are some drivers which succeed even if a device isn't connected such as brltty and handyTech. Brltty only succeeds if brltty is running, but that could still be an issue. I guess we could have an attribute which disables automatic detection. We probably need to fix handyTech so that it doesn't do this; it really shouldn't.

[nvaccessAuto]

Comment 14 by ragb (in reply to comment 13) on 2013-10-06 17:36 Replying to jteh:

For auto detection, we can periodically try to create all displays with the default port in a background thread. Windows notifies about device connection via the WM_DEVICECHANGE message. Eventually, we could detect this and trigger a search instead of waiting.

I've been playing with this lately (see branch 51271, for a start). What would be the best approach To listen to WM_DEVICECHANGE? In the FS driver, for instance, we create a dummy window to listen to windows messages, would something like this work or there is a better lace? Windows API IPC is not one of my favorite topics in programing, definitely :).

The only problem is that there are some drivers which succeed even if a device isn't connected such as brltty and handyTech. Brltty only succeeds if brltty is running, but that could still be an issue. I guess we could have an attribute which disables automatic detection. We probably need to fix handyTech so that it doesn't do this; it really shouldn't.

Agreed. See canProbe attribute on branch t1271. I used that for nobraaille but brltty and Handitech should have it as well. On a side note, it would be very very nice from handitech to contribute the code they seem to be developing, whatever that is :).

[nvaccessAuto]

Comment 15 by jteh (in reply to comment 14) on 2013-10-07 23:39 Replying to ragb:

I've been playing with this lately (see branch 51271, for a start).

That's a very nice start. :)

What would be the best approach To listen to WM_DEVICECHANGE? In the FS driver, for instance, we create a dummy window to listen to windows messages, would something like this work or there is a better lace?

I wonder whether we need to have something that manages a window for this and other things in future. This probably needs to be handled in a separate ticket.

Windows API IPC is not one of my favorite topics in programing, definitely :).

Nor one of mine. :)

The only problem is that there are some drivers which succeed even if a device isn't connected such as brltty and handyTech.

Agreed. See canProbe attribute on branch t1271. I used that for nobraaille but brltty and Handitech should have it as well.

I just realised those drivers return 0 for numCells when they didn't detect a display, so we could make those drivers throw an exception if this occurs. That way, they can still support probing.

On a side note, it would be very very nice from handitech to contribute the code they seem to be developing, whatever that is :).

Yeah. It might be time to follow this up. They definitely need to release the code to honour the license and it has been several months now.

Once you get to background automatic detection, we need to think about UI and persistence.

1. How does the user specify that they want automatic detection? Is it a checkbox or an additional option in the braille display list? A checkbox perhaps seems nicer, but it makes the next question a bit tricky.
2. Once a display is detected, does this cause the config to be modified? I guess it probably shouldn't, as that will make that display permanent.
3. Once a display is detected, what should appear as the configured display in the Braille Settings dialog? Currently, the selected display is the one in the config. If we want the detected display to be selected, this will require some changes depending on the answer to (2).
4. We probably need to fix #1555 to make automatic detection truly useful.

[nvaccessAuto]

Comment 16 by ragb (in reply to comment 15) on 2013-10-09 21:54 Replying to jteh:

Replying to ragb:

What would be the best approach To listen to WM_DEVICECHANGE? In the FS driver, for instance, we create a dummy window to listen to windows messages, would something like this work or there is a better lace?

I wonder whether we need to have something that manages a window for this and other things in future. This probably needs to be handled in a separate ticket.

Agreed. We may need to register various handlers from many places in NVDA's code, so a central window where one registers handlers for various messages may be useful. I'll submit a ticket for this.

The only problem is that there are some drivers which succeed even if a device isn't connected such as brltty and handyTech.

Agreed. See canProbe attribute on branch t1271. I used that for nobraaille but brltty and Handitech should have it as well.

I just realised those drivers return 0 for numCells when they didn't detect a display, so we could make those drivers throw an exception if this occurs. That way, they can still support probing.

I like this option. It makes all drivers more consistent.

Once you get to background automatic detection, we need to think about UI and persistence.

1. How does the user specify that they want automatic detection? Is it a checkbox or an additional option in the braille display list? A checkbox perhaps seems nicer, but it makes the next question a bit tricky.

I like the checkbox option. Do we need to have autodetection always running or only when nobraille is selected?

1. Once a display is detected, does this cause the config to be modified? I guess it probably shouldn't, as that will make that display permanent.

I'm not really sure about this. If we don't set this, ton the next restart, the display may not be detected right away and the user will have to wait for probbing (if probbing is fast enough it may not be a problem). However, if we set it in the config, the user will have an error next time NVDA starts and the display is disconnected. If probing is not that fast, I'd say that setting detected displays in the config would be my choice.

1. Once a display is detected, what should appear as the configured display in the Braille Settings dialog? Currently, the selected display is the one in the config. If we want the detected display to be selected, this will require some changes depending on the answer to (2).

As you say, it depends on 2 :).

1. We probably need to fix #1555 to make automatic detection truly useful.

Yes. If a display starts throwing errors it should be disconnected and auto detection put to work.

Regards,

Rui Batista

[nvaccessAuto]

Comment 17 by jteh (in reply to comment 16) on 2013-10-15 04:30 Replying to ragb:

1. How does the user specify that they want automatic detection? Is it a checkbox or an additional option in the braille display list?

I like the checkbox option. Do we need to have autodetection always running or only when nobraille is selected?

On further thought, I'm not sure it makes sense to allow the user to manually select a display while automatic detection is enabled; see below.

1. Once a display is detected, does this cause the config to be modified?

I'm not really sure about this. If we don't set this, ton the next restart, the display may not be detected right away and the user will have to wait for probbing (if probbing is fast enough it may not be a problem). However, if we set it in the config, the user will have an error next time NVDA starts and the display is disconnected.

Are you saying that automatic detection would resume if the previously configured display threw an error? To me, this seems odd and difficult to explain. It's one of the reasons I think manual and auto should be mutually exclusive.

My feeling is that plug-and-play detection should be mutually exclusive with manual selection to avoid confusion. For example, if the user manually selects a display but then plugs in another one, what should NVDA do? If they are mutually exclusive, then it probably makes most sense for this to be a choice in the braille display list.

Of course, the user might want to probe for/detect their display just once; e.g. if they want to save the selection but don't know which display they have or which driver/port to use. My only concern is that manual probing will intrusively probe arbitrary serial ports, which the user might not want. This suggests we need an option to just probe USB and Bluetooth, but then this starts to get confusing for the user. Arrrg!

[nvaccessAuto]

Comment 19 by jteh on 2013-11-19 05:23 Rui, are you likely to be able to work on this in the next month or so? If not, are you happy for me to take it?

[nvaccessAuto]

Comment 20 by ragb (in reply to comment 19) on 2013-11-24 12:18 Replying to jteh:

Rui, are you likely to be able to work on this in the next month or so? If not, are you happy for me to take it?

Hi, I'd like to say yes, I will do it, but I'm not really sure if I have the time (software estimates, software estimates...). As yo know, I already have a branch with some code. I'll try hacking into it a bit more next weeks, at least the non-gui part (we both hate that). If you want to take it, of course I'll be happy with it, as I' really not able to promise anything.

Regards,

Rui Batista

[nvaccessAuto]

Comment 22 by jteh on 2013-12-04 08:28 Rui, I'm taking this for now, but feedback and code is still very much welcome!

[nvaccessAuto]

Comment 23 by jteh on 2013-12-05 05:25 How important is manual probing for displays; e.g. serial? it's a valid feature, but I think we should implement it separately once this is complete. The reasons are that i think it is less important (so lower priority) and I don't see any reason to block plug-and-play for it.

[nvaccessAuto]

Comment 24 by ragb (in reply to comment 23) on 2013-12-05 11:28 Replying to jteh:

How important is manual probing for displays; e.g. serial? it's a valid feature, but I think we should implement it separately once this is complete. The reasons are that i think it is less important (so lower priority) and I don't see any reason to block plug-and-play for it.

Plug-and-play is far more important, IMO. Serial ports are an older technology, although I believe we should support it, in this case I don't see it as a huge priority.

BTW, thanks for taking this, I'm really struggling with IOS development these days...

Regards,

Rui Batista

[nvaccessAuto]

Comment 25 by ragb on 2013-12-11 11:39 Hi,

Have just read the last commits. Here is some feedback:

1. I was expecting many more changes to my initial probing implementation :). I see you are caching all ports when _probe is called. I suppose it is to be used only for checking if there are ports to poll, and poll at time intervals when there are. However, imagine the following situation with the FS driver, as an example:
   • You don't have any display connected via bluetooth, so no ports are to be polled.
   • You pair the display with the computer, or it gets connected somehow (user enables bluetooth for instance).
   • AS there were no ports to poll, the prober is not triggered unless a device notification is sent (which may happen somehow when you pair or enable bluetooth, if so my problem doesn't exist).
   • In this situation, does the user have to re-enable automatic detection, restart NVDA, or something? Or am I missing something obvious? :).
2. I like the port refactoring. We had many duplication in automatic port handling in some drivers, and this makes this responsibility of the braille subsystem itself, so code is not scattered. It also simplifies probing of course.
3. There are some strings drivers (handitech for instance) which maybe tricky to rewrite. I may be able to hack on the braille note driver, if my device gets to work :).

The overall code seems very very good to me :). Manual probing, if gets to be done, is very straight forward with this.

[nvaccessAuto]

Comment 26 by jteh on 2013-12-11 12:29 Pairing a device will add a Bluetooth com port, which will be treated as a device change, thus triggering a notify/poll probe, so that bit should be fine.

Stupidly, now that I've pretty much got all of the core code working, I've realised there is one major problem I'd never really considered. This means that all drivers will be imported whenever automatic detection is enabled. Since we want to do this by default, this is probably pretty bad. It seems to only increase memory footprint by a few mb, but this is still not nice.

So, believe it or not, I'm considering throwing all of this code away and starting from scratch. My current thinking is to have a central module which maps all relevant USB IDs and Bluetooth names to their corresponding drivers. This way, the driver only needs to be imported if a matching device is detected. To make things easier, the driver will contain this info in a specific format and the build system will scan all drivers and generate this central module. This should also be more efficient than constantly calling drivers.

If we do go with this, it pretty much eliminates the need for most of my current changes. The ports stuff is nice, but it probably doesn't serve any actual purpose with this proposed approach. This also avoids issues with strange drivers such as handyTech. Of course, this doesn't take manual probing into account. We'll probably go back to something more like your original code for that.

Thoughts?

[nvaccessAuto]

Comment 27 by ragb (in reply to comment 26) on 2013-12-11 13:55 Hi, Replying to jteh:

Pairing a device will add a Bluetooth com port, which will be treated as a device change, thus triggering a notify/poll probe, so that bit should be fine.

Ok, nice.

Stupidly, now that I've pretty much got all of the core code working, I've realised there is one major problem I'd never really considered. This means that all drivers will be imported whenever automatic detection is enabled. Since we want to do this by default, this is probably pretty bad. It seems to only increase memory footprint by a few mb, but this is still not nice.

It depends on how many are "few MB" :). As far as I know one can't properly unload modules in Python....

So, believe it or not, I'm considering throwing all of this code away and starting from scratch. My current thinking is to have a central module which maps all relevant USB IDs and Bluetooth names to their corresponding drivers. This way, the driver only needs to be imported if a matching device is detected. To make things easier, the driver will contain this info in a specific format and the build system will scan all drivers and generate this central module. This should also be more efficient than constantly calling drivers.

In theory it is more efficient and avoids the unneeded imports. However, can we know all the device ids for all drivers somehow? Won't it be kind of a maintenance nightmare?

If we do go with this, it pretty much eliminates the need for most of my current changes. The ports stuff is nice, but it probably doesn't serve any actual purpose with this proposed approach. This also avoids issues with strange drivers such as handyTech. Of course, this doesn't take manual probing into account. We'll probably go back to something more like your original code for that.

Yes. Note that I like your approach, as far as I know bratty uses something similar in Linux, but I'm afraid of complexities like manufactures don't providing proper USB ids (and we having no access to devices or windows device drivers to get them).

Thoughts?

Above :).

Regards,

Rui Batista

[nvaccessAuto]

Comment 29 by bramd on 2014-12-21 09:42 What's the status of this? I would like to see this implemented and would like to write some code to get it done.

I use a Brailliant display on the road and a Handytech display at the office. The Handytech driver probes quite aggressively (might have something to do with my current driver settings as well...). So, a system that only initializes that driver if a Handytech device is connected and switches to my brailliant automagically is very welcome.

[nvaccessAuto]

Comment 30 by jteh on 2014-12-21 23:36 The current code is in the branch t1271a2. I got quite far with it, but got pulled away to other work and haven't had a chance to revisit it.

To be honest, I actually can't quite remember what problems still exist, though I know there was still work to do. I think one major problem is that there are some displays which stupidly use generic USB ids which might also be used by other displays, so the code needs to support trying more than one driver per USB id, which it doesn't currently support.

[nvaccessAuto]

Comment 31 by jteh on 2014-12-21 23:38 Oh, and of course, some of the drivers still need to be updated and detection data added. In particular, auto detection of Handy Tech is not yet supported and I also need to make the driver fail properly when no display is connected instead of succeeding and silently waiting for a display. This is all fairly tricky since I don't have access to a Handy Tech device.

[nvaccessAuto]

Comment 32 by dkager on 2015-08-09 11:09 This is another really useful feature. I still have to look at the diff, but there's some good stuff in the comments for sure.

I find this most interesting for the use case of unplugging your display, say to move your laptop somewhere, then re-attaching it and being able to continue working without restarting NVDA. Thus even if you were to turn off auto-detection it would be nice if individual drivers supported re-connecting. Or of course you could leave auto-detection enabled and that should take care of things. This leads to the suggestion that auto-detection always starts with the previously active display.

Keeping the above in mind I think the configuration file should save the last detected display, if only to avoid probing for other ones. It is likely that the previous display is still the current one, e.g. on a desktop PC with a braille display.

Another way to avoid importing all drivers (and maybe also some confusion) is to allow users to check the displays they use and only probe for those. The default would be to probe all displays.

Just a few initial thoughts. :)

[LeonarddeR]

Regarding the Hims driver, it seems that NVDA is currently unable to detect when a display is disconnected. This has the advantage that, when removing and re-inserting the device, the driver automatically picks up the display again. However, this conflicts with @bramd's desire to switch from one manufacturer's display to another's.

[jcsteh]

I'm not going to get to work on this before I leave NV Access :(, so here's a brief brain dump.

Most of the ideas (and even a lot of the code) in the t1271a2 branch should still apply. Here are some things that should probably be changed (based on new functionality added since the last work on this branch):

1. We should only support detection for displays which are declared as thread safe (isThreadSafe).
2. Rather than bdDetect using its own thread to probe, it should use the same background thread that is used for background writing to thread safe displays (braille._BgThread). Note that APCs can be used to queue functions to run in that thread. See braille._bgThread.queueApc (and its callers). This means we can initialise displays in the background thread, rather than queuing this to the main thread.
3. Previously, we had to check whether Bluetooth ports could be opened in the bdDetect thread before initialising the display in the main thread. This is because we didn't want to block the main thread if the device wasn't present. This could result in the display "flickering" before connecting, as well as taking longer to connect. If the init is done in the background thread (as per point 2), we can just init the display without the other probe, since we don't have to worry about it blocking.
4. Some displays use generic USB ids which might also be used by other displays, so the code needs to support trying more than one driver per USB id, which it doesn't currently support.
5. If we're doing the init in the background thread (point 2), trying multiple drivers (as required by point 4) should be a lot easier.
6. Rather than having a CustomWindow in bdDetect which handles WM_DEVICECHANGE, we should consider watching for this message in core.CustomWindow and using an extensionPoints.Action to report that there was a hardware change. This way, other components/add-ons can be easily notified about hardware changes.
7. In order to support detection, a driver must fail to initialise if a device is not present and must report an error when trying to write to the display once it is disconnected. Several drivers don't do one or both of these (e.g. Handy Tech). They either need to be updated accordingly or they must be excluded from support for detection.
8. More data (USB ids and Bluetooth names/addresses) needs to be added to bdDetect for various displays. BRLTTY is generally a good source of info, failing good docs on this from the manufacturer.

[LeonarddeR]

Thanks for your brain dump, @jcsteh, that will be helpful.

@MichaelDCurran, is this likely to be picked up by NV Access in the near future? If not, how about other contributors interested in making Jamie's brain dump a reality? cc @dkager @bramd

[michaelDCurran]

Sadly it is not on the roadmap for now.

However, as always, contributions are welcome, especially with this as there is existing code and thoughts from Jamie.

[LeonarddeR]

Here is @jcsteh's list in a task list

1. [x] We should only support detection for displays which are declared as thread safe ( isThreadSafe ).
2. [x] Rather than  bdDetect  using its own thread to probe, it should use the same background thread that is used for background writing to thread safe displays ( braille._BgThread ). Note that APCs can be used to queue functions to run in that thread. See  braille._bgThread.queueApc  (and its callers). This means we can initialise displays in the background thread, rather than queuing this to the main thread.
3. [x] Previously, we had to check whether Bluetooth ports could be opened in the  bdDetect  thread before initialising the display in the main thread. This is because we didn't want to block the main thread if the device wasn't present. This could result in the display "flickering" before connecting, as well as taking longer to connect. If the init is done in the background thread (as per point 2), we can just init the display without the other probe, since we don't have to worry about it blocking.
4. [x] Some displays use generic USB ids which might also be used by other displays, so the code needs to support trying more than one driver per USB id, which it doesn't currently support.
5. [x] If we're doing the init in the background thread (point 2), trying multiple drivers (as required by point 4) should be a lot easier.
6. [x] Rather than having a  CustomWindow  in  bdDetect  which handles  WM_DEVICECHANGE , we should consider watching for this message in  core.CustomWindow  and using an  extensionPoints.Action  to report that there was a hardware change. This way, other components/add-ons can be easily notified about hardware changes.
7. [x] In order to support detection, a driver must fail to initialise if a device is not present and must report an error when trying to write to the display once it is disconnected. Several drivers don't do one or both of these (e.g. Handy Tech). They either need to be updated accordingly or they must be excluded from support for detection.
8. [x] More data (USB ids and Bluetooth names/addresses) needs to be added to  bdDetect  for various displays. BRLTTY is generally a good source of info, failing good docs on this from the manufacturer.

I updated this code to current master in the i1271 branch @babbagecom

[LeonarddeR]

@jcsteh commented on 30 aug. 2017 07:32 CEST:

1. We should only support detection for displays which are declared as thread safe (isThreadSafe).

Are Alva BC640 and Freedom Scientific considered tread safe, even though this property is not yet declared on them?

7. In order to support detection, a driver must fail to initialise if a device is not present and must report an error when trying to write to the display once it is disconnected. Several drivers don't do one or both of these (e.g. Handy Tech). They either need to be updated accordingly or they must be excluded from support for detection.

Done in #7458. Also, #7459 will support this.

8. More data (USB ids and Bluetooth names/addresses) needs to be added to bdDetect for various displays. BRLTTY is generally a good source of info, failing good docs on this from the manufacturer.

This is actually part of your current implementation I tend to disagree with. Personally, I belief usb ids and bluetooth names shouldn't be placed in bdDetect itself, but rather as properties on the braille display driver classes. E.g. for Baum:

deviceIds={
    hwPortUtils.DEVICE_TYPE_BLUETOOTH:
    lambda m: any(m.name.startswith(prefix) for prefix in (
        "Baum SuperVario",
        "Baum PocketVario",
        "Baum SVario",
        "HWG Brailliant",
        "Refreshabraille",
        "VarioConnect",
        "BrailleConnect",
    ),
    hwPortUtils.DEVICE_TYPE_HID: {
        "VID_0904&PID_3001", # RefreshaBraille 18
        "VID_0904&PID_6101", # VarioUltra 20
    }
}

This also takes away the need for a globalPlugin as part of a custom braille display driver, among other benefits.

[jcsteh]

Neither alvaBc6 or freedom-Scientific have been verified as being thread safe. First, at least one of them relies on a windows message loop and the background thread doesn't currently run one, so that possibly won't work. Even if it works now, it will probably fail once displays get init in the background thread, since the window will get created in the init thread. Second, because these both use a proprietary dll, we can't know anything about thread safety without testing. Regarding your dislike of the bdDetect approach, having this info in the driver itself is a lovely idea... until you realise that requires all drivers to be imported. That's pretty hideous and wasteful, especially since some of these might load dlls at import (though you could move that into the constructor to avoid that particular issue). My first attempt at this suffered from this exact problem before I realised this. :)

[jcsteh]

Btw, yes, we do import all drivers when you open the braille settings dialog. However, a user won't always do that and I think it's reasonable in that instance. In contrast, importing every driver on startup seems pretty ugly to me.

[LeonarddeR]

@jcsteh commented on 1 Sep 2017, 22:20 CEST:

Neither alvaBc6 or freedom-Scientific have been verified as being thread safe. First, at least one of them relies on a windows message loop and the background thread doesn't currently run one, so that possibly won't work. Even if it works now, it will probably fail once displays get init in the background thread, since the window will get created in the init thread. Second, because these both use a proprietary dll, we can't know anything about thread safety without testing.

Does that mean you belief these displays should be excluded from auto detection, in contrast to your initial implementation? Or should we support non-trheadsafe displays and just initialize them on the main thread?

Regarding your dislike of the bdDetect approach, having this info in the driver itself is a lovely idea... until you realise that requires all drivers to be imported. That's pretty hideous and wasteful, especially since some of these might load dlls at import (though you could move that into the constructor to avoid that particular issue). My first attempt at this suffered from this exact problem before I realised this. :)

Hmm, sounds reasonable. However, it feels extremely ugly to me to have hardware identifiers declared twice, once in the display driver and once in bdDetect.

How about creating an additional module called hwDefs, which would basically contain the code for hardware identification which is now in bdDetect? This way, braille display drivers can be modified to get their hardware id's from the new hwDefs module instead of declaring the identifiers in the braille display modules themselves. Having said that, I don't think that there's a problem with having hw ids declared twice for the initial implementation, as long as we agree that hw ids in the braille display modules should be phased out when auto detection has landed.

[LeonarddeR]

@leonardder commented on 2 Sep 2017, 09:14 CEST:

How about creating an additional module called hwDefs, which would basically contain the code for hardware identification which is now in bdDetect?

Ah, forget this. We can just give bdDetect a broader scope, not just for auto but also for manual detection.

[LeonarddeR]

@jcsteh: I'm intending to change the behavior of hwPortUtils.listUsbDevices to yield dictionaries (like listHidDevices) instead of regular strings, which takes away the need to dive into the registry in order to get the device path, for example in the case of Hims devices. listUsbDevices is currently used nowhere in the code of NVDA, however I'd like to know what you think about this before I make this decision and incorporate it in the code of my i1271 branch.

[jcsteh]

@leonardder commented on 2 Sep 2017, 17:14 GMT+10:

Does that mean you belief these displays should be excluded from auto detection, in contrast to your initial implementation? Or should we support non-trheadsafe displays and just initialize them on the main thread?

I don't think we should support non-thread safe displays. They either need to be made thread safe (which requires testing and maybe some windows message loop code for the background thread) or they need to be excluded (which, yes, would be contrary to my initial implementation). We can't keep maintaining backwards compatibility layers forever.

Hmm, sounds reasonable. However, it feels extremely ugly to me to have hardware identifiers declared twice, once in the display driver and once in bdDetect.

Agreed. I think we could have some utility functions in bdDetect that drivers could call to get relevant identifiers. I do recall thinking about this, so I guess I just never got around to it.

Regarding your concerns about a global plugin being needed for a braille display driver add-on, perhaps we could allow identifiers to be specified somehow in the manifest. However, this gets a bit messier when you need a lambda to match Bluetooth ports, for example.

[jcsteh]

@leonardder commented on 3 Sep 2017, 03:10 GMT+10:

@jcsteh: I'm intending to change the behavior of hwPortUtils.listUsbDevices to yield dictionaries (like listHidDevices) instead of regular strings, which takes away the need to dive into the registry in order to get the device path, for example in the case of Hims devices.

I never imagined that device path would be useful, since it is the raw device path, rather than the path associated with some sort of communication layer such as HID or serial. However, your HIMS work makes it clear this wasn't a valid assumption on my part. :) So yes, this makes sense.

listUsbDevices is currently used nowhere in the code of NVDA

True enough. I also doubt any add-ons use this, so we shouldn't need to worry about backwards compat. Please make sure to mention this in the "Changes for Developers" section of your What's New entries, though.

[LeonarddeR]

@jcsteh commented on 4 Sep 2017, 01:26 CEST:

I never imagined that device path would be useful, since it is the raw device path, rather than the path associated with some sort of communication layer such as HID or serial.

How about the braillant BI? Is the raw device path somehow related to the device path that is retrieved in the driver for this display?

[jcsteh]

brailliantB is either HID or serial. For serial, you need the com port, which you get from the registry, but you never use the USB device path.

[LeonarddeR]

@jcsteh commented on 4 sep. 2017 01:22 CEST:

... and maybe some windows message loop code for the background thread

Could you elaborate on this? What would be a good starting point to implement this?

@dkager, would you be able to test thread safety of the Alva BC driver?

[dkager]

I would argue that testing can only prove that something is thread unsafe, not that it is thread safe.

[LeonarddeR]

@dkager commented on 5 Sep 2017, 18:34 CEST:

I would argue that testing can only prove that something is thread unsafe, not that it is thread safe.

I agree. However, there will probably be one point in time where we can safely assume thread safety. for example, I'm using the current handy tech driver as a thread safe driver, and it actually improves both speed and stability.

[dkager]

Multi-threading is a notoriously evil subject, though. Seems safer to write a native driver at some point, or at least ping Optelec to see what they have to say (very little I assume).

[jcsteh]

@leonardder commented on 6 Sep 2017, 00:48 GMT+10:

... and maybe some windows message loop code for the background thread

Could you elaborate on this? What would be a good starting point to implement this?

1. You first want to identify whether this is actually an issue, though I'm pretty sure it is. If you can't write or read from the display when it's initialised in the background thread, it's almost certainly this problem.
2. In braille._BgThread.func, we call SleepEx, which is an alertable wait. That needs to be changed to be an alertable wait that also dispatches window messages. To do that, you use the MsgWaitForMultipleObjectsEx function (though you don't actually need to wait on any handles).

[jcsteh]

@dkager commented on 6 Sep 2017, 05:09 GMT+10:

Multi-threading is a notoriously evil subject, though.

Certainly. However, if the driver works during normal usage when initialised and written to from our background thread, it's reasonable to assume it will be okay for our purposes.

Seems safer to write a native driver at some point

There are several good reasons to write a native alvaBc6 driver (avoidance of probing all serial ports, direct access to braille keys, etc.). We do have the tech docs to manage this (and we have a display as well). However, I don't think anyone has the cycles to work on it right now, so that is a factor worth considering.