Feature Request: External/Secondary Display Support

[anasik]

If:

• An external/secondary display is connected,
• E.g. a Smart TV or a monitor
• Via HDMI or screencast/miracast/chromecast/ with "force desktop mode" enabled

Then:

• The video should be displayed on the secondary display
• The phone screen acts as an input source.
  • Virtually everything should stay the same.
  • Screen acts as a touchpad unless two-finger tap to bring up keyboard
  • keyboard always shows special keys and everything since we now have more real-estate on the screen.
• All of this could of course be optional and enabled/disabled by a toggle button somewhere.
• Maybe when connecting it can detect any secondary displays and maybe ask if the user wishes to have video on either of them or the phone screen.

Potential Caveats:

• In my experience, secondary displays don't deal well with touch events being sent from the primary display. e.g. most trackpad apps (these can usually detect secondary displays) or bluetooth mice (connected via bluetooth to tv) don't work properly. The cursor moves but you can't click anything so it's useless.
  • Connecting a mouse via OTG to the phone however had the desired result.
  • There's an app called "Real Mouse" that literally simulates a physical mouse. That too worked perfectly.
  • We might need to go by the real mouse approach if the existing interface refuses to translate well.
• Normally, the on-screen keyboards don't exactly work across screens either. Meaning, if you click on an input box on your secondary screen, the on-screen keyboard will pop-up on the secondary screen itself, instead of your phone screen as would be ideal. Because both screens work as independent desktops.
  • This means that again we might have to go for a simulated hardware kind of approach where our keyboard module is pretending to be a physical keyboard. I am yet to test how well hardware keyboards work with secondary screens and when I do, I will edit this issue.
  • The Samsung Smart Things app, however, does this thing where for supported TV apps, it automatically opens an input box on your phone screen and that input box is synchronised with the input box in your TV app, and a keyboard on your phone pops up so you can send input from phone to keyboard. This kind of approach can also work. Meaning, if sending video to secondary display breaks the keyboard, we can add an extra middle layer that captures the keyboard input and deliberately sends it to the external display.

Lastly, I just wanna add that I would be more than willing to collaborate on this. I'm a little new to native android development, having always been overly comfortable with react-native, but I plan to study your code a bit and try to make the necessary changes myself if possible. If you and I communicate on this, we might be able to combine efforts and achieve results even faster.

[gujjwal00]

Hi @anasik , thanks for the detailed write-up. This seems like a nice feature to have when you have external display.

• I don't think we need to worry about input handling on secondary screen, we should just project the video. If user needs interaction, then they should launch the entire app on secondary display.
• For starter, we should just show a button in toolbar whenever external display is detected. Logic/options for automatically switching can be added later.

Lastly, I just wanna add that I would be more than willing to collaborate on this. I'm a little new to native android development, having always been overly comfortable with react-native, but I plan to study your code a bit and try to make the necessary changes myself if possible. If you and I communicate on this, we might be able to combine efforts and achieve results even faster.

Sure, I am all for collaboration. But please understand that video rendering & input handling are the most complex areas of AVNC. They were not designed with external displays in mind. So refactoring these is going to be the main hurdle, instead of actual video projection on secondary display. I already have many issues in my backlog so this is not something I will actively work on. But if you want to give it a go, please go ahead. I am happy to answer any questions.

[gujjwal00]

Closing, as right now I don't have any plans to work on this. But, feel free to reopen if you want to.