search

tiny-pilot / tinypilot

Use your Raspberry Pi as a browser-based KVM.
https://tinypilotkvm.com
MIT License
3k stars 252 forks source link

Add support for multiple target systems #81

Open owendelong opened 4 years ago

owendelong commented 4 years ago

The pi might be a little underpowered for multiple simultaneous KVM sessions, but it would be nice to be able to attach more than one HDMI capture device and then connect to whichever one was desired at a particular moment by using multiple instances of tinypilot running on the same host (whether PI or other).

owendelong commented 4 years ago

I plan to attempt to implement this, so if you wish, feel free to assign to me.

mtlynch commented 4 years ago

uStreamer is actually what manages the HDMI dongle to capture video. You can see how tinypilot integrates it here:

https://github.com/mtlynch/ansible-role-tinypilot/blob/2cef73c36805ffcde30b6828e078b49adb3cc16b/meta/main.yml#L23

I think multiple instances of uStreamer would be fine because it's pretty low-overhead.

I don't know how you'd map the keyboard impersonation to multiple machines, though. Do you have any thoughts there?

A more hardware-driven solution might be to pull in a basic, non-networked multi-port KVM (like this one) that supports switching target computers via hotkeys. Then you could connect something like this:

                   TinyPilot
                       |
                Legacy 4-Port KVM
                       |
      ----------------------------------------
      |            |           |              |
    Server 1    Server 2   Server 3      Server 4

You'd still be able to cycle between servers by pushing hotkeys from the TinyPilot web interface and the Legacy KVM would intercept them and cycle machines.

lars18th commented 4 years ago

Hi,

Regarding the idea of MULTIPLE target systems (in the sense of concurrent access), I have this suggestion:

I'm sure the current code can be compiled without much effort for multiple platforms including x86. Futhermore, the HDMI input over USB can work with VMs without troubles. And several cheap devices can be used for the USB Gadget part.

I understand the advantage of the "clean" All-In-One model of the current implementation of this project. And I agree with it. However, if you want to explore a MULTI-TARGET support, then I suggest to consider this way.

I hope it helps. Regards.

owendelong commented 4 years ago

I was thinking that the USB-Gadget might even end up being a Pi-0W. A $10 server dongle isn’t nearly as bad as (e.g. Raritan $500/dongle).

Owen

On Jul 27, 2020, at 00:23 , Lars The <notifications@github.com mailto:notifications@github.com> wrote:

Hi,

Regarding the idea of MULTIPLE target systems (in the sense of concurrent access), I have this suggestion:

The best option can be to provide a low coupled implementation between the core and the USB Gadget part (the manager of the USB KEYBOARD emulation). Using this way a central core can manage multiple HDMI capture streams and communicate (over the network) with a "slave" USB Manager. Then you can run, for example, the core inside a VM in a x86 server (very powerful) and multiple USB Managers in small Linux devices with USB Gadget support. I'm sure the current code can be compiled without much effort for multiple platforms including x86. Futhermore, the HDMI input over USB can work with VMs without troubles. And several cheap devices can be used for the USB Gadget part.

I understand the advantage of the "clean" All-In-One model of the current implementation of this project. And I agree with it. However, if you want to explore a MULTI-TARGET support, then I suggest to consider this way.

I hope it helps. Regards.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mtlynch/tinypilot/issues/81#issuecomment-664168451, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTWJ6JTAJJ5VM6MNM5TR5UTNPANCNFSM4PG5NVWQ.

lars18th commented 4 years ago

I was thinking that the USB-Gadget might even end up being a Pi-0W. A $10 server dongle isn’t nearly as bad as (e.g. Raritan $500/dongle).

Well, you actually need to count this way:

The best with this disacoupling of the USB-Gadget part is the flexibility to use different hardware solutions. I hope this will be implemented at some point. Regards.

owendelong commented 4 years ago

On Jul 29, 2020, at 00:02 , Lars The notifications@github.com wrote:

I was thinking that the USB-Gadget might even end up being a Pi-0W. A $10 server dongle isn’t nearly as bad as (e.g. Raritan $500/dongle).

Well, you actually need to count this way:

Core (physical device or virtual machine): variable cost (perhaps nearly zero if you reuse it) HDMI capture: ~$15 each server. USB-Gadget: $10/server if you use Pi-zero, but some other options can be used... more expensive or more cheap.

If you want to do that, then you also need to count the Raritan example this way:

Raritan base unit $3000+ Raritan dongle per server: $500

So the fair comparison is:

Core: Raritan core: $3000+ Core (Raspberry PI, VM, etc.): $50+ (PI+SD card, etc.)

Per server: Raritan Dongle: $500 HDMI Capture ($15) + USB-Gadget ($7 Adafruit Trinket or more) + Cabinetry (3d-print for ~$2)

Now what would be really cool (and I think it’s possible, but slightly beyond my abilities) would be if we could build a dongle device that was relatively cheap and looked like the following:

Key: [physical device] <-cable->

Diagram: TinyPilot Dongle [host (core)] <-USB->[USB-Hub] | +— HDMI Capture Device — <-HDMI-> [Target HDMI Port] — This connection receives video from target +— Enumerable Custom USB Device — <-USB-> [Target USB Port]. — This connection provides HID input to target (KB/Mouse)

The Dongle would make a single USB connection to the TinyPilot, but would have a USB and HDMI connection to the target. I believe that an Adafruit Trinket (or similar cheap microcontroller) could be made into the Enumerable Custom USB device. By that, I mean a device that appears to the TinyPilot as a USB Target device. This might be possible using a pair of CH554 chips (Dev boards available for ~$2) tied together by SPI. One to act as USB Device and take commands from host, the other to act as USB-HID Keyboard and Mouse to the target system. Another option might be a single CH559 (which might also eliminate the need for the hub)

The hub could be as simple as a Microchip USB2422 ($1.55 Q1, ≤$1.45 in quantity) with a few cheap passives (total probably <$2).

The HDMI capture device is probably not going to get any more economical than the existing proposals, so that could be incorporated (with or without its shell, depending on design requirements.

Add in connectors and PCB production and we’re probably looking at a total somewhere around The best with this disacoupling of the USB-Gadget part is the flexibility to use different hardware solutions.

I hope this will be implemented at some point. Regards.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mtlynch/tinypilot/issues/81#issuecomment-665474765, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTVBM3L4L5XRLEZ37QLR57CRBANCNFSM4PG5NVWQ.

lars18th commented 4 years ago

Hi @owendelong ,

Just to clarify the model:

So a best diagram can be this:

      CORE
       |
       |
    / USB /
       |
       |
       +--- HDMI 1       <<--- Server 1 Video Output
       |
       +--- USB-Gadget 1 <<--- Server 1 USB Keyboard/Mouse
       |
       +--- HDMI 2       <<--- Server 2 Video Output
       |
       +--- USB-Gadget 2 <<--- Server 2 USB Keyboard/Mouse
       #
       +--- HDMI n       <<--- Server n Video Output
       |
       +--- USB-Gadget n <<--- Server n USB Keyboard/Mouse

You agree?

lars18th commented 4 years ago

Futhermore, regarding the USB-Gadget module that's my opinion:

As you can see a lot of options are posible. I hope it helps.

owendelong commented 4 years ago

There’s certainly no requirement for them to be in one dongle. However, I think if you consider the history of KVM switches, you’ll see that the best designs do have a small package close to the target with a single cable back to the KVM core.

The single dongle approach would approximate that.

I do like the idea of an abstraction so that multiple underlying transports and end-points could be used to suit the operator’s individual needs. Mostly this amounts to defining an API and then building drivers to that API.

If we can get to a good hardware design for the dongle (ideally one that includes a USB port that the video capture device could be connected to), then I think we could begin defining the desired protocol for the USB interface version.

I think ideally, we implement the USB version as a TCP listener that drives the USB. In this way, each instance can be a different TCP port on localhost if we’re using direct USB, or run the listener on a remote system if that is desired.

This provides a generic TCP interface for the application which abstracts the underlying details and would facilitate the ability to also develop non-usb keyboards/mice, etc.

lars18th commented 4 years ago

Hi @owendelong ,

I agree with you in nearly all, however please note these aspects:

I hope in any case that @mtlynch will want to support your development. Regards.

mtlynch commented 4 years ago

Am I understanding correctly that this hub could exist independently of TinyPilot?

It sounds like you're describing a device that re-routes USB connections based on a control channel, so TinyPilot and the USB hub would be loosely coupled. As a very simple example, I'm imagining TinyPilot has a button in the web UI that says "cycle target in hub" and that causes the TinyPilot backend to send a message to the hub causing it to re-route the HDMI and USB connection to the next target system in the hub. That'd be pretty easy to integrate with.

I wouldn't have the bandwidth to produce the hub, but if someone can produce that hub, I'd be interested in doing the work necessary to allow TinyPilot to support it.

lars18th commented 4 years ago

Hi @mtlynch

Am I understanding correctly that this hub could exist independently of TinyPilot? [...]

That's not my idea. My idea is that just one CORE of TinyPilot will manage more than one server at the same time. To achieve this, multiple HDMI inputs will be used in parallel and multiple USB-Gadgets will be used too. That's very different to a KVM switch!

Why we need this? To have concurrent access to multiple servers, as this is not possible using a KVM switch.

I hope in any case you will agree with the idea of the abstract USB-Gadget device. Regards.

owendelong commented 4 years ago

I’m not sure I get your point…

The USB Capture device needs a USB connection back to the tiny pilot. The USB K/M connection needs a USB connection back to the tiny pilot. A small USB hub with both of them connected on the downstream port and the tiny pilot upstream is NOT going to add significant delay to the video. From a technical point of view, then, the Tiny Pilot has software talking to the USB Capture device (already exists) and a daemon to talk to the USB K/M emulator device (TBD).

If you know of an already implemented protocol for sending K/M over IP to a USB K/M emulator, then please provide pointers. To the best of my knowledge, does not exist yet.

Your last comment makes even less sense. If those functions in an API are capable of sending TCP, then it’s clearly possible to implement them as functions in a TCP protocol.

IMHO, at a minimum, we need to specify the following to have a complete API:

1.  The following functions:
        send_kbd_key_dn(uint8_t keycode)
        send_kbd_key_up(uint8_t keycode)
        send_mouse_data()
        recv_mouse_config() // I believe some systems expect to be able to set DPI and other things by sending to mouse. Someone who knows more than me, please clarify
        set_gpio_mode(uint8_t pin, enum mode)
        set_gpio_state(uint8_t pin, bool state)
        get_gpio_state(uint8_t pin)

2.  A TCP protocol which allows a daemon implementing those functions to be controlled by a remote device implementing the protocol.
        Ideally with some form of secure authentication to prevent rogue connections.
        All data sent should be fully encrypted using negotiated keys (whether asymmetric or DH negotiated symmetric).

I’m not saying we should hard-code or tie things to the single-dongle concept in any way. I’m saying it would be possible to use these building blocks as parts of a single dongle which could be easily and inexpensively constructed.

On Jul 31, 2020, at 00:12 , Lars The notifications@github.com wrote:

Hi @owendelong https://github.com/owendelong ,

I agree with you in nearly all, however please note these aspects:

The delay of the video capture is a critical point. So I prefer to not link the USB capture device with the USB-Gadget part. I agree that the "single dongle" approach for each server is conceptually very easy to understand and use. But from the technical point of view is a pain. So please, try to maintain them totally unlinked. Regarding the USB-Gadget API I suggest to use some already implemented protocol, and not define a new API. Futhermore, I'm not sure if the TCP interface needs to be the main communication channel (even it can be). Perhaps it will be more simple and flexible to define a simple interface of functions like: send_kbd_keys(), receive_kbd_data(), send_mouse_data() and so on. Then every implementation only requieres to adapt these functions to the hardware... using TCP, GPIO, USB or whatever. I hope in any case that @mtlynch https://github.com/mtlynch will want to support your development. Regards.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mtlynch/tinypilot/issues/81#issuecomment-666970846, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK6GTXS2USC2A7Z4ADUUSDR6JVGJANCNFSM4PG5NVWQ.

lars18th commented 4 years ago

Hi @owendelong ,

Sorry for my not good explanation. I'll retry:

That's all to comment. I hope it will be more clear now. :smile: Regards.

owendelong commented 4 years ago

On Aug 1, 2020, at 09:15 , Lars The notifications@github.com wrote:

Hi @owendelong https://github.com/owendelong ,

Sorry for my not good explanation. I'll retry:

Why you need one USB hub for each pair of HDMI+USB-Gadget ? My concern is about that pair doesn't have sense to be required. You can connect N HDMI Inputs with N USB-Gadget devices using the N ports of the CORE hardware or with just one USB HUB. That's the point I want to be noted. It’s not required, nor am I proposing baking that requirement into any of the software design here. However… It is convenient in the case of desiring to have N dongles that can be connected to N systems for a given core. It’s one possible implementation. My suggestion about an abstract USB-Gadget is to make the "daemon to talk to the USB K/M emulator device (TBD)" nearly abstract with the option to instantiate more than one instance with different implementations. My suggestion is similar. Let’s unpack this a little bit. I think we’re not as far apart as you appear to think.

My thinking is that we have:

  1. Management software — Mostly this exists already in Tinypilot. It runs on the “core” system and provides the end-user UI, allows for selection of the target system to interact with and provides a mechanism for configuring initial and additional target systems to be available in the UI. (Obviously it also provides mechanisms for changing or deleting targets).

  2. A mechanism for the management software to obtain video from the video capture device(s) and pass that along to the user when desired. Currently this is built into the core and is tightly coupled (capture interface must be attached to the same system as core via USB).

    Ideally, we would also be able to make this a loosely coupled interaction that could be
transported over IP (most likely UDP).

  3. A mechanism for the management software to manage a keyboard emulator.

    Ideally, this should be loosely coupled and available to the management system via
IP, most likely TCP.

  4. A mechanism for the management software to manage a mouse emulator.

    Ideally, this should be loosely coupled and avaialable to the management system via
IP, most likely TCP.

This creates a generic interface where the Management System can be either tightly or loosely coupled with any given target system(s). In fact, it could even be tightly coupled with some target(s) while loosely coupled with others.

While it is most likely that most of the target devices will be USB based, going direct from management system to USB has disadvantages… It makes it hard to extend the system over an IP network (which I believe is a desirable feature) and it limits you to the USB structure of a single host platform.

By decoupling the USB aspects form the management system via an IP protocol, it provides significant additional flexibility while not significantly increasing overhead.

Now, within the framework of the above capabilities, there are multiple possible implementations:

  1. USB devices directly attached to management system. Daemons implementing the IP<->USB functions are installed on the management system and each daemon instance controls one set of {video capture(s), keyboard, mouse}. The IP connection is via ::1/128 (or worst case 127.0.0.1). (TBD whether this is best implemented as a “KVM” daemon, or a “KM” daemon and a “V” daemon, or three separate daemons.)

  2. Individual USB-connected Dongles per target system still directly connected to management system.

    Software sees no difference from (1) above. At the hardware implementation level, ti is convenient for each dongle to incorporate a small USB Hub-chip internal to the dongle. The upstream port on the hub faces the management system. The downstream ports on the hub are attached to the video capture device(s) and a USB-based keyboard/mouse emulator, which is subsequently connected to the target host and appears to the target host as a directly attached keyboard and/or mouse.

    I think it is desirable to have the keyboard emulator and the mouse emulator present as separate USB HID targets, but I am not firm in this opinion and admit that USB design is not my field of expertise.

  3. IPKVM Network

    For performance or capacity reasons, or simply because the systems that are to be managed are geographically diverse, it may be desirable to have IP-based target system interfaces (dongles as it were). These could be as simple as a PI0W or similar device attached to the host via a video capture and implementing the K/M emulator or as elaborate as one wants to get. The key is that daemons running on the target dongle would run the same IP protocol suite as the daemons in the example above, but instead of the management software connecting to them at ::1/128 (or 127.0.0.1), it would use a global unicast address to reach the target.

This allows development of different daemons to support different video capture hardware and/or keyboard and mouse emulators with a host of different attachment mechanisms as well as permitting the association between the management host and the target adapters to be either tightly or loosely coupled.

rvdwegen commented 4 years ago
  • HDMI capture: ~$15 each server.

Small sidenote, I bought one for about $7 shipped a few weeks ago.

djclueless commented 3 years ago

This isn't that difficult, the ustreamer runs on its own ip and port, so, technically you can pull that from anywhere as long as you have access over the network. The USB portion is just a python script running a socket server with its own ip and port, which again can be running on any piece of hardware that is connected to the same network.

The UI will need some tweaks for the configuration and actually store the settings.

You need a form to input information such as, The name of the instance, the url to reach ustreamer, and the ip:port for the socket server, and maybe a default / preferred checkbox.

You need a table to list the "KVM's" that are available, maybe include a column that indicates if they are online/offline and a "Connect to" button to switch over..

So for example setup would look like.. Name: TinyPilot 1 Streamer: 192.168.0.2:80 WS: 192.168.0.2:80 Default: checked

Name: TinyPilot 2 Streamer: 10.0.0.6:8080 WS: 10.0.0.6:80 Default:

owendelong commented 3 years ago

@djclueless Well, that covers the video side, but we still need something that allows us to emulate keyboard and mouse to multiple destinations. Because of limitations in USB (including USB OTG) hardware design and specification (it's a strict master/slave system and there's no provision for multi-master), we'd need a HostC(Master)->Target(Slave)->HostR(Master) type of set up where the Target(Slave) presents a USB Gadget to HostC and an emulated standard Keyboard and Mouse to HostR.

lars18th commented 3 years ago

Hi,

I feel that the most simple solution is to support a "remote USB OTG slave module". With a regular host (a x86 server, a Rasberry Pi, a Virtual Machine, etc.) with USB master support we can connect one or more HDMI capture devices. So the only problem is the USB OTG slave support to emulate the Keyboard/Mouse/MassStorage. Then, a simple protocol to drive a tiny device will be sufficient. Why not try with the Raspeberry Pico? https://www.raspberrypi.org/forums/viewtopic.php?t=299863