search

viktorgino / headunit-desktop

HeadUnit Desktop is a car PC software built with Qt 5 and QML
https://github.com/viktorgino/headunit-desktop/wiki
GNU General Public License v3.0
287 stars 64 forks source link

RFE: AA Proxy mode #8

Closed lars18th closed 4 years ago

lars18th commented 7 years ago

Hi,

Great project! Congratulations!

I suggest to implement one AA Proxy Mode:

The idea is quite simple: the video/audio output is done using the AA protocol to the Car (native HU), and the rest of the data/interaction is passed through the Pi.

What is the objective?

1) Provide an alternative (not physical) output to the "headuni-desktop" app.

2) Impersonate the native HU using all the time the screen/touch/control to execute all the Apps implemented in the "headunit-desktop". And without losing the option to use AA in the phone.

3) Use the AA protocol over Wifi with the phone but the cable connection with the native HU.

4) Incorporate more channels, for example CAN BUS or ODB-II data obtained by other interfaces with the Pi and send this info from the "headunit-desktop" app to the Smartphone. This provides Extended data not implemented by the original car.

And several other useful stuff. You agree?

viktorgino commented 7 years ago

Hi

I've actually thought of something similar to this when I first came across mikereidis/headunit, but back then I didn't have a working AA capable head unit (still don't, but here is this), and only had an experimental client implementation by mikereidis.

With all the effort put in by the guys over at gartnera/headunit it is possible to do this and could be useful in a lot of scenarios. There is still a lot of programming to it, but I think doing a simple proof-of-concept should be straight forward.

Also I know this is a long shot, but this could be useful for the https://microg.org/ project.

If you are interested in working on this, just let me know. I have some ideas on how to do this.

lars18th commented 7 years ago

Hi @viktorgino ,

If you are interested in working on this, just let me know. I have some ideas on how to do this.

I'm interested on discussing about these ideas. Please, can you explain more?

viktorgino commented 7 years ago

It's mainly implementation details, such as getting OpenSSL act nicely, getting frames to send.

I've actually have a little snippet that's supposed to act as the SSL server (the code in headunit implements the client). I'll try to find it, as it is a good pointer on what needs to be done, plus there are some improvements for both sides.

Getting frames to send: I'd definitely stick with gstreamer. To start with just get frames from either a video source or from videotestsrc, plus you can overlay text on it with gstreamer to debug button presses and maybe display touches. As for getting the images from the Qt GUI, we could run the GUI in a headless X session and grab the output with ximagesrc.

Tweaking the libusb part: We have to come up with a way imitate "Android Accessory mode" with a Pi. Which I don't think is possible with the Pi 3, only with the Pi 0, because you want to act like an USB device, not as a host. This is an issue for later, because I'd go developing the whole thing using a TCP connection, then worry about USB (this is something that could be prototyped separately). Adding one more Pi to the system is probably a good thing anyway, because I think its a lot of processing that'd need to be done by a single board otherwise.

lars18th commented 7 years ago

Hi @viktorgino ,

Thank you for the explanation! Here my 5cents about it:

You agree on do some experiments in this way?

viktorgino commented 7 years ago

Several options exist for writing the a FrameBuffer and encode it as video stream Yes, the simplest way probably is gstreamer. videotestsrc just generates an ntsc test pattern with some static noise, which can the be grabbed, encoded to H264 and sent over to the HU. AA is using a simple H.264 video stream, right? Yes For the proxy mode, is required that the program can execute as a client and as server at the same time. You think this is feasible now? No, only the client part is implemented. (just to clarify in this context the client is the head unit, the server is the phone with the AA app on it)

Focusing on the "Android Accessory mode" with a Pi, I'm not sure if the USB slave support is required. It is required to work with an actual head unit as they will only have an USB host controller. A Pi 0 could be used to forward packages from network to the USB host (plus do some processing, such as encrypting the packages). When developing the AA server, we can test it using TCP rather than USB. The client side is already implemented here.

viktorgino commented 7 years ago

@lars18th the way the Android Open Accessory Protocol works is the following: The accessory IDs itself as an accessory, exchanges protocol version and asks the phone to start in accessory mode. The phone then reconnects to the accessory with different USB VID and PID depending on its support for debug bridge. When reconnected the phone will expose two bulk interfaces (in and out) and that's about it. (talking about connecting a device to a HU, the HU is the accessory, the Pi0 will be the phone)

The AA protocol part can then be used from gartnera/headunit with only a few extensions.

Vitalic66 commented 7 years ago

You guys have very good skills... Sounds great what you plan. So if I understand right, the Pi0 should become the AA host and the Pi3 would still be the client. But what do you mean with usb and wifi connection? Does it mean the real phone should only connect to Pi0 via wifi and this passes all video and audio packages via usb to the Pi3? That would be great because the problem with starting HU and not working because phone mit connected before start would make many things much easier. Or did I understand something wrong?

viktorgino commented 7 years ago

@Vitalic66 So if I understand right, the Pi0 should become the AA host and the Pi3 would still be the client. The Pi0 could connect to a headunit that supports Android Auto and stream the output of headunit-desktop or anything else But what do you mean with usb and wifi connection? AA does work through WiFi, but AFAIK it is not widely adopted. And if I remember correctly it only works if you enable developer mode on the phone the problem with starting HU and not working because phone mit connected before start would make many things much easier. This is not a problem on desktop, its only not working on the Pi, but it should be an easy fix. Still don't have an supported AA phone, so can't fix it ATM.

lars18th commented 7 years ago

Hi @viktorgino ,

Thank you for your comments!

only the client part is implemented. (just to clarify in this context the client is the head unit, the server is the phone with the AA app on it)

I don't see this point... so I re-review the goal (from my point of view):

I feel this can be an interesting project!

Vitalic66 commented 7 years ago

Just to understand it right, what is the big advantage of that? Saving ressources or more flexibilty to output to different in car displays?

viktorgino commented 7 years ago

@lars18th The objective is AA in proxy mode, so the environment will be: HU Native AA (car) <------> AAproxy in "server mode" (emulates AA) (Pi zero) <------> AAproxy in "client mode" (emulates HU) (Pi2/3 running headunit-desktop) <-------> Smartphone with AA This sounds good to you? Yes, this is the point of it. I was going into implementation details a bit more than I should've 😄 This is about acting as a head unit and a phone a the same time.

In proxy mode two modes can be possible: full re-streaming (no video processing, just bypass the H.264 stream), and full rendering (video encoding is done in the AAproxy). Essentially yes, but as soon as we got a working implementation for the AA server the possibilities are endless.

For the "server mode" (AA emulation) it's required one device with USB device (slave) support. This can be a Pi zero or a rooted Android device.

This is a simple "interface". It's right?

I'd prefer the Pi zero, but yes could be an Android device as well. Not sure about rooting, the API should be there for it. Once its done in C++, then it can be ported to Android using JNI. As for the Pi, I need to do some research on how to change the USB device IDs and create bulk endpoints.

viktorgino commented 7 years ago

@Vitalic66 Just to understand it right, what is the big advantage of that? This could help people install headunit-desktop, or per-se any other app just by plugging a Pi 0 in their AA enabled head unit. So basically you could extend the functionality of your HU without getting some other hack or just completely replace it (again you need a headunit with AA support and AA is being installed in more and more cars).

Vitalic66 commented 7 years ago

Oh ok. Thats why I didnt understand. My system is working another way. As i once wrote i got an extra board connected to the pi. Pibus.info This board sends the commands to the ibus of bmw and also sends the video signal to the nav screen. Output is done via av out signal. Thats why I don't need what you plan... At least not for bringing the video to the screen...

viktorgino commented 7 years ago

@Vitalic66

As soon as I'll have some more time to spend on this project I'll write up a guide on how to get an AV signal working. The CAN/IBUS control is a long shot, but have some ideas on how to get started on it.

Vitalic66 commented 7 years ago

It already works ... :) It's no problem to show the content in my nav screen. From the beginning of the boot i can see everything in the screen.

viktorgino commented 7 years ago

@Vitalic66 Didn't realise :D. Its only getting controls is the issue then. I've got pretty much know how I'm gonna go about it and keep it user-friendly at the same time, but will take some time to get an implementation done.

Vitalic66 commented 7 years ago

That is no problem too. Already found a solution for my car :)

Like this

http://m.ebay.com/itm/121521817413

plus

https://www.adafruit.com/product/1580

So i got a semi touch screen. It's found as usb mouse from Raspberry so i can control everything.

The only problem right now is DAB+ and that phone has to be connected before start.

lars18th commented 7 years ago

Hi @viktorgino ,

The first goal I think, it's a the most simple "proxy mode": that is, the simple re-streaming of all the information in both directions. So, the first technical challenge is the USB slave mode...

My suggestion is quite burden: one Linux device with usbgadget support connected to the car and with wifi support; and the "AA proxy App" running in a Pi. The real smartphone then connects to the Pi running the proxy over USB wire. The Linux device with usbgadget connects to the car as well with another USB wire. And betweeen the both middle devices (Pi and Linux) we can use wifi.

You agree with this complex environment?

viktorgino commented 7 years ago

@lars18th

I agree, the only problem is that the Pi 0 can either be an USB slave or host, but not both. You can connect peripherals to it via the GPIO pins, but it might be better to use two boards from the beginning and communicate through WiFi or maybe serial. Also in a dev environment you could substitute one Pi with a desktop to speed thing up a little.

Vitalic66 commented 7 years ago

Pi0 only has hdmi out ? So my option of av out gets lost. Perhaps there should be the possibility to choose what you want to compile. With pi0 or as standalone what i would prefere.

viktorgino commented 7 years ago

@Vitalic66 As it stands this will be a separate project. Will only use the Pi 0 as a way to connect any graphics application to an AA enabled headunit.

blezalex commented 4 years ago

I've got raspbpi working in proxy mode if anyone still interested. It emulates usb device using gadgetfs + functionfs and connects to a phone in AA dev mode, simliar to how this guy did it https://forum.xda-developers.com/general/paid-software/android-3-0-proxy-gateway-android-auto-t3813163