Porting openpilot to other HW than Neo

[kozuch]

I think it could be interesting to port openpilot to some other hardware than the current Neo platform (OnePlus 3 or 3T) . Looks like all the essences are open source so this may actually not be of a big problem. However I myself have no experience in creating a custom Android ROM (=customizing NEOS). Anyone here interested in this topic? The general thought is openpilot could be used WITHOUT itself connecting to a car just as a front/lane assist (audible or visual warnings).

Parts: https://github.com/commaai/neo https://github.com/commaneos https://github.com/commaai/openpilot

Such porting may help this project to really take off since now the HW options are very limited and if ported than people may use it (and TEST IT!!!) on a smartphone (or completely other HW platform) they already own and would not need to spend $400 or more for hardware.

I would try to support off-the-shelf and plug&play hardware - to allow install (ok I mean ROM flashing) on stock (smartphone) devices without the necessity to build the CAN hardware themselves. Some branded quality CAN dongles may be supported (OBDLink or OpenXC dongles etc.) just for reading the CAN bus when testing...

I think generally a greater HW abstraction on all levels is what this project needs.

[ingenieroariel]

I have experimented with using openpilot in other platforms (osx, linux) and believe that apart from visiond, it is straightforward to use openpilot in other hardware. You would only need to write your own manager.py that starts a different visiond daemon and a different boardd daemon.

Your main obstacle is going to be running visiond outside of the OP3, that part ties this solution to that hardware and until there is an open source version of it, any conversation of porting to other platforms is kind of pointless.

[kozuch]

@ingenieroariel

I am sorry I know about the visiond binary blob but I completely forgot to mention it. I am not a low level coding geek but I think there the visiond binary may actually be portable to similar ARM processors. OP3 has a Qualcomm MSM8996 Snapdragon 820 CPU which is ARMv8-A architecture. From what I read binary compatibility works within same ARM architecture and even forward compatibility should work (code compiled on ARMv7 should actually run on ARMv8 etc.). If this is true that would open a whole bunch od devices to port to (see Wikipedia), This is why I was thinking about porting. If there would be problems within whole ARMv8 the I think porting to other Snapdragon 820 device (there is at least a dozen of other smartphones using this chip) may be much simpler. Also, there is the upcoming Dragonboard 820c.

[NeoFromMatrix]

While I can understand the benefits of running this on an arm smartphone, it might be very interesting to use a (linux) laptop for testing.

Has anyone comma currently running on an x86 device with external camera? Documentation for that seems kinda limited.

[jyl087]

Most of the opendrive code is written in Python. How does Neo specifically run this code? Does it include a Python interpreter, or is the OpenDrive pre-compiled into C and machine code (maybe using Cython??) to run as a normal Android binary? The test simulator runs Python directly under Linux. The instructions for running Neo are a bit cryptic; they say: "Install openpilot on a neo device by entering https://openpilot.comma.ai during NEOS setup." which redirects back to github. Any detail on the boot, loading and Python execution would be welcome!

[ingenieroariel]

@jyl087 it uses https://termux.com/

That gives them apt to install python and a lot of the requirements needed to bootstrap the system. The rest of the dependencies can be compiled.

They stripped android to basically just run that app, their NEOS looks for one file to start after boot and openpilot writes on that file instructions to run manager.py which starts each of the daemons, all of them just send and receive data between each other using zmq and capnp.

[jyl087]

@ingenieroariel if that's the case, couldn't OpenPilot theoretically run on ANY late-model Android phone as long as Termux is installed? Does the HAL abstract the camera enough to allow visiond to run, or is the OnePlus phone hardcoded into visiond?

[ingenieroariel]

You are asking too much about the closed sourced parts and I have not seen them :)

Try running it on another phone and let us know the results. I think you will be able to run everything except visiond as that one will require the dynamically linked libraries and probably has a basic camera driver statically linked. To be able to run the neural nets you will need a chip like that qualcomm snapdragon that comes with the oneplus.

I think visiond could be easily ported to other platforms if you are compiling it from source but I would not bet on taking that binary and trying to run it elsewhere.

-a

On May 10, 2017, at 10:36 AM, jyl087 notifications@github.com wrote:

@ingenieroariel if that's the case, couldn't OpenPilot theoretically run on ANY late-model Android phone as long as Termux is installed? Does the HAL abstract the camera enough to allow visiond to run, or is the OnePlus phone hardcoded into visiond?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jyl087]

Sorry if I've hijacked this thread... I can open a new one if that is deemed appropriate. I guess the thought of porting openpilot to "Android" counts as not Neo, right?

Anyway, I'm working through the list of packages listed on "requirements_openpilot.txt". I've managed to install everything except for "fastcluster". I'm seeing this error:

ImportError: dlopen failed: cannot locate symbol "__mulodi4" referenced by "/data/data/com.termux/files/usr/lib/python2.7/site-packages/numpy/core/umath.so"...

I also could not install "scipy". Maybe this is related. I take it that scipy and Android are not really best buddies at this stage: http://www.leouieda.com/blog/scipy-on-android.html

So, one question I have is, what comes loaded with Neo to make everything work? Is SciPy and fastcluster part of that ROM? Again, sorry if this is too far off topic for this thread.

[ingenieroariel]

scipy and fastcluster are part of that rom, if you just want to keep hacking until you see something try removing the parts that use scipy and the other lib in the code, you can cross compile them at a later stage by creating an apt repo to complement what termux provides.

My humble suggestion is for you to keep going at it until you can load the ui.

-a

On May 11, 2017, at 10:37 PM, jyl087 notifications@github.com wrote:

Sorry if I've hijacked this thread... I can open a new one if that is deemed appropriate. I guess the thought of porting openpilot to "Android" counts as not Neo, right?

Anyway, I'm working through the list of packages listed on "requirements_openpilot.txt". I've managed to install everything except for "fastcluster". I'm seeing this error:

ImportError: dlopen failed: cannot locate symbol "__mulodi4" referenced by "/data/data/com.termux/files/usr/lib/python2.7/site-packages/numpy/core/umath.so"... I also could not install "scipy". Maybe this is related. I take it that scipy and Android are not really best buddies at this stage: http://www.leouieda.com/blog/scipy-on-android.html

So, one question I have is, what comes loaded with Neo to make everything work? Is SciPy and fastcluster part of that ROM? Again, sorry if this is too far off topic for this thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[kozuch]

@jyl087

You are perfectly on topic - Android runs on great variety of HW so supporting different HW via Android is ok.

I just posted a related issue - a call to open source most of visiond except their deep neural network binary blob - see #96.

[jyl087]

@kozuch , I admire your call for the "," team to open source more of visiond! In the meantime, I'm just trying to understand what has already been put out. @ingenieroariel says that scipy and fastcluster are part of the Neo ROM, so Neo also seems to be a bit of a black box. I would love more information on how Neo is built (i.e. how all the required libraries are loaded) and how it boots up. Finally, I agree that the custom H/W should be replaceable with other interface boards such as IOIO.

[ingenieroariel]

I would not call NEO a black box, after you install it on a oneplus 3 you get sudo access and can poke around all you want, copy that compiled library, etc.

Maybe you can figure out a way to run the NEO rom on an emulator?

On Fri, May 12, 2017 at 11:50 AM, jyl087 notifications@github.com wrote:

@kozuch https://github.com/kozuch , I admire your call for the "," team to open source more of visiond! In the meantime, I'm just trying to understand what has already been put out. @ingenieroariel https://github.com/ingenieroariel says that scipy and fastcluster are part of the Neo ROM, so Neo also seems to be a bit of a black box. I would love more information on how Neo is built (i.e. how all the required libraries are loaded) and how it boots up. Finally, I agree that the custom H/W should be replaceable with other interface boards such as IOIO.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/commaai/openpilot/issues/63#issuecomment-301128645, or mute the thread https://github.com/notifications/unsubscribe-auth/AADW1wnZ6dViVfweirsKaaXPrzWKrII6ks5r5I3ZgaJpZM4MNIUW .

[jyl087]

@ingenieroariel oohh I'm getting close. Have installed scipy (via "its-pointless" repo) and everything else, EXCEPT for fastcluster. Still getting:

Collecting fastcluster==1.1.21 Using cached fastcluster-1.1.21.tar.gz Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/data/data/com.termux/files/usr/tmp/pip-build-9f3zep/fastcluster/setup.py", line 5, in import numpy File "/data/data/com.termux/files/usr/lib/python2.7/site-packages/numpy/init.py", line 142, in from . import add_newdocs File "/data/data/com.termux/files/usr/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in from numpy.lib import add_newdoc File "/data/data/com.termux/files/usr/lib/python2.7/site-packages/numpy/lib/init.py", line 8, in from .type_check import * File "/data/data/com.termux/files/usr/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in import numpy.core.numeric as _nx File "/data/data/com.termux/files/usr/lib/python2.7/site-packages/numpy/core/init.py", line 21, in from . import umath ImportError: dlopen failed: cannot locate symbol "__mulodi4" referenced by "/data/data/com.termux/files/usr/lib/python2.7/site-packages/numpy/core/umath.so"...

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /data/data/com.termux/files/usr/tmp/pip-build-9f3zep/fastcluster/

Any ideas?

[jyl087]

Just wanted to update folks in case they run into this issue. I thought I had tried this previously, but trying again with the command:

LDFLAGS=" -lm -lcompiler_rt" pip2 install numpy==1.11.2

worked to make the missing mulodi4 error when installing fastcluster.

[johnnwvs]

I have been following this thread with interest. Any recent news?

[kozuch]

@johnnwvs Sorry but no news from myself. Comma.ai does not seem to be interested to make to project more accessible in terms of cheaper hw options etc.

[geohot]

Doubt there exists hardware that is too much cheaper to run openpilot. Also the vision model expects that specific camera, and will likely not get good performance with a different one. Closed for now.

[kozuch]

What is the base HW for the current EON Dashcam DevKit (EON DDK-1)? Both OnePlus 3 and 3T have already been discontinued...

[ghost]

Thats what im saying. If the comma ai crew has them which seems logical. It would be nice to secure the newer one plus model

[MANOJ851]

can we have a cheaper variants of EON dashboard kit or custom ROM to port it to other android phones , i dont want to spend 700 $ purchasing EON kit thanks

[kozuch]

Bump here. The visiond was open sourced lately. I created #562 for new discussion.

[Smartype]

FYI. I am trying to port op to my google pixel 1. It took a lot of effects to get camerad running on AOSP (android 8.1, google officially provided).

p1 is equipped with imx378 and imx179 cameras. There is no ois in imx378, and the actuator is a different one, I am able to set pos/move focus. But the auto focus feature is not working yet. After copying some sos, the neural engine is fully working (CPU, GPU and DSP).

Talking about the camera differences mentioned by geohotz, the lane lines it not 100% match with roads when the phone is in the center of the wind shield. I think this is due to the resolution I chose and the lens of the camera. Do not know if I will get better after calibration. Anyway, if I tilt the phone a little, the lines can be perfectly matched.