bmartin5692 / bumper

A standalone and self-hosted implementation of the central server used by Ecovacs vacuum robots.
GNU General Public License v3.0
302 stars 50 forks source link

add new endpoints for app v2 #116

Closed edenhaus closed 3 years ago

edenhaus commented 3 years ago

Adding the new endpoints for app version 2 Fixes https://github.com/bmartin5692/bumper/issues/114 Fixes https://github.com/bmartin5692/bumper/issues/107

Feel free to test it :). For testing purpose I have also created a new docker image edenhaus/bumper, which will be always up to date with this PR.

Testing instructions The easiest ways to test is using the docker-compose example below. Please adjust it for your needs **Note:** You need to recreate the certs as some new sub domains have been added. I have only tested the manual ways (see updated docs) **Currently only domains/urls for europe are included. Please comment if your app uses different ones** A docker-compose example can be found in the folder `example` The docker-compose example has an nginx proxy, which redirects mqtt traffic on port `443` to port `8883` The redirection is required as the app v2+ and robots with a newer firmware are connecting to the mqtt server on port 433.
edenhaus commented 3 years ago

Good news I was able to connect successfully with the app v2 to bumper with mitmproxy in between. I was able to run some basic commands (auto clean, locate,...) but until bumper supports the whole app it will take some time. Still some endpoints doesn't exist.

As the new version expects the MQTT server also on port 443, I have placed a nginx in the front. See #103 and #102 Here the relevant part of the nginx config (Thanks @puddly):

stream {
        map_hash_bucket_size 64;

        map $ssl_preread_server_name $internalport {
                # redirect all requests, which contain "mq" in the SNI -> MQTT
                ~^.*(mq).*\.eco(vacs|user)\.(net|com)$    8883;

                # the rest of eco(user|vacs) requests
                ~^.*\.eco(vacs|user)\.(net|com)$          443;

                # mapping default to MQTT as the bots are connecting directly to the ip without SNI
                default                                   8883;
        }

        server {
                listen 443;
                ssl_preread  on;
                proxy_pass 172.19.200.10:$internalport;
        }

        server {
                listen 5223;
                proxy_pass 172.19.200.10:5223;
        }

        server {
                listen 8007;
                proxy_pass 172.19.200.10:8007;
        }

        server {
                listen 8883;
                proxy_pass 172.19.200.10:8883;
        }
}

I will create a complete docker-compose example in the next days.

Ecovacs uses now a lot of sub sub domains and for them the current certs are not valid. We need to find a way to generate them dynamically or find out all used subdomains

d4rkd3v1l commented 3 years ago

Unfortunately I didn't get this working after following your instructions. But I have to admit, I'm not into docker at all. Previously I did setup bumper manually.

Now I always get this error, as far as I can tell, it may be related to the architecture? Is it correct, that the image was not build for ARM? I'm trying to use it on a raspi. So maybe that's the problem?!

bumper_1 | standard_init_linux.go:219: exec user process caused: exec format error bumper_bumper_1 exited with code 1

I also had to change the nginx path to get rid of the following error.

nginx_1 | 2037/04/23 10:58:16 [emerg] 1#1: open() "/etc/nginx/nginx.conf" failed (2: No such file or directory) nginx_1 | nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (2: No such file or directory)

from

 - ./nginx/:/etc/nginx:ro # See config file below

to

 - /etc/nginx/:/etc/nginx:ro # See config file below

Don't quite understand this either, as I would have expect it to use exactly the same path for inside the container, no matter how my system is set up in the end... Isn't this the idea of docker?!

However, my "current issue" is the first error.

edenhaus commented 3 years ago

Unfortunately I didn't get this working after following your instructions. But I have to admit, I'm not into docker at all. Previously I did setup bumper manually.

Now I always get this error, as far as I can tell, it may be related to the architecture? Is it correct, that the image was not build for ARM? I'm trying to use it on a raspi. So maybe that's the problem?!

Please get familiar with docker and docker compose. You need to mount the folder containing the certs to the container. I also have provided only a image for x64, but you can create a image for the raspi yourself with the source code or run it manually without docker

When you certs are found, bumper will call a go program to generate it. As I‘m not familiar with go, I can‘t change the go code. Also a bug there exists #117. I only tested the manual way, which is described in the docs.

Don't quite understand this either, as I would have expect it to use exactly the same path for inside the container, no matter how my system is set up in the end... Isn't this the idea of docker?!

For your second issue I will update the docs above, but you should get familiar with docker first. In the folder, where you have created the docker-compose.yaml, create a new folder named „nginx“ and within create a file nginx.conf and place there the whole Nginx configuration from above.

You don‘t docker to run bumper. You can still download the code of this pr and run it manually. Be aware that you need probably configure nginx manually as at least my deebot 950 is connecting on port 443 instead of 8883

Bustel commented 3 years ago

Is the edenhaus/bumper image rpi compatible? The container instantly dies with standard_init_linux.go:219: exec user process caused: exec format error.

I guess the go code was not compiled for ARM.

edenhaus commented 3 years ago

Is the edenhaus/bumper image rpi compatible? The container instantly dies with standard_init_linux.go:219: exec user process caused: exec format error.

I guess the go code was not compiled for ARM.

I have tested it only on my linux x64 server.

I have no idea about the go program... But if you create the certs manually and map it as volume inside the container the go program will not be executed. I would recommend it to create it manually as a bug in the go program exists #117

Bustel commented 3 years ago

Does not seem to be an issue with the go code in bumper but rather docker itself. I tried to build the image on my rpi4 and the build failed with the same error:

pi@rpi4:~/smarthome/bumper-dev $ docker build -t foobar/bumper .
Sending build context to Docker daemon  41.99MB
Step 1/16 : ARG FROM_ARCH=amd64
Step 2/16 : FROM alpine as builder
 ---> b6342193e7c0
Step 3/16 : ADD https://github.com/balena-io/qemu/releases/download/v3.0.0%2Bresin/qemu-3.0.0+resin-arm.tar.gz .
Downloading [==================================================>]  1.719MB/1.719MB

 ---> Using cache
 ---> 4b90884cb9bd
Step 4/16 : RUN tar zxvf qemu-3.0.0+resin-arm.tar.gz --strip-components 1
 ---> Using cache
 ---> 8552add857b4
Step 5/16 : ADD https://github.com/balena-io/qemu/releases/download/v3.0.0%2Bresin/qemu-3.0.0+resin-aarch64.tar.gz .
Downloading [==================================================>]  1.857MB/1.857MB

 ---> Using cache
 ---> 56fb8e80c4f2
Step 6/16 : RUN tar zxvf qemu-3.0.0+resin-aarch64.tar.gz --strip-components 1
 ---> Using cache
 ---> e0a169240d89
Step 7/16 : FROM $FROM_ARCH/python:3.7-alpine as base
 ---> ec8ed031b5be
Step 8/16 : COPY --from=builder qemu-arm-static /usr/bin
 ---> Using cache
 ---> 1ce35e30773f
Step 9/16 : COPY --from=builder qemu-aarch64-static /usr/bin
 ---> Using cache
 ---> b9d7e4a15f11
Step 10/16 : FROM base as builderfinal
 ---> b9d7e4a15f11
Step 11/16 : RUN apk add build-base
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
 ---> Running in a64e938c918e
standard_init_linux.go:219: exec user process caused: exec format error
The command '/bin/sh -c apk add build-base' returned a non-zero code: 1

I tried different FROM_ARCH values and --platform options but I cannot build the image. Ill try again tomorrow on my laptop.

edenhaus commented 3 years ago

Does not seem to be an issue with the go code in bumper but rather docker itself. I tried to build the image on my rpi4 and the build failed with the same error:

pi@rpi4:~/smarthome/bumper-dev $ docker build -t foobar/bumper .
Sending build context to Docker daemon  41.99MB
Step 1/16 : ARG FROM_ARCH=amd64
Step 2/16 : FROM alpine as builder
 ---> b6342193e7c0
Step 3/16 : ADD https://github.com/balena-io/qemu/releases/download/v3.0.0%2Bresin/qemu-3.0.0+resin-arm.tar.gz .
Downloading [==================================================>]  1.719MB/1.719MB

 ---> Using cache
 ---> 4b90884cb9bd
Step 4/16 : RUN tar zxvf qemu-3.0.0+resin-arm.tar.gz --strip-components 1
 ---> Using cache
 ---> 8552add857b4
Step 5/16 : ADD https://github.com/balena-io/qemu/releases/download/v3.0.0%2Bresin/qemu-3.0.0+resin-aarch64.tar.gz .
Downloading [==================================================>]  1.857MB/1.857MB

 ---> Using cache
 ---> 56fb8e80c4f2
Step 6/16 : RUN tar zxvf qemu-3.0.0+resin-aarch64.tar.gz --strip-components 1
 ---> Using cache
 ---> e0a169240d89
Step 7/16 : FROM $FROM_ARCH/python:3.7-alpine as base
 ---> ec8ed031b5be
Step 8/16 : COPY --from=builder qemu-arm-static /usr/bin
 ---> Using cache
 ---> 1ce35e30773f
Step 9/16 : COPY --from=builder qemu-aarch64-static /usr/bin
 ---> Using cache
 ---> b9d7e4a15f11
Step 10/16 : FROM base as builderfinal
 ---> b9d7e4a15f11
Step 11/16 : RUN apk add build-base
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
 ---> Running in a64e938c918e
standard_init_linux.go:219: exec user process caused: exec format error
The command '/bin/sh -c apk add build-base' returned a non-zero code: 1

I tried different FROM_ARCH values and --platform options but I cannot build the image. Ill try again tomorrow on my laptop.

Please give a look at https://github.com/bmartin5692/bumper/blob/9b1ae262a807ad9fe09b33841ba7054b23c91453/hooks/build

I used the hook scripts to push the image for the same arch as the official one. So probably you should be able to use it on the rpi

Bustel commented 3 years ago

Thank you for the hint! I managed to get it running for the rpi4 by running this on my laptop: docker build --build-arg FROM_ARCH=arm32v7 . --tag bumper-arm32

and the using docker save and docker image load to get the image onto the rpi. However, after recreating the bumper.crt the app now shows an empty screen.

grafik

Do I need to update the CA cert as well? I had already reduced the validity before.

edenhaus commented 3 years ago

Do I need to update the CA cert as well? I had already reduced the validity before.

I don't think so but I have recreate it also as I was not sure what the go program does.

Also I have tested only the iOS App version. Maybe the android one uses different endpoints. As I don't have a rooted phone I couldn't test it

Bustel commented 3 years ago

I checked the logs via adb and the app throws RobotUI Errors for the following endpoints:

If I try them out manually they all return 404. Are those different in the iOS app?

edenhaus commented 3 years ago

I checked the logs via adb and the app throws RobotUI Errors for the following endpoints:

* https://api-app.dc-eu.ww.ecouser.net/api/rapp/sds/user/data/map/get?key=EcovacsHome&subKeys=name

* https://api-app.dc-eu.ww.ecouser.net/api/pim/product/software/config/batch

* https://api-app.dc-eu.ww.ecouser.net/api/ecms/app/ad/res?locations=robot_device_list_first

If I try them out manually they all return 404. Are those different in the iOS app?

The endpoints exists and return also some data when called correctly. Be aware that you need to call them correctly. For example: https://api-app.dc-eu.ww.ecouser.net/api/pim/product/software/config/batch must be called as post request with some pid in the body.

A GET request on https://api-app.dc-eu.ww.ecouser.net/api/rapp/sds/user/data/map/get?key=EcovacsHome&subKeys=name returns

{"code": 0, "data": {"data": {"name": "My Home"}, "tag": null}, "message": "success"}

Probably you don't have setup bumper correctly or you don't use the version with the code of this PR....

Bustel commented 3 years ago

Probably you don't have setup bumper correctly or you don't use the version with the code of this PR....

Just had the same realization and noticed that I build the main branch instead of the PR branch of your repo. Sorry for wasting your time on this :see_no_evil:

Also the manual build was not necessary. I just had to run docker pull with the correct platform on the rpi: docker pull edenhaus/bumper --platform linux/arm/v7. I naively assumed that docker is able to figure out which platform it is running on...

~Now the Deebot is shown in the app but marked as offline. Bumper status shows that the client did not connect to MQTT, but I dont see any errors~ After restarting the app a couple times I just had my first successful cleaning run :tada:

d4rkd3v1l commented 3 years ago

After pulling the docker image like @Bustel suggested, and after fixing https://github.com/linuxserver/docker-papermerge/issues/4#issuecomment-735236388, I finally got the docker containers running on my rpi using docker-compose up.

ports unreachable

Unfortunately the services are not reachable on their ports (either not running, or not exposed, or otherwise "blocked"?).

While docker-compose ps shows the containers running and the ports:

docker-compose ps
     Name                    Command               State                       Ports
--------------------------------------------------------------------------------------------------------
bumper_bumper_1   python3 -m bumper                Up
bumper_nginx_1    /docker-entrypoint.sh ngin ...   Up      443/tcp, 5223/tcp, 80/tcp, 8007/tcp, 8883/tcp

They won't appear in netstat, e.g.:

sudo netstat -ntl | grep "8007"
sudo netstat -ntl | grep "443"

And of course everything else cannot reach them as well (e.g. curl):

curl -X GET https://api-app.dc-eu.ww.ecouser.net/api/rapp/sds/user/data/map/get?key=EcovacsHome&subKeys=name
curl: (7) Failed to connect to api-app.dc-eu.ww.ecouser.net port 443: Connection refused

The domain is resolved correctly by pi-hole:

host api-app.dc-eu.ww.ecouser.net
api-app.dc-eu.ww.ecouser.net has address 192.168.178.3

Stopping docker and starting up the old "bumper.service" works just fine.

sudo netstat -ntl | grep 8007
tcp        0      0 192.168.178.3:8007      0.0.0.0:*               LISTEN

Note: I don't want to keep using the "old service" approach, as I would have to setup nginx etc. manually and actually therefore prefer the docker-way.

iptables

Since docker seems to add a lot of iptables rules to the system and therefore broke my piVCCU installation, I had to modify the rules like here: https://github.com/alexreinert/piVCCU/issues/264#issuecomment-753907782 But I already reverted all those changes and still none of the ports is reachable for me :/


Did anyone else face a similar issue? Or does anyone have an idea what might go wrong here? Any help is highly appreciated! Thx in advance.

edenhaus commented 3 years ago

@d4rkd3v1l I have updated the docker-compose example. The problem was that I only exposed the ports in my example as I use in my setup macvlan.

A working example can be found here

d4rkd3v1l commented 3 years ago

Thx @edenhaus I got it working now as well, after using ports instead of expose. Also be aware that the example docker-compose.yaml uses the original docker image from bmartin5692, and not the one you provided on docker hub, so this must also be changed to edenhaus/bumper in order to make it work (for the moment, until everything got merged, I guess).

edenhaus commented 3 years ago

@d4rkd3v1l I know that, put I don't want to change it as my images are only test images, which can be updated and also be broken anytime

edenhaus commented 3 years ago

@bmartin5692 Have you time for a review? I think some people are waiting for this PR

bmartin5692 commented 3 years ago

I'll try to get this tested and reviewed this weekend.

slev1n commented 3 years ago

@edenhaus system: rpi 4 @arm32 with deebot 901 and adguard for dns rewrites

I had to add 'privileged: true' to start bumper from composer. The iobroker plugin worked and i see the login in the logs. But the IOS app didnt not work, timeout -> no login attempt in the logs.

Do u have any suggestions?

edenhaus commented 3 years ago

I use the docker-compose setup described in https://github.com/bmartin5692/bumper/tree/master/example/docker-compose

I have no idea what is wrong with your setup, but it can be a lot.... Wrong certs, wrong dns config, endpoints not reachable,.... Also I don't have your model and as I can remember it correctly your robot uses the protocol MQTT/xml and my 950 uses MQTT/json. So it can also be that there are missing endpoints... Also you need to trust the certificate on your apple device. As you can see there can be a lot and sorry but I currently don't have time to give you support.