search

Hypfer / Valetudo

Cloud replacement for vacuum robots enabling local-only operation
https://valetudo.cloud
Apache License 2.0
6.72k stars 397 forks source link

Support for the new persistent map format #44

Closed Hypfer closed 5 years ago

leonardpitzu commented 5 years ago

nvmap attached in new rr.gz format

navmapfirst.ppm.log.rr.gz

JohnRev commented 5 years ago

Some findings:

The static encryption key (RoCKR0B0@BEIJING) is no longer used. Instead, a random key is generated for each encrypted file. The key is sourced from /dev/urandom. Then, this key is itself encrypted using mbedtls_pk_encrypt. The first 512 bytes of the file represent the encrypted key. The remaining file content past the first 512 bytes is the map encrypted with mbedtls_aes_crypt_ecb and the random key.

That's what I have found so far, but it would be cool if someone else can independently confirm this.

JohnRev commented 5 years ago

Here's another update.

The encrypted rr.gz files go through this process:

  1. A random key is obtained by reading /dev/urandom/
  2. The key is encrypted with an RSA public key, using mbedtls_pk_encrypt
  3. The encrypted key is stored in the file header (first 512 bytes)
  4. The gzip-ed file is encrypted using AES ECB mbedtls_aes_crypt_ecb and appended to the .rr.gz file

This is presumably so that Xiaomi can later decrypt the key using their RSA private key, which will then enable them to decrypt the gzip archive.

Some possible workarounds:

  1. Intercept the calls to /dev/urandom and redirect to a custom file. This way, we will always know what the "random" key is.
  2. Replace the public key with one which we know the private key for
  3. Disable encryption all together.

Options 2, and 3 would break the compatibility with the Xiaomi cloud servers, and so they are best used with dummycloud enabled. Option 1 could theoretically still work, but can be easily detected by Xiaomi.

I tried experimenting with Option 1 for a bit, but was unsuccessful. So I decided to just disable encryption, since I do not care about the Xiaomi cloud because my vacuum is not allowed to connect to the internet anyway. Turns out this is the simplest option.

I updated my gen2 vacuum to 001780 and patched rrlogd. The patched file is attached here. Then, I updated the regex that I pointed out in an earlier comment: https://github.com/Hypfer/Valetudo/issues/40#issuecomment-443730275 Valetudo now correctly shows the maps on v001780 when the vacuum is docked.

The patched rrlogd for v001780 is attached here. It should be copied to the vacuum and moved to /opt/rockrobo/rrlog/. Don't forget to give it execution permission, chmod +x /opt/rockrobo/rrlog/rrlogd Note: it is probably wise to keep a copy of the original rrlogd file first: mv /opt/rockrobo/rrlog/rrlogd /opt/rockrobo/rrlog/rrlogd_. Also note: that I did not test this on any other version, and it will probably not work and require different patch offsets. Final note: You will need to either reboot your vacuum, or kill the running rrlogd process first so that the robot will re-spawn a process with the patched rrlogd.

Here are the hex patches used:

0001051C: 4B 47
00010746: FF E3
00010747: F7 20

rrlogd_patched_001780_gen2.zip

leonardpitzu commented 5 years ago

@JohnRev can you patch a rrlog (of course i will privide it) for gen2? i would try this and see how the new persistant pams work out...by the way - what's the setting needed to enable the map storage feature?

JohnRev commented 5 years ago

This is gen2. I should have clarified. Not sure about the map storage. I was mainly focused on finding means for decrypting the rr.gz file :)

leonardpitzu commented 5 years ago

gr8. i'll download 1780 and get myself to work. i guess we should e carefull and stop the automatic update mechanism - xiaomi will most probably close some doors and we might remain stuck with some default stuff...

JohnRev commented 5 years ago

My vacuum has never connected to the internet, so I'm not too worried about the automatic updates. If you do use the patched rrlogd file, it would probably be wise to use dummycloud too as I'm pretty sure the patched rrlogd breaks compatibility with the xiaomi cloud services.

leonardpitzu commented 5 years ago

i was connected to the internet (i got the gadget exactly when it came on the market). now i am on dummycloud and ccrypt is removed (just an empty file). my firewall is also plugging all holes so i should be safe. have you tried to enable the map storage feature? and if yes how did you do it?

JohnRev commented 5 years ago

No, sorry, I haven't looked into the map storage feature. Tho if you do figure out how to enable it, let me know!

leonardpitzu commented 5 years ago

@JohnRev i updated the beast and used your patched rrlogd but i keep getting error 500 from the webserver...

JohnRev commented 5 years ago

@leonardpitzu, you will have to compile Valetudo with the updated regex from https://github.com/Hypfer/Valetudo/issues/40#issuecomment-443730275

leonardpitzu commented 5 years ago

@JohnRev Already done that - I get the correct path to the files and I am trying correctly to open them. That’s where I got with the debugging. I user the rrlogd you provided. Added some more logging to figure out what’s wrong...

JohnRev commented 5 years ago

Interesting. It works fine for me. How does your /mnt/data/rockrobo/rrlog/ folder look like? If there are a bunch of subfolders there, i suggest you delete them and try again with a fresh sweep of the vacuum. Then, after the vacuum is docked, there should be a new subfolder containing a bunch of .gz file. Can you confirm they are unencrypted gzip files?

JohnRev commented 5 years ago

Oh, and did you reboot your vacuum to make sure it uses the patched rrlogd? If you dont want to reboot, try killing the running rrlogd process (find process id with ps aux | grep rrlogd then kill -9 [pid]). The vacuum will respawn one with the patched rrlogd (assuming it is placed in the right path)

dugite-code commented 5 years ago

@JohnRev How did you go about patching rrlogd?

leonardpitzu commented 5 years ago

@JohnRev i've done some more debugging (i know linux quite well but i'm 250km away from the vacuum so debugging is a bit more tricky :) ). anyway here's what i found (again, using fw 1780 on gen 2, your patched rrlogd and a patched/modified version of valetudo):

and here it stops with the error:

digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

Previousely the log was being encrypted with the standard symmetric key which we all know. So the webserver was decryptiong it using the key, unpacking it and do the rest of it's thing. From your description i figured you disabled the encryption all together so the webserver now failes to decrypt the data. Even more, when i download the encrypted logs locally and try to unpack them it fails - probably the magic number is wrong.

JohnRev commented 5 years ago

@leonardpitzu :Just to confirm, did you make sure the running rrlogd process is the patched one? Simply replacing the file won't do it. It sounds to me the gz files are still being generated by the unpatched rrlogd

leonardpitzu commented 5 years ago

@JohnRev copied the patched rrlogd and rebooted the thing... i'm doing some more testing to see what's going on...

JohnRev commented 5 years ago

Oh! I think I attached the wrong rrlogd. In the one you're using right now, the first 512 bytes are the encrypted key and the rest of the file is the (unencrypted) gzip. So if you do something like dd if="navmap702931.ppm.0001.rr.gz" of="navmap-unencrypted.gz" bs=1 skip=512 you should get the clean unencrypted gz. I have updated my initial comment. If you don't mind, please use this rrlogd. It is the same one as I have on my vacuum right now. md5sum is 93e8df884b4c6162ffa2a11161f47429 Send it to the vacuum, kill the running rrlogd process (or reboot), and try again. rrlogd_patched_001780_gen2.zip

JohnRev commented 5 years ago

@dugite-code: I disassembled rrlogd, looked for the function call that appends the encrypted key to the header, and disabled (NOP) that call. Then I looked for the call that stores the encrypted gz bytes to the file, and made them store the unencrypted buffer instead. The hex patches posted in my first comment do just that, and you can apply them to your unpatched rrlogd (of version 1780) to get the same patched rrlogd that I posted here.

leonardpitzu commented 5 years ago

@JohnRev i looked at the attached rrlogd and it's not the patched one. I patched it and am testing it now. will report back

JohnRev commented 5 years ago

Thanks! I am waiting. Sorry for attaching the wrong rrlogd initially ... I got my files mixed up :smile:

leonardpitzu commented 5 years ago

@JohnRev sudo "you deserve a beer!" it's working!

JohnRev commented 5 years ago

Awesome!! And thanks for being willing to testing it :smile:

I discovered there's some 500 errors still showing up after some time, but these are strictly related to the code logic of how the /mnt/data/rockrobo/rrlog is scanned in FIND_LATEST_MAP_IN_ARCHIVE function. I will write about them in #40 .

Glad it's working, and hope this helps others too!

JohnRev commented 5 years ago

@leonardpitzu, I have a question for you. After rebooting the vacuum, does the robot forget the last map when performing a new sweep and generates a new one? Or does it trace correctly over the previously generated map?

leonardpitzu commented 5 years ago

@JohnRev mine also has Alzheimer and generates a new map. This is a "feature" i would guess unless we figure out what the "save map" button actually does. If we can set the same flag, by hand, we can keep the map between reboots. I read somewhere that the setting is some new file in /mnt/data/rockrobo. The file has to contain the value 1 in order to enable the map storage feature. Yet i can't seem to find that post nowhere so i am stuck right now...

JohnRev commented 5 years ago

I think I was able to get the saved map functionality to work. Would you be willing to test it?

  1. Do a full sweep to generate the map to be saved
  2. Install python-miio pip install python-miio, then execute the following (replacing with the correct variables, obviously).
  3. Execute: 
    
export MIROBO_IP=vacuum_ip
export MIROBO_TOKEN=vacuum_token_key

mirobo raw-command set_lab_status 1 mirobo raw-command save_map


4. Reboot the vacuum
5. Do a new sweep
JohnRev commented 5 years ago

Indeed, the above commands seem to produce a file /mnt/data/rockrobo/lab.cfg with content 1. Not sure if just creating that file (echo 1 > /mnt/data/rockrobo/lab.cfg) would suffice, but the above commands worked well for me, as far as I can tell.

leonardpitzu commented 5 years ago

@JohnRev that was the config file + flag. I am not sure though how this should work... Between reboots the maps is lost but i still get to see the track/path of the vacuum (well, at least the last part of it as it's split as o lately on more pieces). I have a persistent gridmap in /dev/shm (around 1MB) but even witht the "preferGridMap":true i only get to see the path...

JohnRev commented 5 years ago

Note sure I understand. Do you mean in Valetudo you see the path but without a map?

leonardpitzu commented 5 years ago

yes

JohnRev commented 5 years ago

Now that is something I haven't seen yet. It would be interesting to see which path file is being parsed by Valetudo. Using which method did you enable the persist flag? Manually in lab.cfg, or with the mirobo set_lab_status and save_map commands?

leonardpitzu commented 5 years ago

i used lab.cfg manual edit. valetudo is selecting the latest folder and searching for navmap and slam.

JohnRev commented 5 years ago

Perhaps the mirobo raw-command save_map command was needed? I just tried a reboot and my old map was still there, with the last path drawn. I did a new sweep, and the robot "remembered" the last map and just started tracing over it

leonardpitzu commented 5 years ago

@JohnRev somehow my setup was messed up... now it saves the map correctly and it survives reboots. i'll make a diff to see what was screwed up... In between i made a script to have my "own" firmware release - custom apps, scripts, telegram notifications for start-up and status updates (whenever something changes), removed xiaomi stuff and my custom valetudo. This should keep me safe from failures - whenever something goes wrong the vacuum resets itself and i have to start everything from almost scratch.

JohnRev commented 5 years ago

Nice setup!

So now we have a way to use Valetudo on gen2 v001780 (with patched rrlogd) and a way to keep maps after reboot (with lab.cfg)!!

Hypfer commented 5 years ago

So they are still using ppm maps?

How are virtual walls stored?

JohnRev commented 5 years ago

Yes, ppm is still used.

I haven't been able to figure out where virtual walls are stored (yet), and I don't have the mi app to test it either (my vacuum isn't connected to the internet).

It would be helpful if someone who does use the mi app can provide some insight

dugite-code commented 5 years ago

Hey @JohnRev could you post this to the https://github.com/marcelrv/XiaomiRobotVacuumProtocol github? I know the miio guys check there for newly discovered commands.

Hopefully someone will figure out the virtual walls soon

On December 14, 2018 11:16:48 AM UTC, JohnRev notifications@github.com wrote:

I think I was able to get the saved map functionality to work. Would you be willing to test it?

  1. Do a full sweet to generate the map to be saved
  2. Install python-miio pip install python-miio, then execute the following (replacing with the correct variables, obviously).
  3. Execute: {code} export MIROBO_IP=vacuum_ip export MIROBO_TOKEN=vacuum_token_key

mirobo raw-command "set_lab_status" 1 mirobo raw-command "save_map {code}

  1. Reboot the vacuum
  2. Do a new sweep

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/Hypfer/Valetudo/issues/44#issuecomment-447295298

JohnRev commented 5 years ago

Posted it there https://github.com/marcelrv/XiaomiRobotVacuumProtocol/issues/15#issue-391360556 with some additional info

JohnRev commented 5 years ago

How are virtual walls stored?

@Hypfer : These are possibly stored in /mnt/data/rockrobo/user_map0 and/or /mnt/data/rockrobo/PersistData*.data. Other interesting files: /dev/shm/PersistInfo , /dev/shm/SharedMapInfo

leonardpitzu commented 5 years ago

@JohnRev after the scheduled reboot (every day my vacuum reboots on its own at 3:30am) the maps were not visible any more in valetudo - again only the last part of the track was visible. i removed dummycloud and checked with the official app - the map is saved correctly and all settings are as well correct. I made a full factory reset of the device and made a sweep. Maps were saved (persistent files were generated). After a reboot, boom, maps were gone again in valetudo. I am out of ideas...

JohnRev commented 5 years ago

Hmm... I haven't faced this issue yet, and I have been rebooting my vacuum daily. Are you getting error 500 in Valetudo? Maybe the issue is from what I described here: https://github.com/Hypfer/Valetudo/issues/40#issuecomment-446504197 ?

Also, would you be willing to try all the steps, including the mirobo commands, from my earlier comment (https://github.com/Hypfer/Valetudo/issues/44#issuecomment-447295298)? I think this was the only difference between what you have done and what I have.

mrwsl commented 5 years ago

@leonardpitzu & @JohnRev Why do you reboot your device daily if I may ask?

JohnRev commented 5 years ago

For some reason that I haven't been able to figure out yet, my vacuum loses connection to my wifi overnight (but otherwise remains functional), and so I must reboot it to get it connected to my router again

leonardpitzu commented 5 years ago

@JohnRev done that. No change at all. What those commands do is identical to what the original app does. I've done the same settings with both and still the same behaviour. I also encounter #40 but i overcome it by removing the new folder. Still, the map is not visible from the "old" folder - only the track (see attached screenshots - one is valetudo, the other is the original app. both screencaps are after a manual reboot [sudo shutdown -r now]). @herrwusel i'm not rebooting it - it does so on its own at 3:30 every morning. It's some sort of scheduled tasks. I would assume all vacuums do so but you don't see it because either you're not monitoring the network or you don't have startup/shutdown/reboot notifications as i do. img_6331 img_6333

JohnRev commented 5 years ago

@leonardpitzu Would you be willing to sharing the latest .rr.gz navmaps from the newest folder?

leonardpitzu commented 5 years ago

@JohnRev here you go (there are also some new generated folder so you can have an idea what happens @reboot). Archive.zip

JohnRev commented 5 years ago 
# gzip -d /000548.20181215162410931_R0018S81502125_2018101702REL/navmapfirst.ppm.log.rr.gz
gzip: navmapfirst.ppm.log.rr.gz: decompression OK, trailing garbage ignored

After reboot, your navmapfirst.ppm is empty (or corrupt?), and that is why the map is blank and Valetudo can't display it. The official app probably pulls the map from the Mi cloud, and that's why it's displayed (just a guess)

I don't know why your navmapfirst.ppm.log.rr.gz is like that ... My guess is the "saved" persistent map is incorrect, or incomplete somehow? Maybe re-save a new one with the mi app? Btw, is there a file /mnt/data/rockrobo/user_map0 on your vacuum? What's it size?

leonardpitzu commented 5 years ago

@JohnRev user_map0 is 198k. i'll try a fsck... the map is resaved. i made a new one this morning, rebooted and valetudo is what you see it above...