search

Hypfer / Valetudo

Cloud replacement for vacuum robots enabling local-only operation
https://valetudo.cloud
Apache License 2.0
6.72k stars 397 forks source link

Support for the new persistent map format #44

Closed Hypfer closed 5 years ago

leonardpitzu commented 5 years ago

@JohnRev another idea - if i delete all folders would it bring something? what else do i need to refresh to make everything "like new"? factory reset brings nothing - i've done it and everything is still there. Edit: watchdog log - pSharedRegionCreate,85: Failed to create shared memory region '/PersistInfo'

JohnRev commented 5 years ago

I have once deleted all the folders inside the rrlog folder to refresh things after a factory reset. You could try that, but I'm not sure it's related.

leonardpitzu commented 5 years ago

@JohnRev at what level have you set the logging? Is upload still enabled? are the other scripts still running or do you have them exit as soon as they are called? i made a full recovery - press power+plug+wifi reset. i just want to have "the same" setup as you do so we can exclude errors

JohnRev commented 5 years ago

I don't recall messing with the logging levels and the scripts (they don't exit). I do have tje xiaomi servers in the hosts file and the iptables rules in rc.local (from dummycloud setup). Upload is enabled, but the vacuum can't connect to the internet to do the upload.

leonardpitzu commented 5 years ago

@JohnRev a full wipe and many hours later i got the damn thing to have consistent logs - the rr.gz are extractable and have no errors. still they are empty and the map is as well empty - i only see the path of the vacuum as in the screenshot above. again i am out of ideas...

Hypfer commented 5 years ago

I think that there might be no "new map format" 🤔 Maybe the virtual walls are stored the same way as zones are stored.

JohnRev commented 5 years ago

Any idea how zones are stored?

Hypfer commented 5 years ago

https://github.com/marcelrv/XiaomiRobotVacuumProtocol/issues/15#issuecomment-447636171

I guess you can fetch them with a miio command? Or is that write only?

cryptomilk commented 5 years ago

@JohnRev thanks for your work. I've added support for patching rrlogd to the imagebuilder script of the dustcloud:

https://github.com/dgiese/dustcloud/pull/162

leonardpitzu commented 5 years ago

@JohnRev my partition was bad. i had to kill almost all services to be able to unmount the partition. i did so and in the end i run fsck. now i have (again) a normal behaviour where maps are stored in between reboots. the patched rrlog + fw 1780 work great.

cryptomilk commented 5 years ago

FYI: The latest 0.9 valetudo release works with v001780 and a patched rrlogd quite fine.

I wonder how the persistent map feature really works. Normally I guess you want to store a complete map of your rooms as persistent and be able to load it. The robot might do a run and might not able to enter a room because the door is closed, however you still want a map with all rooms mapped so you can send the robot for a zoned clean up.

You can activate the persistent map feature doing the following:

echo -n 1 > /mnt/data/rockrobo/lab.cfg

Time for more investigation.

cryptomilk commented 5 years ago

I've patched v001792.

radiff2 rrlogd.v001792 rrlogd.v001792.patched 
0x000105a8 4b => 47 0x000105a8
0x000107d2 fff7 => e320 0x000107d2
radiff2 -a arm -D rrlogd.v001792 rrlogd.v001792.patched 
(locked: no new keys can be created (anal.split))
--- 0x000105a8  4b
- add r0, sp, 0x12c
- movs r2, 0x10
+++ 0x000105a8  47
+ add r0, sp, 0x11c
+ movs r2, 0x10

--- 0x000107d2  fff7
- bl 4294966930
+++ 0x000107d2  e320
+ movs r0, 0xe3
+ invalid

Here is support for patching rrlogd directly when building the image (uses bsdiff/bspatch)

https://github.com/cryptomilk/dustcloud/tree/master-imgbuild-rrlogd

JohnRev commented 5 years ago

Maybe the virtual walls are stored the same way as zones are stored.

After spending a ridiculous amount of time reverse engineering the saved map formats, I think I have finally figured it out!! The no-go zones and virtual wall coordinates are stored in : /mnt/data/rockrobo/PersistData_1.data.

The file first needs to be zlib decompressed, and inside it it contains a block of fixed size where virtual wall coords are stored, and also no-go zones. The fixed size of these blocks is why the xiaomi app has an upper limit of virtual walls that you can create (50 or so, I think).

I need to spend a bit more time writing some codes to parse these files. This will hopefully allow us (with some effort) to have the virtual walls and no-go zones managed and displayed from within Valetudo!

JohnRev commented 5 years ago

Here's a quick and dirty python code to parse PersistData_1.data. Can anyone who is using virtual walls test it?

import zlib, sys, io
import struct

def read_int(data):
    return struct.unpack('<i', data.read(4))[0]

def read_short(data):
    return struct.unpack('<h', data.read(2))[0]

def convert_coord(coord):
    return 25600 + coord

infile = 'PersistData_1.data'

# Read PersistData starting from 0x17
with open(infile, "rb") as infile:
    infile.seek(0x17, 0)
    data = infile.read()

# decompress the zlib
decompress = zlib.decompress(data)

mapdata = io.BytesIO(decompress)

# Jump to no-go zones block offset
mapdata.seek(0x000318E8)
zonecount = read_int(mapdata)
print("There are %d no-go zones" % zonecount)
zones = []
for i in range(zonecount):
  zones.append([])
  for j in range(8):
    zones[i].append(convert_coord(read_short(mapdata)))

print(zones)

# Jump to virtual walls block offset
mapdata.seek(0x00031C0C)
wallcount = read_int(mapdata)
walls = []
print("There are %d virtual walls" % wallcount)
for i in range(wallcount):
  walls.append([])
  for j in range(4):
    walls[i].append(convert_coord(read_short(mapdata)))
print(walls)

Run this on your PC after copying /mnt/data/rockrobo/PersistData_1.data to the same folder where the python script is saved.

Note: The offsets above should have only been tested on v1780 gen2. Other firmwares may require different offsets.

cryptomilk commented 5 years ago

Do you know what else is stored in this file?

JohnRev commented 5 years ago

There's at least 1 more thing stored in there, but I am not sure what it is

MeisterTR commented 5 years ago

tested the py script and it shows me the nogo zones

Hypfer commented 5 years ago

Timers maybe?

cryptomilk commented 5 years ago

@JohnRev did you look into libNavDriver.so for that?

mrwsl commented 5 years ago

I just updated to 1780 and still get 500 errors with valetudo 0.9. Is it just me?

cryptomilk commented 5 years ago

Did you patch rrlogd?

mrwsl commented 5 years ago

I did when building the firmware but will try again today. Maybe I did something wrong there.

cryptomilk commented 5 years ago

./imagebuilder.sh --patch-rrlogd then check the log that it has been patched. My latest changes to the image builder script allow to already add valetudo to the flashable firmware image.

mrwsl commented 5 years ago

Here is what I did:

sudo ./dustcloud/devices/xiaomi.vacuum/firmwarebuilder/imagebuilder.sh -k id_ed25519.pub -f v11_001780.pkg --disable-xiaomi --patch-rrlogd
Scriptpath: ./dustcloud/devices/xiaomi.vacuum/firmwarebuilder
.../dc/imagebuilder.sh
compatible readlink found!
Generate SSH Host Keys if necessary
decrypt soundfile
ccrypt: .../dc/english.pkg: Kennwort passt nicht -- unverändert
unpack soundfile
decrypt firmware
ccrypt: .../dc/v11_001780.pkg: Kennwort passt nicht -- unverändert
unpack firmware
patch ssh host keys
disable SSH firewall rule
integrate SSH authorized_keys
reconfiguring network traffic to xiaomi
pack new firmware
encrypt firmware
2ef9c67f1a7d9b6a9ea292837e4bbf2a  output/v11_001780.pkg

I don't read anything from the rrlogd patch.

cryptomilk commented 5 years ago

I don't see that it patches rrlogd so is the git branch up to date and do you have bspatch installed?

mrwsl commented 5 years ago

I was missing the bsdiff package. I got this now:

./dustcloud/devices/xiaomi.vacuum/firmwarebuilder/imagebuilder.sh -k id_ed25519.pub -f v11_001780.pkg --disable-xiaomi --patch-rrlogd
Scriptpath: ./dustcloud/devices/xiaomi.vacuum/firmwarebuilder
.../dc/imagebuilder.sh
compatible readlink found!
Generate SSH Host Keys if necessary
decrypt soundfile
ccrypt: .../dc/english.pkg: Kennwort passt nicht -- unverändert
unpack soundfile
decrypt firmware
ccrypt: .../dc/v11_001780.pkg: Kennwort passt nicht -- unverändert
unpack firmware
patch ssh host keys
disable SSH firewall rule
integrate SSH authorized_keys
reconfiguring network traffic to xiaomi
checking if we can patch rrlogd
pack new firmware
encrypt firmware
bce3c1cdf5fdd16f14b0209636a8538e  output/v11_001780.pkg

After pushing the firmware to to vacuum I still get the error 500. How do you add valetudo while building?

Hypfer commented 5 years ago

Did you start a new cleanup? Since you were running an unpatched rrlogd it might be that the data it's trying to display is still encrypted.

The removed encryption only applies to new cleanups.

JohnRev commented 5 years ago

In addition to what @Hypfer pointed out, it seems that your rrlogd patch didn't work? In the log you pasted, I can only see checking if we can patch rrlogd, but no follow-up .. it should be followed by creating backup of rrlogd and patching rrlogd. On your vacuum, can you do md5sum /opt/rockrobo/rrlog/rrlogd, if it is not 93e8df884b4c6162ffa2a11161f47429, then it is not the patched rrlogd for gen2 v1780. You can replace it manually with this one https://github.com/Hypfer/Valetudo/files/2670885/rrlogd_patched_001780_gen2.zip and kill the running rrlogd process (or reboot the vacuum). See earlier posts for more details.

mrwsl commented 5 years ago

Pushing the patched rrlogd manually did the trick - thanks @JohnRev and @Hypfer. Still dont know why the image wasnt build correctly.

JohnRev commented 5 years ago

Other things stored in PersistData_1.data include: "Goto Routing", "Clean Routing" ( paths/routes?), "Dock position", "Robot position". Haven't had time to update my code for parsing them ... might do that if I get some free time :)

Fietspomp86 commented 5 years ago

I don't know if this belongs here, but I also get a 500 error when charging. When roborock (gen1, version 3514 rooted) is running, map is generated just fine. When charging the roborock/api/map/latest shows: Error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt

So does mine also need a patched rrlogd?

ralf-ms commented 5 years ago

I'm having the same problem as Fietspomp86, during cleaning the generated map can be viewed without any difficulties, as soon as the robo has finished cleaning, the map vanishes and the above html error 500 or with "../latest" error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt) is displayed.

cryptomilk commented 5 years ago

It is possible that on the gen1 firmware version, rrlogd has a new encryption method. However someone needs to reverse engineer it to see what actually is going on.

JohnRev commented 5 years ago

@Fietspomp86 rrlogd for gen1 will need to be patched too, but none have been published yet. @ralf-ms are you also using gen1 with 3514 firmware?

ralf-ms commented 5 years ago

yes, correct: gen1 / 3514

Fietspomp86 commented 5 years ago

Ok allright, what do I need to do then? I'm fairly ok with linux so I can try to patch it with some instructions.

JohnRev commented 5 years ago

Try the attached rrlogd, if you want. I haven't tested it myself as I don't have a gen1 vacuum.

gen1_v11_003514_rrlogd_patch.zip

Fietspomp86 commented 5 years ago

Ok, I've copied the rrlogd and rebooted. Apparently that removed Valetudo which is strange. Installed Valetudo again and started vacuum to test. Now it's turning off and on every 10 seconds. And right now it's speaking Chinese to me.... So for me it's not working yet....

EDIT: Strange enough it has resetted itself to old firmware I guess, speaking Chinese and disconnected from WiFi.

JohnRev commented 5 years ago

I am guessing the "patched" rrlogd failed and ended up crashing the robot, which then went into recovery mode... maybe? Sorry about that. As I said, I don't have the gen1 to test, and I just tried applying the same patches that I made for gen2 onto the gen1 rrlogd (while adjusting the offsets accordingly). I may look at this more if I get some time

JohnRev commented 5 years ago

After transfering the rrlogd to the robot, did you make sure you gave it execution permissions? chmod +x /opt/rockrobo/rrlog/rrlogd

ralf-ms commented 5 years ago

The patched rrlogd is working. I've just tried a cleaning session and could retrieve the map after the finished session. Thank you very much, JohnRev!

JohnRev commented 5 years ago

No problem! I will try to post an "auto patcher" script that would patch rrlogd on any firmware/generation.

Hypfer commented 5 years ago

Can we please not use this issue or any issue at all for basic tech support.

This is not a forum but rather a place to gather information and use said information to implement the feature mentioned in the title.

Fietspomp86 commented 5 years ago

After transfering the rrlogd to the robot, did you make sure you gave it execution permissions? chmod +x /opt/rockrobo/rrlog/rrlogd

Quite possible I forgot that.... I'll retry it when it's back online.

Hypfer commented 5 years ago

Oh ffs.

JohnRev commented 5 years ago

@Hypfer: This thread should probably be closed. As you pointed out, the conversation has gotten out of hand. The new map format parsing is now well understood. The rrlogd patches (in gen1 or gen2) have already been documented. No changes are required in Valetudo itself. Maybe just update the README.md ?

There is another issue created for the newer features that come with the persistent maps #72