some remarks from a new (and successful) imagebuilder/dummycloud/valetudo user

josch commented 5 years ago

Hi,

yesterday I successfully rooted my xiaomi roborock s50 using imagebuilder.sh with dummycloud and valetudo. Thanks for your work! It works really great! The fact that it only took me a few hours to go from zero to having my whole flat vacuumed with a live map really shows how mature and user-friendly the rooting process has become. :)

There are some issues I encountered though which are probably rooted in the highly experimental state the whole ecosystem is in right now, so one sometimes gets conflicting or outdated information. I thus wanted to start pointing out some of these issues in case it helps. I'm collecting everything I wrote down in this issue because since I'm new to this, I'm not sure which things are actually issues and which are not. I can open individual issues, if desired.

In this howto from the wiki it says to run sudo ./dustcloud/devices/xiaomi.vacuum/firmwarebuilder/imagebuilder.sh but that path doesn't exist. One would first have to git clone dustcloud into the current directory for that to work. I saw that the step was removed here and replaced by the download of firmwarebuilder_0.1.zip. But then the path doesn't fit. Was that just an oversight?

Using firmwarebuilder_0.1.zip seems to have another problem: The line endings are DOS line endings \r\n which means that the shebang line cannot work:

zsh: ./imagebuilder.sh: bad interpreter: /bin/bash^M: no such file or directory

Also, because it was packed as zip, the executable bit is missing.

The instructions still say to install git but it's not used anywhere?

The commit also added the sentence "english.pkg is obsolete now, all new firmwares contain the new english soundfiles". Unfortunately, it seems that imagebuilder.sh expects a english.pkg in the current directory. I did not find a way to run imagebuilder.sh without english.pkg and the new version of that wiki page has the download link to english.pkg removed so it took me quite some searching to figure out where to get it from.

To download the firmware, I headed to https://github.com/dgiese/dustcloud/wiki/Xiaomi-Vacuum-Firmware but when I click on the link at the top to https://dustcloud.seemoo.de/public/temp then there only seems to be a directory called diffs to which permission to access it is denied. Why is it listed?

The wiki page also says "Find the latest .pkg link from here" but maybe it's not ideal to advice finding the "latest" pkg because things might break. Maybe instead of saying "latest" some known-working version should be suggested?

As far as I understand, version 1518 is the latest version for gen2 that doesn't encrypt the map and valetudo seems to have problems with the encrypted map format, so maybe that revision should be suggested for gen2?

I see a table about the tested firmware revisions at the bottom of https://github.com/dgiese/dustcloud/wiki/Xiaomi-Vacuum-Firmware but from that list I fail to see whether there are any benefits in choosing a newer firmware version? From reports in the dustcloud and valetudo issue tracker I rather get the impression that newer firmware versions are problematic because of additional encryption or wifi shutting off. I think that kind of info should be part of that table.

Also, why does the wiki page suggest using the flasher.py script even though that tool seems to be maintained by python-miio now?

I'm sorry if all of this sounds like a rant of a person who only complains without doing something about the problems but I just got my vacuum yesterday, so I don't know if the things are maybe how they are for a very good reason that I don't know about yet and I don't want to make edits or make pull requests if I am already having the wrong idea about multiple things.

Lastly, some small remarks about imagebulder.sh (the version from git master) which each by themselves probably don't warrant their own issue:

"install and enbale dummycloud" should that be "enable" instead of "enbale"?
--valetudo-path is implemented but not part of the --help output
--disable-xiaomi is listed in the --help output but not implemented anymore (did the option get replaced by another?)
maybe there should be a set -e somewhere? Currently, the script will exit successfully even when some of the steps (like copying files) failed
maybe ssh keys should not be generated by imagebuilder.sh but on the device during first boot? This would allow imagebuilder.sh to output bit-by-bit identical .pkg files, so one could say something like "here is the hash of the official xiaomi image and here is the hash that you should get after running imagebuilder in that version with these options". Such a property would allow better debugging and more assurance for the end user that there is no problem with the image they built. Or is maybe the entropy on the robot too low to generate ssh keys during boot?

cryptomilk commented 5 years ago

I've created a HOWTO you can find here: https://hackmd.io/s/HkwF33oWE

This should be put in the dustcloud wiki here. I haven't had the time to do that yet.

cryptomilk commented 5 years ago

The ssh host keys are only generated once. If you build a new firmware image they are reused so that the keys don't change!

josch commented 5 years ago

@cryptomilk that howto looks great! Thanks!

Yes, I know that the ssh keys are only generated once. What I'm proposing is, to let the xiaomi generate the keys upon first boot. The advantage of generating the keys on the device instead of on the computer running imagebuilder.sh would be, that in your howto you could say: After all these steps, the md5sum of your generated .pkg file should exactly be XYZ. Would that not be an advantage? It would show the user that they did everything correctly. Or is there a reason against generating the keys on the robot? Maybe entropy is very low (like on other arm boards) and thus it would take too long?

cryptomilk commented 5 years ago

Sorry, I don't understand what this should have to do with an md5sum at all?

josch commented 5 years ago

My initial post was very long, so instead of going more back and forth about the reproducible imagebuilder.sh sub-topic in this issue, I opened #193 where we can specifically discuss that topic. Thanks!

josch commented 5 years ago

@cryptomilk could I ask why you recommend firmware version "v11_001702 or newer" in your howto? It seems that Hypfer/Valetudo/issues/40 is still open. Or do you think the issue is completely resolved with the rrlogd-patcher?

Also, if somebody could give me some comments about my remarks in my initial posts, then I can open pull requests that fix them. I just don't want to create pull requests for things that I'm wrong about (like the reproducible firmware topic).

@cryptomilk also thanks for already addressing some of my remarks in 20ee645!

cryptomilk commented 5 years ago

Everything works with the rrlogd-patcher or do you see any issues?

maartenla commented 5 years ago

@cryptomilk Glad I found your tutorial, because the current wiki article is very bad, like josch said. Question though: I can never ssh into my vacuum. It keeps telling me server refused our key. :( I've given up on ever getting SSH access. I have made multiple builds with different keys and after every install I couldn't get access. Or does the build do something weird? I see it outputs 4 ssh key pairs...

I know that my build has worked because I can type the IP Address into my browser and see the Valetudo page.

josch commented 5 years ago

@mlakerveld the keys you are seeing are the private ssh keys generated for the robot. That way, each image will have a different private ssh key.

maartenla commented 5 years ago

Ah Gotcha, so then the only issue I'm having is that I make an image with my personal public key, install that image onto the vacuum, and after that I can't ssh in with my private key.

cryptomilk commented 5 years ago

Did you really provide your public key and not your private one?

maartenla commented 5 years ago

yes, 100% sure. I generated the keys with puttygen. And I did not add the ppk to the firmwarebuilder.

cryptomilk commented 5 years ago

Well, then the question is in what format puttygen stores it and if it is compatible with openssh ...

josch commented 5 years ago

@mlakerveld maybe the ssh problem could be discussed in a separate issue? This one is already not very focused. :D

@cryptomilk rrlogd-patcher did solve the error 500 problem for me after vacuuming once. Now to close Hypfer/Valetudo/issues/40, I think the only missing thing would be to replace the very initial error 500 (the one before the first vacuuming) with a message saying that the robot must've at least vacuumed once before a map can be shown.

I also see that you uploaded your new instructions! That's great, thanks!

This also means that many of my initial remarks became invalid which is also great! I followed the new howto and have a new list of remarks. It's not possible to open pull requests against wiki pages, right? I don't want to just do edits that you don't like, so I'm instead making a new list here:

At the top it says that the latest tested version for v2 is v11_001712 but in the text v11_001792 is used, so I guess v11_001792 is actually the latest tested version?

Maybe it would be nice to put a prefix in front of those commands that are different for v1 and v2 and then list the exact command that is needed for each explicitly? That way, one could easily copypaste the right instruction, resulting in (hopefully) less mistakes.

With some additional options, ssh-keygen will be completely silent and not interactively ask for any additional stuff: ssh-keygen -q -t ed25519 -f ~/.ssh/id_ed25519 -N "" -C "your_email@example.com". By giving the output filename on the commandline, it's also less likely to make errors later when copying the public key into the firmware image.

You don't explicitly spell out the wget commands but I think it would be a good idea to do so because that way, you document which version of each tool to use. If any of the projects you link to is doing a new release which might invalidate the howto, then one will still download the older but working version until the new version was tested and the howto adjusted.

In the same direction: When downloading github files from the raw urls, include the correct git commit hash to make sure to get the right version and not a newer version that wasn't tested. For example instead of downloading https://github.com/Hypfer/Valetudo/raw/master/deployment/valetudo.conf download https://raw.githubusercontent.com/Hypfer/Valetudo/3868ffd85c76795037e053fc173c1e5a7f2cdcec/deployment/valetudo.conf.

The link for downloading firmware image v11_001792 does not work. When I wget https://dustcloud.seemoo.de/public/temp/v11_001792.pkg then I get error 404. A working url seems to be https://dustcloud.seemoo.de/public/xiaomi.vacuum.gen2/original/v11_001792.pkg.

Why is english.pkg downloaded with wget? Earlier in the howto, dustcloud.git was cloned, so the file was already downloaded and does not need to be downloaded again.

Instead or in addition to showing how the resulting directory structure should look like, one could also provide a file that the md5sum tool could use to check all the downloaded artefacts for their integrity. For example with a file hashes.txt like this:

ede620d6b320ea3c4abd5a673e1aec2f  ./rrlogd-patcher/patcher.py
b0531dbd629680bf41643a04134afdc5  ./valetudo/deployment/valetudo.conf
91cb0e3e7280e1c4935cb3ebdc9a236c  ./valetudo/valetudo
c60ea75cc41e422ade9c82de29b78c36  ./firmware/english.pkg
48e72caaf74049965f400981d2a62202  ./firmware/v11_001792.pkg
828961a17045d20b991292bca92acffb  ./dummycloud/doc/dummycloud.conf
785a7d23f3bd66200da2bb8e45ed9c47  ./dummycloud/doc/etc_hosts-snippet.txt
afc38be07f818b90aceeae4e33f052f9  ./dummycloud/doc/etc_rc.local-snippet.txt
c08f32825e2cd83836dd21291cfafc02  ./dummycloud/README.md
d8391ae7b9adba250c0f2f7a3761cb90  ./dummycloud/dummycloud
0306b67868fc85139b963d07e2ce5a17  ./dummycloud/dummycloud_0.1.zip

One could call md5sum --check hashes.txt and get:

./rrlogd-patcher/patcher.py: OK
./valetudo/deployment/valetudo.conf: OK
./valetudo/valetudo: OK
./firmware/english.pkg: OK
./firmware/v11_001792.pkg: OK
./dummycloud/doc/dummycloud.conf: OK
./dummycloud/doc/etc_hosts-snippet.txt: OK
./dummycloud/doc/etc_rc.local-snippet.txt: OK
./dummycloud/README.md: OK
./dummycloud/dummycloud: OK
./dummycloud/dummycloud_0.1.zip: OK

When calling tree one is currently in the rockrobo directory, so the argument to tree should probably be . and not rockrobo.

Just a small spelling mistake: you're -> your

Why is flasher.py used and not mirobo update-firmware from the python-miio package? As I understand it, flasher.py is not developed anymore but got replaced by the update-firmware command in python-miio

In the last part of the howto, the instructions suddenly start using cd instead of pushd/popd as in the beginning. For consistency it would be nice to rely on a single method throughout the whole document.

Lastly, a short note could be added for new users about how to enable the roborock-vacuum-s5_miap_**** wifi network.

cryptomilk commented 5 years ago

It is a wiki with a page history. It is easy to revert things. Feel free to edit it. If you're not sure, then ask.

I've changed everything to v001712 as this is the latest official version for gen2 robots.
I think we should not have a silent ssh-keygen. Normally you should protect them with a key. If someone wants a silent command he can read the manpage.
It might be easier to have complete wget commands, but people should not stupidly follow howto's. They also should think about what they are doing. Else we could just provide a pkg with a ssh key and everyone runs the same image with the same bad security. Not something we want to do.
The english soundfile is obsolete anyway, see the open pull request.
I don't want to include hashes and update the tutorial each time something changes.
flasher.py is part of dustcloud, doesn mirobo update-firmware start a webserver?
What do you mean by enabling the wifi network, it runs if you boot the robot the first time.

josch commented 5 years ago

Quoting Andreas Schneider (2019-02-11 09:19:07)

It is a wiki with a page history. It is easy to revert things. Feel free to edit it. If you're not sure, then ask.

Okay, thanks! I wanted to make sure not to step on anybodies feet. :)

I've changed everything to v001712

You missed three spots which I fixed now.

as this is the latest official version for gen2 robots.

I thought 1792 was the latest official version?

I think we should not have a silent ssh-keygen. Normally you should protect them with a key. If someone wants a silent command he can read the manpage.

Okay.

It might be easier to have complete wget commands, but people should not stupidly follow howto's. They also should think about what they are doing. Else we could just provide a pkg with a ssh key and everyone runs the same image with the same bad security. Not something we want to do.

Okay.

The english soundfile is obsolete anyway, see the open pull request.

Yes, your pull request #192.

I don't want to include hashes and update the tutorial each time something changes.

The point is not to update the tutorial each time something changes but to update it once new versions were tested and deemed to work. If one just pulls the latest versions of imagebuilder.sh, dummycloud, valetudo, patcher.py, official firmware, and so on, then the tutorial might not apply anymore because of changes in any of these components and the user will not know which version of all of these was last known to work.

flasher.py is part of dustcloud, doesn mirobo update-firmware start a webserver?

Yes it does. But so does flasher.py, no?

I can confirm that both work fine but mirobo update-firmware is probably the better maintained codebase, no?

What do you mean by enabling the wifi network, it runs if you boot the robot the first time.

I only got my robot second-hand. After unpacking it, I had to press the spot-clean and wifi buttons together for about three seconds to create a wifi to connect to. I don't know how it works for brand new robots.

cryptomilk commented 5 years ago

If mirobo update-firmware works, then just update the howto accordingly and create a pull request to remove the flasher.py script from the dustcloud repo. No need to have the code duplicated.

For wifi reset we should either add a new sub page to describe it and add a link in the appendix.

cryptomilk commented 5 years ago

Instead of hashes, maybe write down a working combination of version numbers in the appendix

dgiese / dustcloud

some remarks from a new (and successful) imagebuilder/dummycloud/valetudo user #191