[yi-hack-v4] Support and continue the development - Trying to contact shadow-1

TheCrypt0 commented 5 years ago

Hi everyone, first of all I'd like to say thanks to @shadow-1 and to all of the contributors for the effort and the work that has been put into this project. That said, I'd like to create an updated version of the firmware called yi-hack-v4.

Long story short, I started to work on the RTSP implementation (previously developed by @andy2301) for my Yi Home camera and I created a daemon called viewd to get /tmp/view circular buffer information. Meanwhile, I started to dig into the open and closed issues of the various yi-hack projects and I noticed that there are a some cool tweaks and scripts that could be implemented directly in the firmware (like @frekel's Telegram alarm system).

I tired to deploy my custom version of the firmware following the fragmentary informations I could find in the issues and I managed to:

Compile and deploy my own programs using the SDK
Make the RTSP server almost work (crystal clear image for about 2/5 minutes, then the client needs to reload the stream), with the help of viewd (https://github.com/shadow-1/yi-hack-v3/issues/126#issuecomment-450201497)
Decrypt and extract the yi firmware file by examining the script that handles this routine (I then discovered that shadow-1 documented the process in another issue)
Unpack the recovery (stock) firmware images
Repack the images including the uboot header (for rootfs I used yi-hack-v3's busybox binaries because the only info I could find was about copying the version present in the Yi Camera because it had more commands enabled)

As shadow pointed out, he hasn't been able to continue the development of the firmware for some time and it's been almost a year after the last release. Since my daily job is related to this field (development hardware/software and adaptation of the Linux Kernel for our company's ARM embedded systems) I'd happily contribute!

I'm now trying to contact him or @batrarobin (which said that he personally knows shadow https://github.com/shadow-1/yi-hack-v3/issues/15#issuecomment-401639700) to give me a couple of hints about the firmware, especially on the BusyBox stuff.

If someone have any contacts (or hints about the firmware) please reply to this issue.

Thanks

Edit: corrected typo.

batrarobin commented 5 years ago

Hey @TheCrypt0

I am pointing @shadow-1 to this thread, let's see if he has some time to help us out.

Cheers.

TheCrypt0 commented 5 years ago

Hi @batrarobin,

thanks for the quick reply, looking forward to hearing from him.

breakfixrepeat commented 5 years ago

@TheCrypt0 I stumbled upon this repo today, hope you find it useful - https://github.com/Necromix/yi-vencrtsp

shadow-1 commented 5 years ago

Hi @TheCrypt0 Yes you are correct that I have not had time to work on this custom firmware for quite a long time. I have other higher priority projects for the time being.

It is good to see that others have picked up what I have worked on and started extending the functionality further.

Please feel free to ask any questions.

The last time I played around getting the RTSP working, I was disassembling the custom Xiaomi applications and examining how they work. There are several applications running on the camera which share information between each other. I believe it is likely that Memory Mapped File(s) are being used to exchange information between the various processes. The '/tmp/view' file is being utilised by the Xiaomi applications and I was looking at how the file is being read by the official applications so this can be replicated for our own purposes (namely RTSP).

I wanted to go down this path because I envisaged this firmware allowing to make use of the Xiaomi smartphone app and/or RTSP concurrently.

There are other approaches which can be investigated (which some other custom firmwares have followed). However this inevidently means ditching the Xiaomi applications altogether (killing Xiaomi smartphone app support). It also means an alternative method for setting up WiFi is required.

EpicLPer commented 5 years ago

Awesome! I hope you guys will be able to get this up and running on the Outdoor 1080p too since that's the one I'd love to use for use with Blue Iris. All it really has to do is provide a stable RTSP stream, nothing more, and I'd be happy already ;)

Btw. wouldn't it be better to make a fork for your project now? I don't think that "hiding" something new and way better in just a normal issue would be good, have seen way too many project getting far too cluttered that way.

TheCrypt0 commented 5 years ago

Hi @shadow-1! First of all I want to say thank you for the time and work you've put into the project. Now a quick update and some questions:

I spent the last week writing scripts and gathering informations about the process of creating and packing a custom firmware.
The process is now almost straightforward with just a few scripts to call that will:
1. Fetch all the submodules required (like proxychains, busybox, etc.)
2. Unpack the original rootfs_XXX and home_XXX images to create a suitable base sysroot to work with.
3. Init each submodule and patch it with the required configs or anything else needed.
4. Compile each submodule and copy the "result" (executable, library or else) to the build directory.
5. Combine the sysroot, the static files (like rootfs/etc/ configurations) and the build files into a final folder
6. Pack the folder into a jffs2 filesystem and then into a flashable rootfs_XXX and home_XXX images.
7. Flash the images onto the camera.
The scripts are in a beta stage but the process works fine. The time required from cloning the repo to have a ready-to-flash custom firmware is around ~5 minutes (depending on the processing power).
The base firmware I'm using is the latest available version.

Some questions:

I noticed that you didn't include a LICENSE file in the repo and without it I'm not legally allowed to copy and redistribute the scripts/configs/patches you wrote. I'm sure it won't be an issue to include your work into the project (the README already has all the acknowledgements to you and the other people), but just to be safe I prefer to ask what kind of license you where thinking to choose. I personally prefer GPLv2/3 for this kind of work.
The stock firmware update images use 7z compression for some specific files (the big ones I think). ~Why is it necessary to compress yi-hack too into a 7z file? - Disclaimer: I'm experimenting and testing this process (and I might be wrong) but when the yi-hack folder isn't compressed (then extracted in the home partition IIRC via system_init.sh) the scripts or executables invoked by init.d just don't seem to work. Btw further testing is needed as I haven't tried to compress the folder yet when repacking the fw.~ Edit: the home image must be <=8MiB, that's the reason why other executables are 7zipped.
How did you handle all the required directories? Sysroot, build folders, static files (example: /etc/profile). My dir structure is already defined but it needs to be modified according to the following question.
Are the original libraries the same between the cameras? (Excluding the Yi Home that features the different chipset) To compile the submodules we need to specify the location of the libraries in the config files. If the libs are the same, the process of compile will be much more easy (1 executable for all the cameras).

Sorry for the wall of text, I just came home and surely there are some other questions, but I currently don't remember them.

The RTSP server I'm currently working on reads directly from /tmp/view so the app works correctly. It just needs some tweaks because after 2/5 minutes (it depends from the buffer head offset) it stops sending the stream, a new request from the client needs to be made to re-init the server. It may be caused by a concurrent write/read to the buffer between rmm and the server.

Btw, Happy New Year!

@imthebossofme I tried to use @Necromix's rtsp implementation too but without much success in compiling with the SDK. My debian vm was a little bit messy and I think that's the cause.

However I created a daemon to allow the buffer to be parsed and checked (https://github.com/shadow-1/yi-hack-v3/issues/126#issuecomment-449659465) and managed to get a pretty consistent image with some tweaks to @andy2301's rtsp2303 server.

@EpicLPer If the chipset and the boot process is the same it should be possible.

Btw. wouldn't it be better to make a fork for your project now? I don't think that "hiding" something new and way better in just a normal issue would be good, have seen way too many project getting far too cluttered that way.

Do you mean the v4 or the current version? If you meant mine, I'm waiting to make the repo public because I want to make sure that everything works fine.

lupo-ch commented 5 years ago

I own few Yi Home cameras US47 and 1 old chinese - I'm available if you need some testing or some other way I can help (donation also) for real RTSP support in this cameras.

BlueBlueBlob commented 5 years ago

Hi @TheCrypt0 First good job 👍 Can you make a public repo without providing rootfs and home images (to be sure people doesn't rush on beta stuff) I'm waiting my y19 (last one died) aka Yi Cloud Dome, I will able to help for this model. Not sure if I can help you with rtsp, I'm a bash/Linux Dev. I will try to compile for y19.

Best regards and good job

TheCrypt0 commented 5 years ago

Hi @lupo-ch, yes, I'll need someone to test the new firmware on the various versions of the cameras. I'm expecting to finish the build scripts in the next days, I'll then make the repo public. I've just published viewd (binary only) to allow the people to test it, this daemon will be used to "drive" the RTSP server.

Hey @BlueBlueBlob, thanks! As I said above, I'm waiting to finish to write the scripts to publish the repo. Even without providing the beta images it will be very easy to build a custom firmware to test, you'll just need to call a few scripts and place the generated images on the sd.

I'm looking for another more direct way to communicate (slack? discord?) to provide you the beta versions of the firmware.

Greetings

EpicLPer commented 5 years ago

@TheCrypt0 Well, a Discord Server definitely would help :) Or even just a Discord group DM would do, you can add me if you want: EpicLPer#0001

lupo-ch commented 5 years ago

maybe just a telegram group / channel?

shadow-1 commented 5 years ago

@TheCrypt0 Sounds like you are on the right track with the build scripts. I was going to write some build scripts but never got around to doing it. Most of the commands I have in my head and I documented the process in a issue post.

Answers to your questions are as follows:

Good point, I did not choose a license for this project. I will add GPLv3 license to the repository.
I see that you have discovered why I packed the firmware in a 7zip archive. It is to keep the file size down so it can be flashed on all cameras. There is a hard limit for the maximum file size for the firmware file otherwise UBoot refuses to flash the firmware. I did not find out exactly what is the maximum file size but it is roughly 8mb. Keep in mind this hard limit only applies to some cameras. I can confirm that the Yi 1080p Dome camera has no firmware file size limit (besides the maximum allowed on the partition of course). I decided to use 7zip format as it is used by the official firmware and all the binaries are already present on the camera.
I start off with a 'base' or initial sysroot in a build directory (based off the official firmware) and replace the relevant files from the repository. I did not create a sysroot completely from scratch. When I compiled a release, I had a build directory for each type of camera (each based off a different 'base' or initial starting point) and ran the same build script for each type of camera.

Keep in mind that whilst the open source programs I cross-compile along with my own custom plugins work on all cameras. Each camera is unique with differing WiFi chipsets, different image sensors, different Xiaomi binaries etc. So it is not possible to create a single universal firmware that will work on all cameras. That is why I decided to start with an initial sysroot which is unique for each type of camera and have the modifications I make universal.

The cameras supported by this firmware run on exactly the same chipset (Hi3518ev200) and have exactly the same directory structure. All cross-compiled applications work across all cameras. The exception to this is the original Yi Home which is based on a different chipset which requires the use of a different cross-compiler. The directory structure is different, so I added a build switch to choose between Hi3518 chipset and Hi3518ev200 chipset. I decided to sort this out largely at compile time rather than run time (unless it is super simple).

EpicLPer commented 5 years ago

@shadow-1 Do you know how to escape the running command when viewing the serial output? I can't Ctrl - C out of it neither does spamming it at boot work. At least on 3.0.0.0D, I hope they didn't just fix that...

batrarobin commented 5 years ago

Maybe your RX wire is connected correctly but not the TX wire so not taking any commands from keyboard? Generally pressing Enter key brings you to the command prompt.

EpicLPer commented 5 years ago

@batrarobin I made sure it's actually sending commands by disconnecting it and then pressing buttons, once that happens I don't see any additional letters/Ctrl-C command appearing in the terminal anymore so it's indeed working and sending everything over. Except if I have to set something inside of Putty for that to work but I don't think so?

shadow-1 commented 5 years ago

@EpicLPer You are talking about the Yi Outdoor camera?

If so, I do not have this camera, so I can not know for sure. However others have said that they have experienced exactly the same issue. I believe Xiaomi disabled access to the terminal over serial. This is probably achieved by deleting or commenting out a line in one of the init scripts (I did not investigate).

Some brave people have tried flashing yi-hack-v3 designed for the Yi 1080p Home on their Yi 1080p Outdoor (by simply renaming the firmware file). Apparently flashing the rootfs partition is successful and results in having access to the terminal. However does not always boot successfully.

Check out this thread which has a LOT of detail: https://github.com/shadow-1/yi-hack-v3/issues/82

If you decide to go down this path, it is high risk as we do not have a backup of the stock firmware. It is possible to soft brick the camera and have no way of restoring the stock firmware at this stage.

EpicLPer commented 5 years ago

@shadow-1 Well that's the thing, I don't want to flash any custom stuff to it before I don't have a backup of the stock firmware. I tried getting those files but yeah, these Chinese sh... disabled the terminal it seems :(

Any other way you can think of to get a backup without desoldering the flash? Cause I'm definitely too unskilled to do that, yet alone dump it afterwards...

shadow-1 commented 5 years ago

@EpicLPer Perhaps it would be best if you start a new Issue for this or continue from one of the two or three existing issues. Can you post a boot log with a FAT32 formatted microSD card inserted.

From the huge thread, it appears that perhaps the boot script may look to execute a shell script off the microSD card. If this is the case, it may be possible to dump the stock firmware to microSD card by writing a shell script which is automatically executed on boot.

BlueBlueBlob commented 5 years ago

@EpicLPer : same issue on Yi Cloud y19 @shadow-1 : I will post log in my open issue

TheCrypt0 commented 5 years ago

Discord Server

I've just created a Discord server with a couple of channels. Invite link.

@shadow-1

Thanks for your time, I see that this new version is beginning to take shape. I'm now almost ready to publish a beta version of it. Thanks for the license too!

I'm having some troubles compiling libwebsockets. When the patch file you created is applied I can't find the correct config file to patch. May I ask you which version or repo you used to generate lwsws and the required shared-libraries?

Greetings

@batrarobin @BlueBlueBlob

The serial console looks completely locked on the Yi Outdoor 1080p camera. Yesterday I was in a discord call with @EpicLPer for like 1 hour trying to figure out a way to stop the boot process or to get the shell access whout luck.

We tried a bunch of different keys, including the magic sysrq, but without luck. The RX pin was correctly soldered since when pressing a key the result was echoed in the serial terminal.

Anyway, let's continue this discussion here https://github.com/shadow-1/yi-hack-v3/issues/242 or here https://github.com/shadow-1/yi-hack-v3/issues/228

shadow-1 commented 5 years ago

@TheCrypt0 I have to dig this up, it would be from some time ago.

The patch I created for lwsws fixes a bug in their source code. Probably doesn't affect most people. However I was squeezing the application to a minimum file size and minimum memory footprint and disabled absolutely every feature which is not necessary via their init script. From memory, after removing a a whole lot of features (which had their own library dependencies) with their init script, the source code still referenced libraries which were removed from the build process. Hence I could not cross-compile lwsws.

This issue may have been resolved in lwsws by now. The patch may not be necessary.

shadow-1 commented 5 years ago

@TheCrypt0 I had a look at the version of libwebsockets I used at the time when I created this firmware. It was based on the official git repository. Commit number d383b94.

Link: https://github.com/warmcat/libwebsockets/tree/d383b94dbf16026478c26f9ba2d2fc15dcfd0127

TheCrypt0 commented 5 years ago

@shadow-1

Thank you. Since I'm not familiar with libwebsockets my plan was to just compile and place the files in the firmware, just to make the webserver work without creating another from scratch. I'll learn how to use the library when everything will be stable.

Greetings

shadow-1 commented 5 years ago

@TheCrypt0 I was in the same position, I did not want to create a webserver from scratch. I found lwsws which was developed by the libwebsockets team perfect. Very powerful making use of WebSockets yet very lean which is perfect for this limited embedded environment.

I found making use of WebSockets much more powerful for server side processing compared to CGI which can be utilised with httpd included in busybox.

enmejia commented 5 years ago

Hi, awesome project! Would this hack work with the Yi Home 1080p that I just got from Amazon with firmware version 4.5 ?

pserranoa commented 5 years ago

Hi, Same problem here, anybody could hack with firmware version 4.5?

rakimbadu commented 5 years ago

Me too, firmware 4.5.0.0A, hope to see a working custom firmware for this version

EpicLPer commented 5 years ago

To the last 3 guys here go check https://github.com/TheCrypt0/yi-hack-v4 instead of this outdated one here.

shadow-1 / yi-hack-v3

[yi-hack-v4] Support and continue the development - Trying to contact shadow-1 #241

Discord Server