search

shadow-1 / yi-hack-v3

Alternative Firmware for Xiaomi Cameras based on Hi3518e Chipset
GNU General Public License v3.0
1.15k stars 152 forks source link

How to compile my own firmware? #23

Open genz10 opened 7 years ago

genz10 commented 7 years ago

Is there is a way to compile my own firmware resulting only home_v201 for Yi Dome?

shadow-1 commented 7 years ago

@genz10 Yes you can compile your own firmware. I have not created build scripts yet to make it easy to compile (largely) automatically. At the moment it is largely a manual process.

Let me know what you would like to do and I will guide you through the process of compiling your own firmware.

genz10 commented 7 years ago

Can you please describe the whole process here ?

shadow-1 commented 7 years ago

@genz10 I will write how to compile your own firmware. However I will probably need to continue over a few posts as I am finding it a bit hard to write everything at once. I already started writing the steps involved once and accidentally lost what I wrote. If you require any further detail in any of the steps, let me know and I will explain that part in greater detail. I will try to write the guide in a step-by-step process as much as possible.

  1. To compile the firmware, you need to have Linux installed. The chipset manufacturer HiSilicon only provides a Linux development toolchain. A virtual machine is sufficient.

  2. The HiSilicon SDK compatible with the HiSilicon Hi3518eV200 chipset needs to be downloaded. On my Box account, I have hosted a number of SDKs. The latest compatible SDK I have is Hi3518E V200R001C01SPC040. Please keep in mind that due to file size restrictions on my Box account, I had to split the SDK into 6 pieces. The SDK can be downloaded from: https://app.box.com/s/cibs7n1mgvhqaqjlidtveegu1uajt5yr

  3. Extract the SDK archive to a convenient location (if you downloaded the SDK I provide, all 6 files are required). This archive contains lots of tools and documentation for software development for several HiSilicon camera chipsets.

  4. The SDK needs to be installed on your Linux machine. Extract the SDK archive. It is located within 01.software --> board directory. For Hi3518E V200R001C01SPC040, the SDK filename is Hi3518E_SDK_V1.0.4.0.tgz.

  5. Depending upon your Linux installation, you may or may not need to modify sdk.unpack and common.sh and change the first line from #!/bin/sh to #!/bin/bash.

  6. Execute sdk.unpack as root to install the SDK and setup all environment variables on your Linux machine.

  7. Clone my git repository. git clone https://github.com/shadow-1/yi-hack-v3.

  8. Create a folder where your home partition will be developed. A good directory structure I would think is yi-hack-v3/build/home.

  9. You will need to obtain a clean 'home' partition for your particular camera. You can use my yi-hack-v3 firmware as a base. However it is a better choice to start with an unmodified base, you can use the recovery images I have hosted on my Box account as a base. Download the appropriate home partition file. For example home_h20.

  10. From my firmware images, strip off the uboot header. For example dd bs=1 skip=64 if=home_h20 of=home_h20.jffs2

  11. The raw jffs2 partition needs to be mounted so that the files can be copied. Example commands are listed below that will need to be adapted to your exact setup. They will need to be executed as root.

    mkdir /mnt/cam
modprobe mtdblock
modprobe mtdram total_size=12992 erase_size=64
dd if=home_h20.jffs2 of=/dev/mtdblock0
mount -t jffs2 /dev/mtdblock0 /mnt/cam

  12. Copy the contents of the firmware image to your development area. For example cp -R /mnt/cam/* ./yi-hack-v3/build/home.

  13. Unmount the partition file. Execute the following example code as root umount /mnt/cam.

  14. For proxychains-ng support, I replace the cloud application with a script. The cloud application needs to be renamed. Example command is as follows mv ./yi-hack-v3/build/home/app/cloudAPI ./yi-hack-v3/build/home/app/cloudAPI_real.

  15. The additional files for the updated firmware need to be copied to the home partition. Example command is as follows cp -R ./yi-hack-v3/src/home/* ./yi-hack-v3/build/home

shadow-1 commented 7 years ago

  1. I am assuming you are going to be building yi-hack-v3 0.1.3. For yi-hack-v3, programs proxychains-ng and proccgi need to be cross-compiled for the camera and placed into the correct locations within the home partition.

  2. Cross compile proxychains-ng. Obtain ProxyChains-ng. I used the latest stable version in my custom firmware. However you can use the latest development release on their git page as a source. I have already provided config.mak file which has all appropriate compiler configuration setup. Example commands:

    cd yi-hack-v3/src/proxychains-ng
git clone https://github.com/rofl0r/proxychains-ng
./configure
make
make install

  3. Copy the appropriate application and library files which have been compiled to the appropriate folders. Example commands as follows:

    cp ./yi-hack-v3/src/proxychains-ng/_install/bin/proxychains4 ./yi-hack-v3/build/home/app/localbin/proxychains4
cp ./yi-hack-v3/src/proxychains-ng/_install/lib/poxychains4.so ./yi-hack-v3/build/home/app/locallib/poxychains4.so

  4. Cross compile proccgi. Information on proccgi is available here: http://www.fpx.de/fp/Software/ProcCGI.html I have created a basic Makefile to compile this tiny program. Example commands:

    cd yi-hack-v3/src/proccgi
wget http://www.fpx.de/fp/Software/proccgi.c
make
make install

  5. Copy the appropriate application which has been compiled to the appropriate folder. Example command as follows:

    cp ./yi-hack-v3/src/proccgi/_install/bin/proccgi ./yi-hack-v3/build/home/app/localbin/proccgi
shadow-1 commented 7 years ago

  1. For my firmware, I create a simple info file and store some static data within it. When I get around to creating proper build scripts, I will automatically generate this file. Create the following file ./yi-hack-v3/build/home/app/.hackinfo.

  2. Populate .hackinfo file with the following contents. Example below:

    CAMERA=Yi 1080p Dome
VERSION=0.1.3

  3. Now we create the firmware file. First we need to package our build directory into a jffs2 partition file. Then we need to wrap the partition with the appropriate u-boot header. Example commands are as follows:

    cd yi-hack-v3/build
mkfs.jffs2 -l -e 64 -r ./home -o home_h20.jffs2
mkimage -A arm -T filesystem -C none -n 0001-hi3518-home -d home_h20.jffs2 home_h20

  4. The newly generated home partition firmware file is now ready to be flashed onto the camera. The generated firmware file from the above command is home_h20.

Generating a replacement rootfs partition is very much the same process as described above. However with a different partition and base firmware file. Keep in mind that the free space on the rootfs partition is extremely limited. Hardly any extra programs/configurations can fit onto the rootfs partition.

I am in the process of moving the web server away from httpd to libwebsockets. The web pages are moving away from utilising CGI to utilising WebSockets. To compile yi-hack-v3 0.1.4 (when it gets released) onwards. Proccgi will no longer be required. However libuv, libwebsockets along with my custom libwebsockets plugins will need to be cross compiled for the HiSilicon Hi3518eV200 chipset. I have not found an easy way to store the compiler configuration into a config file or Makefile yet for libuv and libwebsockets. However I have successfully cross compiled these programs/libraries and tested them.

genz10 commented 7 years ago

Hi,

Thanks for your reply. I haven't try this but someday i'll.

genz10 commented 7 years ago

@shadow-1

What's the different between home partition and root partition ?

shadow-1 commented 7 years ago

@genz10 I will assume you are familiar with Windows. The rootfs partition contains everything that is critical to the OS. It is something like the C:/Windows folder on Windows OS. In this case, the rootfs partition is incredibly small and only contains the base essentials for the OS.

The home partition contains all other information stored on the camera such as the Xiaomi programs. It is something like the My Documents folder and C:\Program Files on Windows. So it contains important information, however not critical to the OS.

genz10 commented 7 years ago

@shadow-1 Thanks for your reply, do this tutorial still valid for your latest src ? I don't know what i'm missed. I've successfully created my own firmware (home_v201) but still using your rootfs_201 and it's not working here :(

shadow-1 commented 7 years ago

@genz10 The latest firmware is a bit different to the previous 0.1.3 release. It utilises libwebsockets along with custom plugins which I made. Libwebsockets has a dependency on libuv so this also needs to be cross-compiled.

I do not have a Makefile or config files for these. However I can post the commands required to cross-compile these programs if you wish. The trick is with the parameters passed onto the configure script.

In addition the original Yi Dome and the Yi Home 17CN/27US have less memory than the Yi 1080p Home and Yi 1080p Dome cameras. I have found that you have to reduce the firmware filesize for these cameras for it to accept the firmware image. So what I did is compress yi-hack-v3 with 7zip format (same as the official firmware files) and automatically extract the archive when the firmware boots up for the first time.

genz10 commented 7 years ago

@shadow-1

Yes please if you don't mind

shadow-1 commented 7 years ago

@genz10 No problems.

First we need to download libuv. In my case I utilised libuv 1.11.0. The compilation commands are as follows:

./autogen.sh
./configure --host=arm-hisiv300-linux --prefix=$PWD/_install
export DESTDIR="" && make install

Next, we need to download libwebsockets. I have modified libwebsockets a bit to fit the requirements of this project. I have generated a patch file with these changes. The compilation script is as follows:

patch < libwebsockets.patch
cmake -DCMAKE_INSTALL_PREFIX:PATH="/home/yi-hack-v3/" -DCMAKE_C_FLAGS="-march=armv5te -mcpu=arm926ej-s" -DCMAKE_EXE_LINKER_FLAGS:STRING="-lm -ldl -lpthread" -DCMAKE_SYSTEM_NAME=linux -DCMAKE_C_COMPILER=arm-hisiv300-linux-gcc -DLIBUV_INCLUDE_DIRS="$PWD/../libuv-1.11.0/_install/include" -DLIBUV_LIBRARIES="$PWD/../libuv-1.11.0/_install/lib/libuv.so" -DLWS_WITH_LWSWS=1 -DLWS_WITH_SSL=OFF -DLWS_WITHOUT_BUILTIN_SHA1=OFF -DLWS_WITHOUT_DAEMONIZE:BOOL=OFF -DLWS_WITH_CGI:BOOL=ON -DLWS_WITHOUT_EXTENSIONS:BOOL=ON -DLWS_WITH_ZLIB:BOOL=OFF -DLWS_WITH_ZIP_FOPS:BOOL=OFF -DLWS_WITHOUT_TESTAPPS:BOOL=ON
make
export DESTDIR="$PWD/_install/" && make install

These commands may need to be tweaked slightly to fit in with your exact build environment.

Muradunal1 commented 6 years ago

Hi Guys, Thank you for your sharing "shadow-1". I wanted to desing an IP Camera with that ic (HiSilicon Hi3518eV200). But i couldnt find any english document about it. I am trying to reach HiSilicon and their distributors but non of them dont reply my emails. I searched from google and baidu using tanslator. I found some of the documents in Chinese. You already shared SDK. I am downloading it. Do you have English documents about the chip?

shadow-1 commented 6 years ago

@Muradunal1 If you download the latest SDK for the HiSilcion Hi3518eV200 that I have hosted (Hi3518E V200R001C01SPC040). There is plenty of documentation in both Chinese and English. Have a look at the following directories for English documentation:

Hi3518E V200R001C01SPC040\00.hardware\board\document_en
Hi3518E V200R001C01SPC040\00.hardware\chip\document_en
Hi3518E V200R001C01SPC040\01.software\board\document_en
Hi3518E V200R001C01SPC040\01.software\pc\DEC_LIB\documents_en
Hi3518E V200R001C01SPC040\01.software\pc\Hi_TOOl\documents_en
Hi3518E V200R001C01SPC040\01.software\pc\PQ_TOOL\documents_en
Hi3518E V200R001C01SPC040\02.Only for Reference\01.Test Report\documents_en
Hi3518E V200R001C01SPC040\02.Only for Reference\02.software\documents_en
Hi3518E V200R001C01SPC040\02.Only for Reference\03.hardware\documents_en
Hi3518E V200R001C01SPC040\02.Only for Reference\04.other\documents_en
Muradunal1 commented 6 years ago

thank you. it is incredible. I searched in google but i couldnt find them. How did you find them? Where can i find them in the future? I have some questions.

Muradunal1 commented 6 years ago

sorry i mean Hi3518E chipset

shadow-1 commented 6 years ago

@Muradunal1 English documentation comes with the SDK. The development board is designed by HiSilicon.

From memory I remember seeing basic boards that look like development boards for sale on Aliexpress/Alibaba.

If you buy a camera based on HiSilicon Hi3518eV200, then you can use Hi3518E V200R001C01SPC040 SDK for development. For the regular HiSilicon Hi3518eV100, I have hosted Hi3518E_V100R001C01SPC081 SDK for development.

imsamurai commented 6 years ago

Hi, thanks for work! Is there a way to run curl on camera? I want to use dropbox uploader

shadow-1 commented 6 years ago

@imsamurai I have been able to successfully cross-compile curl for this camera and can confirm that it works perfectly. However curl is a C++ program which means that it is quite large when compiled. I don't think there will be enough free space on the camera to store curl. However it can be stored on the microSD card and run from there.

If you need help with the compilation, I can help.

kapollo commented 6 years ago

@shadow-1 do you happen to know if the Hi3518E SDKs are backward compatible with Hi3518C ? I do seem to see a lot of Hi3518C references. I'm quite fresh into HiSilicon SDKs themselves and just putting things together.

gooman-uk commented 5 years ago

@shadow-1 I'm trying to install the SDK to cross-compile some other tools (particularly the Mosquitto MQTT client).

I note the following in your SDK install instructions:

6. Execute sdk.unpack as root to install the SDK and setup all environment variables on your Linux machine.

I've done that, but it doesn't seem to set any environment variables. As a test, I've compiled curl as per your other instructions, but it generates an i386 binary.

Where am I going wrong?

420pootang69 commented 5 years ago

@shadow-1

Hi,

Just a quick one.

I'm trying to change the hard coded max bitrate settings on a camera with a 3516D chip. I've obtained the firmware for it, but for the life of me I can't figure out where these settings are stored. Would you happen to know?

The issue I'm having is that the camera seems to have a max bitrate limit that gets set depending on encoding format, resolution and framerate. I want to alter these.

If you could point me in the right direction, that'd be great.

Thanks.

paoloboatto commented 4 years ago

Hi, I'm going to develop a new system using a chip of the family HiSi35xx. I have the following questions:

1 - I found a chinese company (Travellinux) that sells a development board but they don't answer to any e-mail. Does anybody know if there are other companies that sell a development board for that chip family?

2 - If I can't find a development board I have to use a production board; is it possible to debug the firmware through the USB port or another interface; should I use a dedicated programmer?

Best regards Paolo Boatto

Redwid commented 4 years ago

Hi, I'm looking to compile application to the first yi cam revision. All binary from yi-hack-v3 and yi-hack-v4 are not working there. I've set up docker container with Hi3518_SDK_V1.0.7.0. Compiled simple hello word app, uploaded into my camera. And getting Illegal instruction error. The file utility output is the same if I do compare with curl utility from the camera.

For example curl: curl: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, stripped

My hello world: hello-arm: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-, not stripped

What could be wrong? Why Illegal instruction?

Redwid commented 4 years ago

In additional greadelf -A curl Attribute Section: aeabi File Attributes Tag_CPU_name: "5TE" Tag_CPU_arch: v5TE Tag_ARM_ISA_use: Yes Tag_THUMB_ISA_use: Thumb-1 Tag_ABI_PCS_wchar_t: 4 Tag_ABI_FP_denormal: Needed Tag_ABI_FP_exceptions: Needed Tag_ABI_FP_number_model: IEEE 754 Tag_ABI_align_needed: 8-byte Tag_ABI_enum_size: int Tag_DIV_use: Not allowed

And my app looks the same: greadelf -A hello-arm Attribute Section: aeabi File Attributes Tag_CPU_name: "5TE" Tag_CPU_arch: v5TE Tag_ARM_ISA_use: Yes Tag_THUMB_ISA_use: Thumb-1 Tag_ABI_PCS_wchar_t: 4 Tag_ABI_FP_denormal: Needed Tag_ABI_FP_exceptions: Needed Tag_ABI_FP_number_model: IEEE 754 Tag_ABI_align_needed: 8-byte Tag_ABI_enum_size: int Tag_DIV_use: Not allowed