Reproduction of Joying boot images

[lbdroid]

Joying uses non-standard boot and recovery images. The 6.0 boot.img appears to be a lot "closer" to standard than previously, but still off.

Utilities; unmkbootimg: https://forum.xda-developers.com/showthread.php?t=1877807 mkbootimg: https://android.googlesource.com/platform/system/core/+/android-7.1.2_r11/mkbootimg/mkbootimg

The 6.0 boot.img is pretty easy to deal with. unmkbootimg boot.img spits out this;

unmkbootimg version 1.2 - Mikael Q Kuisma <kuisma@ping.se>
Kernel size 8388608
Kernel address 0x2600000
Ramdisk size 5400125
Ramdisk address 0x400000
Secondary size 0
Secondary address 0x2400010
Kernel tags address 0x100
Flash page size 4096
Board name is ""
Command line "idle=halt notsc androidboot.hardware=sofiaboard apic=sofia nolapic_pm firmware_class.path=/system/vendor/firmware androidboot.selinux=disabled nolapic_timer x86_intel_xgold_timer=soctimer_only vmalloc=512m slub_max_order=0 uvcvideo.en_autosuspend=0 console=ttyFIQ0,115200n8 earlyprintk=xgold"

*** WARNING ****
This image is built using NON-standard mkbootimg!
OFF_KERNEL_ADDR is 0x02600000
OFF_RAMDISK_ADDR is 0x00400000
OFF_SECOND_ADDR is 0x02400010
Please modify mkbootimg.c using the above values to build your image.
****************

Extracting kernel to file zImage ...
Extracting root filesystem to file initramfs.cpio.gz ...
All done.
---------------
To recompile this image, use:
  mkbootimg --kernel zImage --ramdisk initramfs.cpio.gz --base 0x0 --cmdline 'idle=halt notsc androidboot.hardware=sofiaboard apic=sofia nolapic_pm firmware_class.path=/system/vendor/firmware androidboot.selinux=disabled nolapic_timer x86_intel_xgold_timer=soctimer_only vmalloc=512m slub_max_order=0 uvcvideo.en_autosuspend=0 console=ttyFIQ0,115200n8 earlyprintk=xgold' --pagesize 4096 -o new_boot.img
---------------

Nice thing is that it doesn't actually know what its talking about when it is telling you to modify bootimg.c. This is not needed in the newer mkbootimg, which is actually a python script.

The command to recompile is close to what is needed, you just need to add in the offsets.

--kernel_offset 0x02600000 --ramdisk_offset 0x00400000 --second_offset 0x02400010

The result is a new_boot.img that is byte for byte identical to the original.

$ md5sum *.img
3cb4ecb1cf9b3a2924e725fe1d91ccd3  boot.img
3cb4ecb1cf9b3a2924e725fe1d91ccd3  new_boot.img

[lbdroid]

The recovery is a bit more challenging. If we do the exact same procedure, we end up with a file that is quite significantly different.

unmkbootimg recovery.img:

unmkbootimg version 1.2 - Mikael Q Kuisma <kuisma@ping.se>
Kernel size 8388608
Kernel address 0x2600000
Ramdisk size 3334559
Ramdisk address 0x500000
Secondary size 204249
Secondary address 0x2400010
Kernel tags address 0x100
Flash page size 4096
Board name is ""
Command line "idle=halt notsc androidboot.hardware=sofiaboard apic=sofia nolapic_pm firmware_class.path=/system/vendor/firmware androidboot.selinux=disabled nolapic_timer x86_intel_xgold_timer=soctimer_only vmalloc=512m slub_max_order=0 uvcvideo.en_autosuspend=0 console=ttyFIQ0,115200n8 earlyprintk=xgold"

*** WARNING ****
This image is built using NON-standard mkbootimg!
OFF_KERNEL_ADDR is 0x02600000
OFF_RAMDISK_ADDR is 0x00500000
OFF_SECOND_ADDR is 0x02400010
Please modify mkbootimg.c using the above values to build your image.
****************

Extracting kernel to file zImage ...
Extracting root filesystem to file initramfs.cpio.gz ...
Extracting second to file second.gz ...
All done.
---------------
To recompile this image, use:
  mkbootimg --kernel zImage --ramdisk initramfs.cpio.gz --base 0x0 --cmdline 'idle=halt notsc androidboot.hardware=sofiaboard apic=sofia nolapic_pm firmware_class.path=/system/vendor/firmware androidboot.selinux=disabled nolapic_timer x86_intel_xgold_timer=soctimer_only vmalloc=512m slub_max_order=0 uvcvideo.en_autosuspend=0 console=ttyFIQ0,115200n8 earlyprintk=xgold' --pagesize 4096 -o new_boot.img
---------------

And then make it back up: (NOTE: The unmkbootimg command is too stupid to add in the --second parameter, so it completely forgets about adding that part in. Don't forget to add it!!!)

mkbootimg --kernel zImage --ramdisk initramfs.cpio.gz --base 0x0 --cmdline 'idle=halt notsc androidboot.hardware=sofiaboard apic=sofia nolapic_pm firmware_class.path=/system/vendor/firmware androidboot.selinux=disabled nolapic_timer x86_intel_xgold_timer=soctimer_only vmalloc=512m slub_max_order=0 uvcvideo.en_autosuspend=0 console=ttyFIQ0,115200n8 earlyprintk=xgold' --pagesize 4096 --kernel_offset 0x02600000 --ramdisk_offset 0x00500000 --second second.gz --second_offset 0x02400010 -o new_boot.img

But we don't even need md5sum to see that they are different;

$ ls -l *.img
-rw-rw-r--. 1 user group 11935744 May 12 10:27 new_boot.img
-rw-rw-r--. 1 user group 16777216 May 12 10:25 recovery.img

So, what is going on here?

Well, a quick visual scan of the two files shows that the starting parts are pretty similar, if not identical, so checking on that more precisely....

$ dd if=recovery.img bs=1 count=11935744 > rec_front.img
$ md5sum *.img
8d0cd8125f975c2edba343cd02e6e836  new_boot.img
8d0cd8125f975c2edba343cd02e6e836  recovery_front.img
715e08287db6c7605bb115e4112ad560  recovery.img

This means that we are doing good. We are able to at least fully reproduce the "first bit" of the recovery.img.

Now the thing is, we've actually cut off a big chunk of the end of the file. Everything from 0x00b62000 onwards. It looks like the end of the file is actually at roughly 0x00be4fff. The rest of it is just 00's padding.

So lets just cut the original recovery.img off there.

dd if=recovery.img bs=1 count=12472320 > recovery.img.truncated
dd if=recovery.img.truncated bs=1 skip=11935744 > extra_blob.bin

Note: 12472320 is 0x00be5000 in base-10.

$ cp new_boot.img new_boot_scratch.img
$ cat extra_blob.bin >> new_boot_scratch.img
$ md5sum recovery.img.truncated new_boot_scratch.img 
d4b3039edd182d0adcb226c1abddf226  recovery.img.truncated
d4b3039edd182d0adcb226c1abddf226  new_boot_scratch.img

So now at least we can reproduce it by appending this extra blob onto the end of the file.

[lbdroid]

Now that extra_blob.bin... just what the heck IS IT?

It appears to be divided into three distinct sections;

Start 0-0x5ff: has some stuff about mountainview california US and android@android.com. Sounds pretty googly. Is that the test key?

Middle 0x600-0x4ffff: No idea.

End 0x5000-end: Appears to be a DTB. Does that mean that the middle part is a kernel?

[lbdroid]

I think that the right way to modify the recovery ramdisk is this;

1) unmkbootimg 2) edit the initramfs.cpio.gz in the usual way 3) mkbootimg as;

mkbootimg --kernel zImage --ramdisk initramfs_new.cpio.gz --base 0x0 --cmdline 'idle=halt notsc androidboot.hardware=sofiaboard apic=sofia nolapic_pm firmware_class.path=/system/vendor/firmware androidboot.selinux=disabled nolapic_timer x86_intel_xgold_timer=soctimer_only vmalloc=512m slub_max_order=0 uvcvideo.en_autosuspend=0 console=ttyFIQ0,115200n8 earlyprintk=xgold' --pagesize 4096 --kernel_offset 0x02600000 --ramdisk_offset 0x00500000 --second second.gz --second_offset 0x02400010 -o new_boot.img

4) cat extra_blob.bin >> new_boot.img

[lbdroid]

Uploaded the extra blob here; https://github.com/lbdroid/JoyingIntelStuff/blob/master/extra_blob.bin

[gtxaspec]

6.0 stock does not enumerate the UVC device. :(

[lbdroid]

Thanks for checking. I'll double down on repackaging the hand edited kernel.

On May 12, 2017 5:51 PM, "gtxent" notifications@github.com wrote:

6.0 stock does not enumerate the UVC device. :(

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lbdroid/JoyingIntelStuff/issues/2#issuecomment-301194348, or mute the thread https://github.com/notifications/unsubscribe-auth/ADN0xlmSwuW5KHHGL4po_114mNqJbJDkks5r5NRzgaJpZM4NZWjI .

[gtxaspec]

Sure, I'm stuck at work for the next few days, but I have the vehicle with me, as soon as you got it I will flash it.

[lbdroid]

You know, another option we may have is to build a 6.0 boot image with the 5.1 kernel...

On May 12, 2017 6:02:05 PM EDT, gtxent notifications@github.com wrote:

Sure, I'm stuck at work for the next few days, but I have the vehicle with me, as soon as you got it I will flash it.

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/lbdroid/JoyingIntelStuff/issues/2#issuecomment-301195965

[gtxaspec]

Yeah, thats possible, if it will work with Android 6.0 . Otherwise, we can compile one, as long as we can repack, and TRY it...

[gtxaspec]

OK, the mkbootimg recovery.img WORKS flashed to unit. I will try to modify the cpio now and reflash.

[gtxaspec]

new firmware posted tonight, the crruptoff file has some new variables:

fixwifi don fapklogon

i believe don is "debug on"

[gtxaspec]

I was able to extract fwu_image.bin which is included inside of 5009_60.zip using: https://github.com/InvisiBot/fwu-shredder

this file contains all the partition .bin INCLUDING mobilvisior files used by intel flash tools!

-rw-rw-r-- 1 aag aag 14320936 May 12 23:21 boot.bin -rw-rw-r-- 1 aag aag 1019384 May 12 23:21 mobilevisor.bin -rw-rw-r-- 1 aag aag 42902 May 12 23:21 mvconfig_smp.bin -rw-rw-r-- 1 aag aag 114492 May 12 23:21 psi_flash.bin -rw-rw-r-- 1 aag aag 11937064 May 12 23:21 recovery.bin -rw-rw-r-- 1 aag aag 603144 May 12 23:21 secvm.bin -rw-rw-r-- 1 aag aag 1020824 May 12 23:21 slb.bin -rw-rw-r-- 1 aag aag 2666496 May 12 23:21 splash_img.bin -rw-rw-r-- 1 aag aag 7168 May 12 23:21 ucode_patch.bin -rw-rw-r-- 1 aag aag 131072 May 12 23:21 vrl.bin

it appears that this file is is written to partition id 122 ( as noted in updater-script )

package_extract_file("fwu_image.bin", "/dev/block/platform/soc0/e0000000.noc/by-name/ImcPartID122"); intel_install_firmware_update("fwu_image.bin", "/dev/block/platform/soc0/e0000000.noc/by-name/ImcPartID122");

THIS is how they update the recovery, which is what I was wondering how they did during 5.1.1 to 6.0.1 update.

Note that the recovery.bin included here DOES NOT include the device tree, the size of the recovery.bin is only 11.9M. (what mkbootimg spits out when you dont add the extra_blob!)

It appears that when this file is dumped into this partition, the mobilvisor will flash all the files contained!

stay tuned.

[lbdroid]

I'm not sure we need to worry too much about writing the fwu_image to the device. From the shredder page readme, it looks like those can just be fastboot (or dd'd) to the proper locations on the device for the same effect.

Maybe what we can do instead of trying to RE the intelinstall... command, we just incorporate fwu_shredder into recovery.

[gtxaspec]

This: https://downloadcenter.intel.com/download/26840

has some very useful information about generating images and signing them, including dev keys. I'm still looking into everything.

[lbdroid]

You know you can just copy the keys from any aosp device?

On Jun 5, 2017 2:37 AM, "gtxent" notifications@github.com wrote:

This: https://downloadcenter.intel.com/download/26840

has some very useful information about generating images and signing them, including dev keys. I'm still looking into everything.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/lbdroid/JoyingIntelStuff/issues/2#issuecomment-306114774, or mute the thread https://github.com/notifications/unsubscribe-auth/ADN0xu08JkJAejRmwm2lZvKOOpoETrrUks5sA6I5gaJpZM4NZWjI .

[lbdroid]

Read this; https://source.android.com/devices/tech/ota/sign_builds

Basically, what you can do, is just COPY the file /res/keys from any build of AOSP into your customized recovery. Then you just need to sign the update.zip file using the same test keys. There is an easy to follow process for that here;

http://wiki.rootzwiki.com/Signing

[superdavex]

This process seems to work pretty good. Adding the extra blob in and changing the keys in the ram disk I am able to get the temporary twrp update launching from recovery.

[lbdroid]

Excellent. When you say "temporary twrp update".... you don't mean that you have a twrp recovery booting on this thing, do you?

Edit: I think you mean an update.zip containing the twrp recovery command, which is executed by the key-updated factory recovery.

[superdavex]

Right its the update.zip from 4pda signed with the public keys that launches a TWRP binary. It doesn't replace the stock recovery. I don't see why we couldn't, but, they did mention there was some random issue flashing from the permanent install. I played around with the mounts and terminal. It was late so I didn't get to do much more. We should be able to get WiFi or USB network adb going. The files I used are on my repo. https://github.com/superdavex/joyingsofia

[lbdroid]

Who said there were "issues" flashing?

I think its important to get a full replacement recovery written to the unit. The factory recovery does too many random things that are a "very bad idea". For instance, automatically run a partial userdata wipe if it fails to boot up fast enough, which is usually caused by the first boot process just taking way too long to complete.... so it automatically starts over again (which will still fail to complete) instead of continuing where it left off.

[superdavex]

I'm not sure it's relevant. It is a different model and the translation is tough to decipher. On 4pda they had a permanent TWRP install and a temp ramdisk version. The ram disk version is the one i'm using. The permanent one was removed because someone had an issue with it. Reading through the thread he may have just done an update that flashed his boot and recovery partitions with a unsupported version. I'm not too concerned with it. Don't do any factory updates and we should be fine. I'm also working on getting slave mode going, i'm sure its possible. I'd like to have fastboot or flashtool support.

[lbdroid]

I would ignore their flash writing problems then. My only concern with a proper custom recovery is whether or not the MCU needs to be told not to reboot the thing after a few minutes. Its possible that a recovery mode boot already provides the proper signal, and its also possible that the signal is implemented somewhere in their recovery.

Unless you can figure out a way to hook up the USB that bypasses the HUB, I don't think we will have any luck with fastboot. flashtool appears to be a sony specific customized fastboot?

In any case, fastboot more or less can be replaced with the "dd" command, we just need to have some kind of remote connection. I was thinking just a non-encrypted local-link wifi hotspot would do the trick adequately.

If your recovery is for a different device, be very careful writing anything with it until you've verified that the fstab you're using is correct for our device.

[superdavex]

The chip does support host/slave. I'm not sure how they have it wired up though. I'm gonna try to get some time this weekend to pull the radio and look at the board. Some units they have had to solder another port on. I was referring to the Intel Flash tool. I've not used it, but, it seems these units support it. Wifi or a USB network adapter. The stock recovery/boot kernels have support for several USB network adapters. I tried 3 and they all worked. In recovery dhcp wasn't running so I couldn't get an IP. That wont be an issue now.

Yeah, I checked the partitions and they seem to match. It was late last night when I got this going so i'll double check it.

[lbdroid]

Pretty much all SoC's support dual mode USB, but given that the USB sockets are wired up to a hub chip, it may not be possible to access it even when its in slave mode. The way most single-USB SBCs are wired is that they have a switch chip somewhere that will connect the USB pins on the SoC to either the hub, OR a slave socket. Now its possible that they do have a switch. I actually have more than one of these units, a couple of UL's and a UM. The UM has a 4-pin plug next to the two USB plugs on the mainboard. If they have a switch, it would be my expectation that it would probably hook up there.

BUT, I don't really think its worth worrying too much about it. The best thing would be to get the on-board wifi running in recovery, then we avoid all hardware concerns.

[lbdroid]

Damn. The only reference I can find about a wifi enabled twrp is here; https://www.theandroidsoul.com/wifi-enabled-twrp-nexus-6/

Which references to xda, and xda being xda, they've banned the user and deleted all his posts.

[superdavex]

I had a bootloop issue yesterday when flashing a full recovery image it didn't like. I was just doing a quick test and thought it would hang, but, wanted to see what it would do. My boot image was stock. It did not go into recovery and I was able to boot the stock image. After a second reboot though it was stuck in a boot loop. I've had recovery hang a few times when trying to launch TWRP and I have to kill ACC and wait for the MCU to fully power off. This usually corrects it. It did not with a bad recovery though. I'm guessing the MCU had it flagged that it needed to go into recovery to do a reset because of a failed boot. Even though I only had one failed boot, not three. I tried every key combination I could, pulled the battery/acc and let it sit all day and it still would not boot. So I pulled the unit and figured I would try to see if I could get a device usb port going. Surprisingly my unit is a 5009, I had thought it was a 6021. I soldered on a usb cable, but, wasn't quite sure of the pinout. I wasn't able to get the device recognized but something in the attempts caused the MCU to load the boot image. I then corrected the recovery with the key modified image and it worked fine. I think i'll start testing the recovery images on the boot partition now.

[lbdroid]

Yeah, I'd agree that testing with the boot partition would be a much safer bet than on the recovery. We know that we can force it into recovery with an update on a usb disk and a paper clip, but once the cache partition is flagged to boot recovery, we can't force it back to normal boot. And for that matter, once your test-keys recovery is installed, the sky is the limit as far as using that as a channel to fix things with, but it would be nice to have a kernel console and access to fastboot (which it actually does have, if only we could hook up to it and figure out a reliable way to launch it).

My guess about what got you back to a normal boot was probably something along the lines of X failed recovery boots.

One thing to note though; the MCU doesn't flag anything. It has no control over the boot process beyond the obvious triggering of reboots (which I suspect is just hitting the reset pin) -- the boot loader / hypervisor has all the control over the boot process.

And actually, we could probably take away the MCU's reboot control as easy as cutting whatever pin is hooked up to the SoM's reset line.

So what were you doing that ended up in a bad recovery image?

[superdavex]

Weird, I replied yesterday but its not here. Yeah, I had meant to say bootloader before. I've been going through that process trying to figure out where it is storing that count. If you haven't already I'd also like to figure out the command to reset or shutdown batt power. It could have been the reboot counts, but, I sure did try a ton before pulling it. After it was on the bench I was able to fully cycle power though. Really not sure, but, I don't plan on testing that again. :)

I was working on replacing the recovery fully, testing ramdisk repacks.

I was able last night to get adb USB network connection using a crossover cable to my laptop. Makes it a lot easier for debugging. Also interestingly the connection is there even on the times that it freezes on the android and does not open TWRP. Should make it easier to figure out why that is.

[lbdroid]

You need to keep in mind that we aren't dealing with a run of the mill bootloader like lk or the likes. This is a crazy intel beast of a hypervisor. It has its own crap to determine how to pick a boot source, and likely stores counts in both RAM and NVM. On top of that, we have the "made in china" counters and other "really bad ideas" If I were looking for where they're storing a counter that affects boot up, first place I would look is in the cache partition.

Not sure I understand what you mean by "the command to reset or shutdown batt power". Could you clarify?

[superdavex]

Yeah, i've been looking at cache. They are definitely using it for boot logging. Should be easier to debug now that I have the ADB connection. I'd like to try to get that earlier in the boot process too. I can certainly do it on the system side, but, it would be nice in recovery too. I still want to get a USB adb going. I've ordered another unit so I can have one to work/play with on the bench.

When i'm in TWRP and need to reset I have to go through a crazy ACC off, wait a minute for the MCU to detect that it did'nt shutdown then it full cycles process. If I don't it gets hung in a boot loop until I do. The reset pin doesn't really full cycle it either. I could make a hardware switch that kills the BATT power, but, it would be nice to command the MCU to do it. I'm guessing that it is the MCU that is doing that, I hear the relays clicking so it seems likely. It could also be a flag not cleared and the hypervisor is doing it I guess. I'll do some more testing.

[lbdroid]

The hypervisor definitely reads /cache/recovery/command. I think if you clear that and run a regular reboot, that should do the trick.

[superdavex]

The original TWRP script was clearing /cache/recovery so its not there. I checked NVM and didn't see it there either. It's not really an issue any more, I documented the recovery entry/exit process on my repo. I can reliably enter and exit recovery from any state. I'm still having issues with TWRP hanging on some starts, I have network adb every time though. Getting to the point where i'm ready to starting working on boot. Recovery is pretty rock solid and I've got the minimum files needed to get into it documented so assuming the hypervisor doesn't crap out once boot is modified we should be good.

[gtxaspec]

I ran into a no-boot situation maybe a dozen times when initially attempting to modify the recovery partition, albeit unsuccessfully due to limited time available. The recovery partition was imaged, and the unit would unsuccessfully try to reboot into that partition, so the unit would get stuck in a black screen boot loop due to the unit presumably attempting to boot from the bad recovery partition every time ( hypervisor ).

The only solution to this was to pull all power and let the unit sit overnight ( at least 6 hours ). Unit would come up right away after and boot from the system partition, and I was able to image stock recovery.

Probably NVM or something in the SoM storing a bit to keep trying to boot from recovery, just a tidbit if someone gets stuck during a bad recovery image.

Also, the SoM is using an FE1.1s usb hub IC, on the underside of the SoM http://tinyurl.com/yafvjah9 . Would be nice to have USB based ADB...but I'll take ethernet as well as my setup has one already.

[superdavex]

That's pretty much what happened to mine, though I had off BATT/ACC pulled for over 8 hours. Once I get a good process going on boot I'll take another shot at the recovery ramdisk. I saw that IC there, I was trying to get what I thought was the front serial port soldered on and it was easier to access the pins on the bottom. Unfortunately mine is a 5009 and doesn't have the header already there. I've ordered a 6021 though so i'll give it a shot with that one too.

[superdavex]

The boot mode/update file is stored in mmcblk0p17

address 4000 contains boot-recovery address 4040 contains recovery.--update_package=/mnt/usb_storage/5009_60.zip.--locale=zh_CN.

[lbdroid]

FYI: I've got sleep/wake working properly in MCUController now :)

Which means that now its time to go native.

[superdavex]

Nice, that's awesome! I found the issue with TWRP not loading every time and got that fixed. I also was able to modify boot and change init to ifconfig eth0. That will make it easier for testing system images. The hypervisor doesn't care if you modify boot and if it fails it just launches recovery.

[superdavex]

Ok, so that's not completely true. I just noticed after a restart without flashing a new image it doe not boot the new image. It doesn't even try so I don't think there is anything wrong with it. It does try 3 times give you the error and then boot fine though. It doesn't even reset so not a big deal, but, something to look into. It maybe the signature at the end of the boot image being checked.

[lbdroid]

So how long until you get an installable TWRP built? :)

[superdavex]

I was focused on getting xposed going so I could finally trigger Alexa from remote. Now thats done I'll go back to getting that. Should be pretty quick. I'm gonna setup a new build instead of tweaking the existing one.

[lbdroid]

Hey have you checked if that "extra blob" is actually even needed? I have to admit that I haven't had a heck of a lot of sleep over the last... 4 years (since my first kid was born), so now I'm thinking that I may have done something really really stupid, like starting off with a dd image of a recovery partition, which is longer than the actual recovery, and thus may have some garbage at the end.

[superdavex]

I've been there, it gets a little better when they're older. :)

It's not, I haven't been using it for my boot images. I have been adding the signature, but, I haven't tested whether that is need either. I think that is why it fails to boot when the hypervisor checks it though. It boots after 3 times fine. I'll try without the sig later.

[lbdroid]

Hey Dave, are you still around anywhere? What signature are you referring to that you've been adding?