No crypto support - Githubissues

flopraden commented 9 years ago

Hi,

I try to mount my /data partition. I use the adb shell method because no auto-detect of crypto partition was made (normal in fact)

So I use ~ # twrp decrypt XXXXXXXXXX Attempting to decrypt data partition via command line. E:No crypto support was compiled into this build.

Is it possible to add the crypto module to unlock the /data partition?

Best

fedeaf commented 9 years ago

Same issue here. @Dees-Troy, Can you help with this?

Dees-Troy commented 9 years ago

I don't own a two. Even if I compile in crypto support, the device will still fail to decrypt unless the proper proprietary libraries and firmware are identified and placed into the proper locations in the recovery ramdisk. Someone who owns a two will have to work on it and figure it out. It's way too time consuming for me to work on it without having the device.

angrox commented 8 years ago

I gladly would provide the files but I am lacking the knowledge of the encryption process in Android. Is there some sort of documentation how to extract the needed libs/files?

Dees-Troy commented 8 years ago

Maybe I haven't made myself clear. I am not going to work on crypto support for this device myself unless someone provides a device physically in my hands to work on it. I don't want / need an invite. I am not going to use my money to buy a Two.

I don't need someone to pull the files for me. It's way too time consuming to work on crypto support remotely. The source code is all there, so if you aren't willing to provide a device, then you are more than welcome to work on it yourself.

We aren't a big business. We aren't even a company. TWRP consists primarily of 3 guys writing code in their spare time because we like doing it. It's not our full time job. We give our software away free of charge, so we're not swimming in money gleaned from our efforts to buy even relatively inexpensive devices. OnePlus doesn't provide us with devices. We happened to have a OnePlus One so we were able to fairly easily work on that device, but we don't own a 2. If you are a developer who owns a 2 and are reasonably competent, then I'll be happy to assist by answering questions in our IRC channel. Otherwise, what I've said at the following link pretty much all applies here too: https://plus.google.com/u/1/+DeesTroy/posts/iyyM1kYULEa

flopraden commented 8 years ago

Hi, I just report this to be sure that the only reason is the lack of the device and not a compile option to enable. I got a 2, so I'll try to see what I can do. I'm aware of the all point : free time, expensive device, ... and just for no misunderstanding i was not requesting to do it.

I'll give a try by following your post. Best.

acran commented 8 years ago

I just report this to be sure that the only reason is the lack of the device and not a compile option to enable.

Unfortunately this is not the case: I built twrp with TW_INCLUDE_CRYPTO := true and now when booting that image twrp correctly detects the encrypted /data and shows the input for the unlock pattern. But entering it as well as manually running twrp decrypt fails. When symlinking /vendor according to the comment on 280f9caed59085772063b44d783ce32ec8de4780 it takes significantly longer (around 20-30s) but still fails:

~ # mount /system
~ # ln -s /system/vendor /vendor
~ # twrp decrypt $password
Attempting to decrypt data partition via command line.
E:Failed to decrypt data.

(I tried this with TARGET_HW_DISK_ENCRYPTION being set to true as well as false)

I'd really like to help with porting encryption support to this device. But now I'd need some hints on how to further debug what is going on/wrong and what to change to make it work. So could you lead me in the right direction please :)

Dees-Troy commented 8 years ago

There's not necessarily a "right direction" that anyone can lead you. On some recent devices, you have to hvae qseecomd running in recovery. On some devices you have to mount another partition like the modem or firmware partition. It's easier when you have full source for the ROM (Nexus or Cyanogen Inc devices) because you can compile an engineering boot with full adb root and grab logcats and run test binaries during a normal boot and use a process of elimination to help identify what needs to be done.

acran commented 8 years ago

There's not necessarily a "right direction" that anyone can lead you.

Well, where you led me was right enough, spoiler: I managed to access encrypted /data :Þ

On some recent devices, you have to hvae qseecomd running in recovery. On some devices you have to mount another partition like the modem or firmware partition.

And on this device it was both of this.

Having built twrp with TW_INCLUDE_CRYPTO and TARGET_HW_DISK_ENCRYPTION I can boot that image and after executing the following (using adb) the decryption of /data succeeds via pattern input or via cli:

mount /system/ -o ro
ln -s /system/vendor /vendor
mkdir /firmware
mount /dev/block/bootdevice/by-name/modem /firmware -o ro,shortname=lower,uid=1000,gid=1000,dmask=227,fmask=337,context=u:object_r:firmware_file:s0
/system/bin/qseecomd

So now is the question how to correctly include this into the build:

(how to find out) which files to copy from the device and include in the recovery.img
what files do I have to edit (sorry, I'm quite a beginner with this...)? Do I just have to copy the commit referenced in your comment? What about qseecomd?
have qseecomd, firmware files and libs to be copied from the device into the image or can they be built from source (libs)/used directly from the device (from /firmware, /system)?

Thank you for your help (=

acran commented 8 years ago

Now I managed to translate the shell commands to an init.rc file and include that in the build, see 4a6db2f58ce152a7777b8a2d93911c239dcdd9ed.

But I'm not yet totally happy with that solution since it depends on an intact /system and /firmware and mounting /system in the init.rc file that way more or less breaks the possibility of mounting/unmounting /system in the UI because once started the open files held on that partition prevent unmounting it again.

I already analyzed which libraries and files are needed by qseecomd to successfully decrypt and came to this list:

/firmware/image/cmnlib.b00
/firmware/image/cmnlib.b01
/firmware/image/cmnlib.b02
/firmware/image/cmnlib.b03
/firmware/image/cmnlib.mdt
/firmware/image/keymaste.b00
/firmware/image/keymaste.b01
/firmware/image/keymaste.b02
/firmware/image/keymaste.b03
/firmware/image/keymaste.mdt
/system/bin/linker64
/system/bin/qseecomd
/system/lib64/hw/keystore.qcom.so
/system/lib64/libbacktrace.so
/system/lib64/libbinder.so
/system/lib64/libcnefeatureconfig.so
/system/lib64/libc.so
/system/lib64/libcutils.so
/system/lib64/libEGL.so
/system/lib64/libgccdemangle.so
/system/lib64/libGLES_trace.so
/system/lib64/libGLESv2.so
/system/lib64/libgui.so
/system/lib64/libhardware.so
/system/lib64/liblog.so
/system/lib64/libm.so
/system/lib64/libnetd_client.so
/system/lib64/libsigchain.so
/system/lib64/libstdc++.so
/system/lib64/libstlport.so
/system/lib64/libsync.so
/system/lib64/libui.so
/system/lib64/libunwind-ptrace.so
/system/lib64/libunwind.so
/system/lib64/libutils.so
/system/vendor/lib64/libdiag.so
/system/vendor/lib64/libdrmfs.so
/system/vendor/lib64/libdrmtime.so
/system/vendor/lib64/libNimsWrap.so
/system/vendor/lib64/libQSEEComAPI.so
/system/vendor/lib64/librpmb.so
/system/vendor/lib64/lib-sec-disp.so
/system/vendor/lib64/libsecureui.so
/system/vendor/lib64/libsecureui_svcsock.so
/system/vendor/lib64/libssd.so
/system/vendor/lib64/libStDrvInt.so
/system/vendor/lib64/libtime_genoff.so
/system/vendor/lib64/libvendorconn.so

Several of this libs also are already included in the recovery. So I (manually) build a recovery image with the missing libs placed under /vendor/lib64, the firmware files under /firmware, qseecomd under /sbin and symlinking /sbin/linker64 as /system/bin/linker64 (since this path is hard coded into qseecomd). Then only qseecomd of the image needs to be started from init.rc and everything works even without mounting any additional partitions.

I tried to properly include this to the build by adding the files to PRODUCT_COPY_FILES in device.mk, just like in the mentioned example code for the hammerhead but unfortunately none of the files in PRODUCT_COPY_FILES get copied when building with make recoveryimage.

What am I missing here/what would be the appropriate way to include these additional files into the image when building? Anyone able to help me with this?? Sorry, I'm not really familiar with the whole android building system )=

jfeise commented 8 years ago

@acran: You don't have the firmware files at build time (unless you pull them from the actual device), so you have to mount /firmware in the recovery. For the other files, the lines should look like this: PRODUCT_COPY_FILES += vendor/oneplus/oneplus2/proprietary/bin/qseecomd:recovery/root/sbin/qseecomd \ ...

BTW, I have been able to compile TWRP for the OP2 with the CM13 sources. It shows the password dialog, but doesn't decrypt... I'm going on vacation this week, so it will be a while before I can look at this again.

acran commented 8 years ago

You don't have the firmware files at build time (unless you pull them from the actual device)

I do have, since this is exactly what I did: I modified proprietary-files.txt to include all required files, so after executing extract-files.sh they were placed in android/vendor/oneplus/oneplus2 (even the /firmware files).

But for all I can say, the step where the PRODUCT_COPY_FILES actually get copied just isn't executed when running make recoveryimage in the android root since also none of the files which were already listed in device.mk before my modifications gets copied, although I double-checked that the paths are correct and in the right format like you wrote...

I wish you a nice vacation.

jfeise commented 8 years ago

FYI, I got decryption working with this in my BoardConfig.mk:

ifneq ($(TARGET_RECOVERY_ROOT_OUT),)
$(shell mkdir -p $(TARGET_RECOVERY_ROOT_OUT)/system/bin)
$(shell mkdir -p $(TARGET_RECOVERY_ROOT_OUT)/vendor/lib64/hw)
endif

PRODUCT_COPY_FILES += \
        device/oneplus/oneplus2/recovery/bin/linker64:recovery/root/system/bin/linker64 \
        vendor/oneplus/oneplus2/proprietary/bin/qseecomd:recovery/root/sbin/qseecomd \
        device/oneplus/oneplus2/recovery/lib64/keystore.msm8994.so:recovery/root/vendor/lib64/hw/keystore.msm8994.so \
        device/oneplus/oneplus2/recovery/lib64/libbinder.so:recovery/root/vendor/lib64/libbinder.so \
        device/oneplus/oneplus2/recovery/lib64/libcnefeatureconfig.so:recovery/root/vendor/lib64/libcnefeatureconfig.so \
        device/oneplus/oneplus2/recovery/lib64/libEGL.so:recovery/root/vendor/lib64/libEGL.so \
        device/oneplus/oneplus2/recovery/lib64/libGLES_trace.so:recovery/root/vendor/lib64/libGLES_trace.so \
        device/oneplus/oneplus2/recovery/lib64/libGLESv2.so:recovery/root/vendor/lib64/libGLESv2.so \
        device/oneplus/oneplus2/recovery/lib64/libgui.so:recovery/root/vendor/lib64/libgui.so \
        device/oneplus/oneplus2/recovery/lib64/libhardware.so:recovery/root/vendor/lib64/libhardware.so \
        device/oneplus/oneplus2/recovery/lib64/libsigchain.so:recovery/root/vendor/lib64/libsigchain.so \
        device/oneplus/oneplus2/recovery/lib64/libsync.so:recovery/root/vendor/lib64/libsync.so \
        device/oneplus/oneplus2/recovery/lib64/libui.so:recovery/root/vendor/lib64/libui.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/libdiag.so:recovery/root/vendor/lib64/libdiag.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/libdrmfs.so:recovery/root/vendor/lib64/libdrmfs.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/libdrmtime.so:recovery/root/vendor/lib64/libdrmtime.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/libNimsWrap.so:recovery/root/vendor/lib64/libNimsWrap.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/libQSEEComAPI.so:recovery/root/vendor/lib64/libQSEEComAPI.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/librpmb.so:recovery/root/vendor/lib64/librpmb.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/lib-sec-disp.so:recovery/root/vendor/lib64/lib-sec-disp.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/libsecureui.so:recovery/root/vendor/lib64/libsecureui.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/libsecureui_svcsock.so:recovery/root/vendor/lib64/libsecureui_svcsock.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/libssd.so:recovery/root/vendor/lib64/libssd.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/libtime_genoff.so:recovery/root/vendor/lib64/libtime_genoff.so \
        vendor/oneplus/oneplus2/proprietary/vendor/lib64/libvendorconn.so:recovery/root/vendor/lib64/libvendorconn.so

This is building with CM13. I think the PRODUCT_COPY_FILES may not work if the directories aren't there, that's why I create them first, in out/target/product/oneplus2/recovery/root. The ifneq stuff is there because the mk is also called when doing make clean, and I got errors /vendor: permission denied. I am mounting the /firmware directory, but not system. I am just creating a /system/bin directory where I put the linker64. I copied the files in device/oneplus/oneplus2/recovery/lib64/ from the device. That's based on acran's list of files needed. I think it can be optimized a bit more. I had to patch some of the TWRP files for CM, mostly missing includes and declarations. I'll submit a patch later.

Now on to my vacation ;)

acran commented 8 years ago

FYI, I got decryption working with this in my BoardConfig.mk:

Indeed, the PRODUCT_COPY_FILES defined in BoardConfig.mk gets used by make recoveryimage and copies the additional files into the image. So I moved my PRODUCT_COPY_FILES from device.mk to BoardConfig.mk and it worked (executing the extract-files.sh script beforehand to copy the proprietary files from the device). I cleaned it up and put it into this pull request: #2

jfeise commented 8 years ago

Cool. I think Dees_Troy mentioned in some other thread that CM is deprecating PRODUCT_COPY_FILES, instead having all the needed files in a specific directory. I'm at SCaLE right now, will search my email when I'm back home. Update: This is the comment I saw: https://github.com/TeamWin/Team-Win-Recovery-Project/issues/454#issuecomment-172903119

jfeise commented 8 years ago

Decryption for the OP2 is working in the new twrp 3.0. Thanks everybody for making it work!

randomhydrosol commented 8 years ago

It is

randomhydrosol commented 8 years ago

Please upload screenshots

ghost commented 8 years ago

I can confirm decrypting by TWRP 3 is working, i encrypted /data with OxySlim (CustomRom based on Stock Oxygen 2.2.0) and TWRP successfully decrypt /data

Xalaxis commented 8 years ago

Has anyone got crypto working in Marshmallow? Is this a new issue? I would've short the method should be the same.

jfeise commented 8 years ago

Sure it is working. I have /data encrypted, and TWRP can decrypt it just fiine.

Xalaxis commented 8 years ago

Which ROM are you using? It may be specific to my BlissMallow one, in which case I will switch.

EDIT: /data is encrypted fine however I can't seem to get TWRP to decrypt it

jfeise commented 8 years ago

Encryption should be independent of ROM, unless BlissMallow screwed up big-time. I have used builds from regalstreak and grarak. They all work. TWRP decrypts /data. Grarak posts his recent builds on Twitter (@grarak). Is this a fresh CM13 encryption, or did you dirty-flash from CM 12.1 with the encryption from 12.1?

Xalaxis commented 8 years ago

This is all fresh CM13 encryption. I'll give Grarak's build a shot.

Xalaxis commented 8 years ago

Here's a thought, could it be due to having the updated MM modem flashed? Would I need that for all MM builds or was it only supposed to be used for specific ones?

Xalaxis commented 8 years ago

My current progress: Tried decryption in latest TWRP for OxySlim (Essentially stock Oxygen) WORKED! First decryption I've had work so far.

Xalaxis commented 8 years ago

Tested Grarak's latest build with latest TWRP. Decryption on TWRP fails, even with no password. Any ideas @jfeise ?

jfeise commented 8 years ago

Are you using SuperSU or the built-in su? On the CM forum for the OPO, somebody had problems with SuperSU.

Xalaxis commented 8 years ago

I never use anything other than the default SU that comes with the ROM.

On 29 February 2016 16:02:25 GMT+00:00, Joe Feise notifications@github.com wrote:

Are you using SuperSU or the built-in su? On the CM forum for the OPO, somebody had problems with SuperSU.

Reply to this email directly or view it on GitHub: https://github.com/TeamWin/android_device_oneplus_oneplus2/issues/1#issuecomment-190269253

Sent from my Android device with K-9 Mail. Please excuse my brevity.

jfeise commented 8 years ago

I just read on the CM OPO forum that CM fixed a bug in the encryption on 2/22: http://forum.cyanogenmod.org/topic/121004-cm13full-device-encryption-is-broken/#entry578857 Not sure if this also applies to the OP2, I haven't had time to locate the actual change yet, but it could explain the problems people have been having recently, if that fix broke things for TWRP.

briancross commented 8 years ago

I'm unable to decrypt /data with 3.0.0-2 and the latest CM13 by seraph08. TWRP asks for a password, even though I have a PIN set, and entering the PIN as text fails.

I can access /system, so it's possible to flash if I can get recovery to see the file (sideload, USB-OTG, /cache, etc), but no dice on /data.

I encrypted within CM13 and had never used encryption before.

jfeise commented 8 years ago

Encryption and decryption was broken in CM13 for the OP2 for a while (since 3/2, I think.) The keystore library wasn't in the build. I submitted a fix for that, which was merged yesterday. TWRP only works with the hardware encryption, which wasn't working in CM. So, you probably have to reformat /data, and restore a backup.

briancross commented 8 years ago

Excuse my inexperience, but I'm trying to figure out if it got merged into seraph08's builds: https://github.com/Seraph08/device_oneplus_oneplus2/commits/cm-13.0 I don't see anything about it there (nothing since 3/18), but would your fix even show up there?

I'm running the latest build, but it's still not working. What would be the best way to go about taking and restoring a backup? I can't do so from TWRP.

jfeise commented 8 years ago

The commit that broke it is in there, https://github.com/Seraph08/device_oneplus_oneplus2/commit/4adacc43214b4a9c8c3cfb9d1f895e4d3b524445, the fix isn't yet. The fix was merged in the CM tree yesterday, it may take a while until other ROMs pick it up. If you can't get to the encrypted partition, and don't have an off-device backup, you may have to reinstall your apps. Been there, done that myself...

briancross commented 8 years ago

Ok cool, I just found the commit on the cm repo. I'll ask seraph if it made it in and try.

If I gotta wipe clean, I gotta wipe clean. Done it a thousand times by now :)

Xalaxis commented 8 years ago

Is this the fix commit here @jfeise ?

Xalaxis commented 8 years ago

I must admit I am being a little stumped by the progression of code upstream to cyanogenmod base. A visual diagram would be handy :P

jfeise commented 8 years ago

Yup, that's the fix. Grarak merged it in his repository. In CM's gerrit, it is http://review.cyanogenmod.org/#/c/137362/

fmckeogh commented 8 years ago

If I understand correctly, compiling the latest TWRP source and Grarak's repo will allow me to encrypt and decrypt?

briancross commented 8 years ago

I flashed seraph's 3/23 build today, which I am reasonably certain he is building from CM sources, which would indicate that your patch is included. When I tried to mount /data in TWRP, I was prompted for a password (with a keyboard, not a PIN screen). Entering the PIN as text put TWRP in to a much longer attempt at decrypting than usual, but it ultimately failed. Thinking something had changed on the ROM side causing the longer decrypt attempt, I as suggested, reformatted, reinstalled, and re-encrypted - but to no avail.

If I have a PIN set, is TWRP supposed to provide a PIN pad? Or is it entered using the QWERTY keyboard?

jfeise commented 8 years ago

@briancross Don't assume things. People who build ROMs may not use everything from CM, at least not immediately, but cherry-pick. Seraph has his own device_oneplus_oneplus2 fork, and that fork hasn't been updated. Right now, it says "This branch is 9 commits ahead, 1 commit behind CyanogenMod:cm-13.0. " The "1 commit behind" is my fix... I use TWRP 3.0.0-2 from https://twrp.me/devices/oneplustwo.html, and build CM-13 from the CM sources. That works. For other ROMs, you always have to check with the people who manage these ROMs. @chocol4te Grarak has updated his device_oneplus_oneplus2 fork (not surprising, since he also was the one who merged my fix into CM), so using that should work.

briancross commented 8 years ago

Thanks...I'm beginning to figure all this out. I've unencrypted for now but I'll wait until I see he's merged that before I try again.

When it is working though, will twrp give me a pin entry screen? Or is it done with the numbers on the qwerty keyboard?

jfeise commented 8 years ago

I am using a password, but if I remember right, if a pin is selected, you get a numeric keyboard.

ihor-sviziev commented 8 years ago

@jfeise I'm using latest Graraks rom (upgraded from build 2016-03-08), however decryption isn't work for me. Now I want to create backup and try OSS 3.0, but I can't do it without data decryption feature((

jfeise commented 8 years ago

Sorry, works for me. You may have more luck on their IRC channel, #twrc

jfeise commented 8 years ago

Note that the latest TWRP version, 3.0.1-0, has broken decryption for what seems to be all phones. I've tried it on Nexus 4, OPO, and OP2. It didn't work for any of them.

ihor-sviziev commented 8 years ago

@jfeise ok, waiting for the fix.

ihor-sviziev commented 8 years ago

Tried 3.0.2, but still can't decrypt data partition =( PS: Currently I'm using Grarak's ROM

jfeise commented 8 years ago

3..0.2 decrypts fine for me, with a password. Which ROM it is generally doesn't matter, because the recovery contains it's own minimal Linux, it is not based on the full CM ROM (they have their own fork, and maintain their own device-specific files, like the proprietary encryption libraries.) In this case, it could potentially matter if /data was encrypted while that code was broken in the ROM (one of the proprietary encryption files was left out of Grarak's builds for a while.) But then you wouldn't be able to decrypt it when booting the fixed version of the ROM, either. So, when saying which ROM you use the actual date of the ROM used when encrypting and used now could be important. Since Grarak doesn't put out a changelist, it is hard to know which fixes are in his ROMs. Without that, it basically is a shot in the dark, and nobody can really debug things.

Xalaxis commented 8 years ago

Got it. Of note is that encryption in OOS 3.0.0 Beta does not work at all and this is a bug in the ROM itself, not TWRP or anything inbetween. Please correct me if I'm wrong on that front anyone.

bkix commented 8 years ago

I installed Garaks latest build 2016-04-09 and Seraph08s build 2016-04-26. Both flashed clean on a fully wiped (/data formated) phone. Afterwards encrypted the device and tried to decrypt with TWRP 3.0.0-0 to 3.0.2-0, without success. After entering the correct 5-digit-PIN it says: "Password failed, please try again!"

How can I help to track down this issue?

TeamWin / android_device_oneplus_oneplus2

No crypto support #1