Closed shirshak55 closed 2 years ago
Reproducing the issue:
I installed CA certs but that goes to user CA which means it will nag me forever. And app shows me System Trust Disabled because CA is not in the proper directory. I am perplexed by this issue.
I am confused because HTTP Toolkit Ui clearly says " Automatically injects system HTTPS certs in rooted phone" but it doesn't seem to reflect it.
Today I tried with USB debugging with wires too. But it is failing too. I wonder what happened.
~/Downloads ➜ adb root
adbd is already running as root
~/Downloads ➜ adb shell raphaelin:/ #
Something is weird.
Some screenshots:
Andriod version 11.
It seems the shellscript used in httptoolkit is not working as expected. From log: Adding cert file as /data/local/tmp/91511479.0
Which means 91511479.0 should be moved into /system/etc/security/cacerts
I remounted and add certs myself
hash=$(openssl x509 -inform PEM -subject_hash_old -in certificate | head -1)
mv certificate "$hash.0"
And copied that to android cacerts and it seems to work now.
Hi @shirshak55. Sorry to hear you're having issues. All the output shown here is as expected though I think... HTTP Toolkit's Android root ADB setup does a few things:
That process starts here and you can see the individual methods it's calling like getRootCommand
and injectSystemCertificate
here.
I think it's normal that some of the root commands might time out. As long as one command works, that's fine. The commands that get tested are:
su root whoami
in an ADB shellsu -c whoami
in an ADB shelladb root
, then whoami
in an ADB shellThey count as succeeding if they print 'root'. Can you check which one of those works on your device?
Once one of those does succeed, the certificate is copied to the tmp path (which is why that path appears in the output above) and then a script running on the device remounts the certificate directory, copy the files to the correct place, and sets the required properties on them.
You said "It seems the shellscript used in httptoolkit is not working as expected" - can you explain exactly what it's doing wrong on your device? If 'Cert Injected' is shown in the output then the script and everything else should have run correctly.
I think shell script is failing to copy certs to android system certs folder. Root access is working properly. I think remounting system dir, removing tmp files and copying certs is probably having issue?
Actually its not a big issue to me because I copied the certs myself which httptoolkit andriod app was able to recognize but many other people might be having similar trouble. I think many people in hacker news were raising this issue that's why i tried to do postmortem
if it had copied certs it should have shown that cert file 91xxxxx.0 in android system cacert directory. But when I ls into that directory there was no such file.
It would be good to look into this, totally agree. It's very surprising that the script would succeed without actually copying the file though.
Can you please:
adb logcat -T1 > logcat.txt
And then post here:
logcat.txt
adb shell
and then:
ls -la /data/local/tmp/
ls -la /system/etc/security/cacerts
mount
That should make it a bit clearer what exactly is happening here.
Logcat is sent to your email (too large to paste here)
raphaelin:/ # ls -la /system/etc/security/cacerts/91511479.0
ls: /system/etc/security/cacerts/91511479.0: No such file or directory
~ ➜ httptoolkit
Gtk-Message: 20:25:31.909: Failed to load module "xapp-gtk3-module"
(node:8417) electron: The default of nativeWindowOpen is deprecated and will be changing from false to true in Electron 15. See https://github.com/electron/electron/issues/28511 for more information.
(Use `httptoolkit --trace-warnings ...` to show where the warning was created)
(node:8465) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:8465) [DEP0125] DeprecationWarning: The _stream_wrap module is deprecated.
Config checked in 7 ms
Certificates setup in 3 ms
Standalone server started in 2 ms
Server started in 10 ms
Total startup took 22 ms
Updating all Docker network aliases...
httptoolkit-server: Updating CLI... already on latest version: 1.5.1
Browser cache updated
httptoolkit-server: Updating CLI... already on latest version: 1.5.1
Error: Timeout for ADB command su,root,whoami
at /opt/HTTP Toolkit/resources/httptoolkit-server/bundle/index.js:487:418818
at runNextTicks (internal/process/task_queues.js:60:5)
at listOnTimeout (internal/timers.js:524:9)
at processTimers (internal/timers.js:498:7)
at async Promise.all (index 0)
at async e.getRootCommand (/opt/HTTP Toolkit/resources/httptoolkit-server/bundle/index.js:487:419678)
at async e.AndroidAdbInterceptor.injectSystemCertIfPossible (/opt/HTTP Toolkit/resources/httptoolkit-server/bundle/index.js:487:417241)
at async e.AndroidAdbInterceptor.activate (/opt/HTTP Toolkit/resources/httptoolkit-server/bundle/index.js:487:415063)
at async activateInterceptor (/opt/HTTP Toolkit/resources/httptoolkit-server/bundle/index.js:479:15278)
at async /opt/HTTP Toolkit/resources/httptoolkit-server/bundle/index.js:311:4252
Adding cert file as /data/local/tmp/91511479.0
Cert injected
httptoolkit-server: Updating CLI... already on latest version: 1.5.1
^CShutting down after API call...
Mount
HttpToolkit is telling me to install the client certs so its not working for me :D
Probably something in here is not working as intended and it works differently on your phone vs mine?
Probably something in here is not working as intended and it works differently on your phone vs mine?
Yes, definitely! That mount output looks particularly suspicious.
On my rooted test device, the initial output looks like:
This device is using LineageOS 17.1.
After setting up ADB interception, one extra line is added:
tmpfs on /system/etc/security/cacerts type tmpfs (rw,seclabel,relatime)
That's what I would expect to happen, but in your case the output has a lot of extra magisk lines, and the tmpfs cacerts mount never appears.
How exactly was this device set up? It would be great if you could share the OS version and any manual rooting steps or any other custom configuration that might be relevant.
I suspect the mount command is what's failing here, but it would be best to check the whole script. Can you try copying the cert file into /data/local/tmp/, running the script commands manually, and see what errors or other results you get? Exit codes (i.e. echo $?
) for any commands that do fail would be very helpful too.
It seems likely that this is failing on the mount step for some reason, but the fact that you can copy the certificate in manually anyway means we might be able to skip that in this case. If there's a way we can detect whether this is necessary then that'd work as a fix. Any clues you can get about exactly why this fails would be very useful!
@pimterry I am myself bewildered here. The file on tmp directory gets removed but it is still not copied to android system cacerts folders. Strange.
Btw have you tried testing on magisk rooted device? (Most of the rooted phone is going to use magisk because of systemless root anyway)
When you run the commands one by one manually, do you see any kind of errors?
@pimterry it works if I do it manually :(
Thanks, that's really great! Very clear on the issue, the behaviour there is very odd, although it doesn't immediately give me many ideas of what might be going wrong.
If you set up a local server, you should be able to edit the script directly in the server, and then test that directly. That should let you debug it up close to work out what it's really doing and why it's failing.
One thing I've noticed already is that we're not using or logging the script output at all, which might have some good clues...
Do you get anything interesting in the server log output if you change this line to log its output, as below?
const output = await run(adbClient, deviceId, rootCmd.concat('sh', injectionScriptPath));
console.log(output);
I'm hoping that that might produce some proper clues immediately. If not, but you do get the command output, it would be interesting to know what ls
and mount
output when run directly inside the script there, so I'd try that.
Does that make sense? Give that a go, if it doesn't work then I'm going to have to put together a test device that matches yours I think and start trying to reproduce this myself.
@pimterry Let's do this in anydesk, it's probably easier in that way? Or we can live chat in discord?
@shirshak55 I think that's a good idea, that'll definitely help. I'm afraid that's a bit difficult for me for the next few days though.
Are you free some time next week to dig into this, let's say on Wednesday? I'm in Spain, so CET time. Pick any time you like here: https://calendly.com/httptoolkit-tim/30min
Just writing up my notes from our call last week and my investigation since:
magisk
via adb to install a given zip. @shirshak55 what happens on your device if you reboot, then run adb shell
, then run magisk
? Does it exist? Do we need to run su
first to run it as root?adb remount
and then manually copying the certificate in does seem to work correctly and persistently, so we should probably do that. We only want to do that when it's strictly necessary though, so we need to detect this issue.mount
output (example output), just to make sure that's we're dealing with Magisk.adb remount
& inject it that way@pimterry Yes it exists and it seems we don't need su
raphaelin:/ $ magisk
Magisk - Multi-purpose Utility
Usage: magisk [applet [arguments]...]
or: magisk [options]...
Options:
-c print current binary version
-v print running daemon version
-V print running daemon version code
--list list all available applets
--remove-modules remove all modules and reboot
--install-module ZIP install a module zip file
Advanced Options (Internal APIs):
--daemon manually start magisk daemon
--[init trigger] start service for init trigger
Supported init triggers:
post-fs-data, service, boot-complete
--unlock-blocks set BLKROSET flag to OFF for all block devices
--restorecon restore selinux context on Magisk files
--clone-attr SRC DEST clone permission, owner, and selinux context
--clone SRC DEST clone SRC to DEST
--sqlite SQL exec SQL commands to Magisk database
--path print Magisk tmpfs mount path
Available applets:
su, resetprop, magiskhide
And, sorry for the delay
Excellent! No worries about the delay. It sounds like there's definitely a possible route there then: if we detect magisk
as a command on the device, then instead of using the mount approach we need to create a Magisk module zip containing the files, and temporarily install that module.
No worries about the delay of course, hope your exams went well!
If you have some free time now, any chance you want to work out how Magisk modules work, and build & test one out on your device? We need to:
magisk
does inject the cert on your device.If you can do that and share a working zip, then I can write the Node code to automatically build and install such zips, and then we'll have fixed this problem permanently.
I'm reading this whole entire blog and would like to say thank you. No one could have made this as clear as it was for me but you guys.
@pimterry I think there is already one magisk module that is available.
https://github.com/victor141516/httpcanary-magisk
We just need to put our certs there.
https://github.com/victor141516/httpcanary-magisk/tree/master/system/etc/security/cacerts
Obviously they are putting hard coded certs. But for security reasons I think we should use generated certs.
@GOTEM321 Thanks! Glad to hear that the blog posts helped you out :smile:
@shirshak55 Excellent find, yes that's a perfect example that we can use as a basis for this, great work.
The key step is testing whether it works to solve the issue for devices like yours though.
If you swap out the certificate in that module with your own, and then install it using magisk
, does it successfully install a system certificate on your device?
HTTP Toolkit is currently one of the best MITM proxy tools used for application development, reverse engineering, etc... However, at the moment there is a small issue with magisk rooted phone. Even though the phone is rooted, it seems CA certificates don't seem to be installed properly.