android: cant find trusted cert after install using root adb

[mixalbl4-127]

Hello. Your application has always worked fine for me, but today I suddenly found a window with “Manual setup required for android 11”. 8 hours of research into the problem showed the following:

1. The certificate successfully appears in the system /system/etc/security/cacerts/4f74014f.0
2. When I launch the application, I get in the logs: Cert already installed, nothing to do

Debugging the Android application showed that the whereIsCertTrusted function returns null. This certificate 4f74014f.0 is not in the keyStore.aliases() array, so the application refuses to start. This certificate is also not in the list of system certificates in the phone settings.

But the best part is, I used a hack and replaced null with "system" to force the application to think that the certificate was installed, and I got success!

All HTTPS traffic appeared in the application (including from YouTube, etc.), which indicates that the certificate is installed and working.

Total:

1. I don’t know why KeyStore.getInstance("AndroidCAStore") does not see this certificate
2. I don’t know why this worked for me a month ago and for the last 3-5 years.

Since my experiment shows that, despite the absence of a certificate in the list obtained by the KeyStore.getInstance("AndroidCAStore") method, it works successfully, I propose to improve this method whereIsCertTrusted in one of two ways:

1. Check for the presence of a certificate in the /system/etc/security/cacerts/ folder, as the desktop application does.
2. Send a test HTTPS request over the network and if it is successful, it means that the certificate is installed and working successfully.

It would also be a good idea to add a Sentry metric and compare what % of devices have a certificate successfully installed and working but do not have it in KeyStore.getInstance("AndroidCAStore") to understand how often this problem occurs.

software: HTTP Toolkit desktop v1.14.3 android HTTP Toolkit 1.3.10 OnePlus 9 Pro 5G Android 11

[pimterry]

Can you share more info about your device setup? Something very unusual is happening there. How exactly was this device rooted?

By email you mentioned that the certificate is installed on disk, but not visible, and that you see an nsenter error message. Can you share those full logs from that issue?

What output do you see if you run nsenter --help on the device? That should print the supported options and the version number.

Does the device have a /apex/com.android.conscrypt/cacerts? Does your certificate exist in that directory? It's surprising if it does, since that was only introduced by Google recently, but maybe it's being created by your root setup? Or your OS is using new system module updates from the Play Store, in an older OS version? I'm not totally sure how that works but that could plausibly explain this. There's some more context about that which you might find interesting here: https://httptoolkit.com/blog/android-14-install-system-ca-certificate/

In general, we can look into workarounds like you suggest, but I'd much rather properly solve the problem. If AndroidCAStore reports that certificate not existing, then there will be real cases where the certificates won't work as expected (any other Android apps that look at AndroidCAStore, for example - these are standard & widely used APIs). It would be much better to install the certificates so that they work successfully.

[mixalbl4-127]

Can you share more info about your device setup? Something very unusual is happening there. How exactly was this device rooted?

Nothing unusual, Systemless root using Magisk 23.0 (23000)

Can you share those full logs from that issue?

Android command [ 'sh', '/data/local/tmp/htk-root-test.sh' ] returned `shell`
Android command [ 'su', '-c', 'sh /data/local/tmp/htk-root-test.sh' ] returned `root`
Android command [ 'su', '-c', 'sh', '/data/local/tmp/htk-root-test.sh' ] returned `root`
httptoolkit-server: Updating CLI... not updatable
Android command [ 'su', 'root', 'sh', '/data/local/tmp/htk-root-test.sh' ] threw Timeout for ADB command su,root,sh,/data/local/tmp/htk-root-test.sh
Error: Timeout for ADB command su,root,sh,/data/local/tmp/htk-root-test.sh
    at /home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/adb-commands.ts:129:37
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at listOnTimeout (node:internal/timers:528:9)
    at processTimers (node:internal/timers:502:7)
    at async Promise.all (index 3)
    at getRootCommand (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/adb-commands.ts:190:34)
    at AndroidAdbInterceptor.injectSystemCertIfPossible (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/android-adb-interceptor.ts:190:25)
    at AndroidAdbInterceptor.activate (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/android-adb-interceptor.ts:69:9)
    at ApiModel.activateInterceptor (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/api/api-model.ts:177:28)
    at /home/xxx/AndroidStudioProjects/httptoolkit-server/src/api/rest-api.ts:81:24
    at /home/xxx/AndroidStudioProjects/httptoolkit-server/src/api/rest-api.ts:173:20
Android command [ 'su', 'root', 'sh /data/local/tmp/htk-root-test.sh' ] threw Timeout for ADB command su,root,sh /data/local/tmp/htk-root-test.sh
Error: Timeout for ADB command su,root,sh /data/local/tmp/htk-root-test.sh
    at /home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/adb-commands.ts:129:37
    at async Promise.all (index 4)
    at getRootCommand (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/adb-commands.ts:190:34)
    at AndroidAdbInterceptor.injectSystemCertIfPossible (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/android-adb-interceptor.ts:190:25)
    at AndroidAdbInterceptor.activate (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/android-adb-interceptor.ts:69:9)
    at ApiModel.activateInterceptor (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/api/api-model.ts:177:28)
    at /home/xxx/AndroidStudioProjects/httptoolkit-server/src/api/rest-api.ts:81:24
    at /home/xxx/AndroidStudioProjects/httptoolkit-server/src/api/rest-api.ts:173:20
Adding cert file as /data/local/tmp/4f74014f.0
Android command [ 'rm', '-f', '/data/local/tmp/htk-root-test.sh' ] returned ``
Android command [ 'su', '-c', 'sh /data/local/tmp/htk-inject-system-cert.sh' ] returned `System cacerts setup completed
Injecting certificates into APEX cacerts
nsenter: setns: Invalid argument`
Error: System certificate injection failed
    at injectSystemCertificate (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/adb-commands.ts:357:15)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at AndroidAdbInterceptor.injectSystemCertIfPossible (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/android-adb-interceptor.ts:213:17)
    at AndroidAdbInterceptor.activate (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/interceptors/android/android-adb-interceptor.ts:69:9)
    at ApiModel.activateInterceptor (/home/xxx/AndroidStudioProjects/httptoolkit-server/src/api/api-model.ts:177:28)
    at /home/xxx/AndroidStudioProjects/httptoolkit-server/src/api/rest-api.ts:81:24
    at /home/xxx/AndroidStudioProjects/httptoolkit-server/src/api/rest-api.ts:173:20
Android command [ 'su', '-c', 'sh /data/local/tmp/htk-set-chrome-flags.sh' ] returned `Chrome flags script completed`
Chrome flags script completed

Android command [ 'su', '-c', 'am force-stop com.android.chrome' ] returned ``
Android Chrome flags set

What output do you see if you run nsenter --help on the device? That should print the supported options and the version number.

OnePlus9Pro:/ $ nsenter --help
usage: nsenter [-t pid] [-F] [-i] [-m] [-n] [-p] [-u] [-U] COMMAND...

Run COMMAND in an existing (set of) namespace(s).

-t  PID to take namespaces from    (--target)
-F  don't fork, even if -p is used (--no-fork)

The namespaces to switch are:

-i  SysV IPC: message queues, semaphores, shared memory (--ipc)
-m  Mount/unmount tree (--mount)
-n  Network address, sockets, routing, iptables (--net)
-p  Process IDs and init, will fork unless -F is used (--pid)
-u  Host and domain names (--uts)
-U  UIDs, GIDs, capabilities (--user)

If -t isn't specified, each namespace argument must provide a path
to a namespace file, ala "-i=/proc/$PID/ns/ipc"

Does the device have a /apex/com.android.conscrypt/cacerts?

yes

Does your certificate exist in that directory?

Yes, 134 certs inside.

It would be much better to install the certificates so that they work successfully.

I'm agree with you)

p.s. Google play 37.6.24-29 Google play services 23.35.15

[mixalbl4-127]

By the way, not so long ago my SafeNet stopped being verified and the Eval type changed from Software to Hardware (as a result, the McDonald's app stopped working :D ). It looks like Google is tightening the screws on users through updates from Google Play.

[mixalbl4-127]

@pimterry it is possible to downgrade services for prevent using apex or maybe update nsenter? Cant find any info about it internet(

[mixalbl4-127]

Ok, I'm fixed: https://github.com/httptoolkit/httptoolkit-server/pull/95 give me the pro license )))

[pimterry]

@mixalbl4-127 if you have a second, would you mind testing again with your device on the current server code on main? There's been another change to this (https://github.com/httptoolkit/httptoolkit-server/commit/8577f6a683ac8723d3c27eb62d8e83f25d809ff4 - not yet released) because it turns out that in practice devices can have multiple zygote64 processes running at the same time (so the individual pid variables can contain spaces too).

I think the new bashism here should cover that case and your setup too, but it'd be great to get a quick confirmation if you have a moment to check!

[mixalbl4-127]

@pimterry checked, everything work ok!

but I have one more problem: the application that I wanted to test (which is where this issue started) changes its behavior after activating the http toolkit. The application begins to think that I am in the country of russia and does not work, since this is not a supported region. Why can it think this way when the HTTP toolkit is active?

[pimterry]

Hard to say for sure, but if you haven't changed any device settings then it's likely that this is IP-based geolocation, and so this means that the server is seeing your traffic coming from a Russian-based IP (this kind of geolocation isn't very accurate, but it's usually good enough for country-level accuracy).

When using HTTP Toolkit, apps on your phone do connect to servers differently, although not in any way that would usually cause this. The difference is that instead of the connection going from your phone to the server (using whatever network configuration your phone has) it goes to your computer, and then uses the standard routes from your computer to the server. It depends on your computer's connection to the internet, not your phone's.

If for some reason your computer connects to the internet differently to your phone, you could see this kind of behaviour. For example, if you connect to a VPN on your computer, then traffic from your phone (sent via HTTP Toolkit on your computer) will appear to come from your computer's VPN instead of your phone's IP. Does that make sense?

[mixalbl4-127]

Does that make sense?

Most likely not, since the laptop and phone are connected to the same wifi access point :(

upd: I used Charles instead of HttpToolkit but with your cert and ProxyDroid instead of vpn connection, everythin working clean without any "russian" locations. Route is same: Phone trafic throw proxy (wifi) => Laptop (wifi) => internet

[pimterry]

What happens if you visit ip-api.com from your phone while intercepted? That will tell you what IP the traffic appears to come from, and all the known metadata about that IP that other services might be using.

Otherwise, I don't know any real reason why this would differ between Charles and HTTP Toolkit. It is possible to configure an upstream proxy for HTTP Toolkit (either in the Pro settings, or system-wide if you've configured one in your OS) in which case the traffic from your computer would appear to come from that proxy. Hard to know what else to say though - when forwarding traffic, HTTP Toolkit just uses your computer's settings and your local connection to reach the relevant upstream server, so if you're seeing traffic being sent via unusual IPs then that's something to do with your local environment.

What happens if you use ProxyDroid with HTTP Toolkit, instead of using the VPN connection? Maybe the issue is that the app detects whether the device has a VPN running, and does something different in that case?

Alternatively, it might be that this app isn't using IP-based geolocation at all - it could make this decision in all sorts of ways, it's very hard to know. If you want more detailed answers, you'd need to either reverse engineer the app (or check if it's defined in one of the server responses) or ask the app developers directly.