Investigate whether setting a proxy on Android is enough

baltpeter commented 1 year ago

To intercept the traffic on Android, we are currently just setting an HTTP proxy.

However, the GUI has a warning which suggests that apps may be able to ignore that setting (https://github.com/tweaselORG/meta/issues/18#issuecomment-1436650823), which we obviously don't want them to be able to.
While there are cases where traffic can evade a VPN as well, they are much less severe, and seem to only be the case for system/privileged apps.

We should investigate whether that is actually a problem for our use case. If it is, we need an alternative solution.

baltpeter commented 1 year ago

If this does turn out to be a problem, we should probably just wait for the HTTP Toolkit solution (cf. #6).

baltpeter commented 1 year ago

The mitproxy docs also mention that Android apps can just choose to ignore the proxy settings, so we should probably deal with this now.

One possible solution is to use transparent mode to direct traffic to the proxy at the network layer, without requiring client configuration.
However, that is quite complex to set up, both on the client and the server/proxy.

But mitmproxy 9 ships with a new WireGuard mode, which presents a very exciting alternative solution:

This new mode makes it incredibly easy to set up proxying for other devices in your network, and allows you to only proxy specific apps on Android.

baltpeter commented 1 year ago

Manual set up of the WireGuard mode is indeed really simple:

Install the WireGuard app on the phone.
Start mitmproxy with --mode wireguard, e.g. mitmweb --no-web-open-browser --mode wireguard.
This will output the WireGuard configuration, mitmweb even has a convenient QR code:
Launch the WireGuard app and add a new tunnel using the QR code or by manually entering the config values.
To choose which apps to proxy, go into the tunnel settings, click the pencil icon, and then click ALL APPLICATIONS. You can either use an allowlist or a blocklist of apps. Save the settings.
Enable and later disable the tunnel using the toggle.

If we can automate this process, this is clearly the way to go.

baltpeter commented 1 year ago

This also works on iOS (steps are the same), but unfortunately, I haven't found a way to choose which apps to proxy there.

baltpeter commented 1 year ago

The WireGuard app on Android can be controlled by intents (excerpt from adb shell dumpsys package com.wireguard.android):

Receiver Resolver Table:
  Non-Data Actions:
      com.wireguard.android.action.SET_TUNNEL_UP:
        92aa08c com.wireguard.android/.model.TunnelManager$IntentReceiver filter c5933d5
          Action: "com.wireguard.android.action.REFRESH_TUNNEL_STATES"
          Action: "com.wireguard.android.action.SET_TUNNEL_UP"
          Action: "com.wireguard.android.action.SET_TUNNEL_DOWN"
      com.wireguard.android.action.REFRESH_TUNNEL_STATES:
        92aa08c com.wireguard.android/.model.TunnelManager$IntentReceiver filter c5933d5
          Action: "com.wireguard.android.action.REFRESH_TUNNEL_STATES"
          Action: "com.wireguard.android.action.SET_TUNNEL_UP"
          Action: "com.wireguard.android.action.SET_TUNNEL_DOWN"
      com.wireguard.android.action.SET_TUNNEL_DOWN:
        92aa08c com.wireguard.android/.model.TunnelManager$IntentReceiver filter c5933d5
          Action: "com.wireguard.android.action.REFRESH_TUNNEL_STATES"
          Action: "com.wireguard.android.action.SET_TUNNEL_UP"
          Action: "com.wireguard.android.action.SET_TUNNEL_DOWN"
      android.intent.action.BOOT_COMPLETED:
        bd52bde com.wireguard.android/.BootShutdownReceiver filter 8d581bf
          Action: "android.intent.action.ACTION_SHUTDOWN"
          Action: "android.intent.action.BOOT_COMPLETED"
      android.intent.action.ACTION_SHUTDOWN:
        bd52bde com.wireguard.android/.BootShutdownReceiver filter 8d581bf
          Action: "android.intent.action.ACTION_SHUTDOWN"
          Action: "android.intent.action.BOOT_COMPLETED"

After a fair amount of trial and error, I have managed to start and stop a particular tunnel using:

am broadcast -a com.wireguard.android.action.SET_TUNNEL_UP -e tunnel <tunnel_name> com.wireguard.android
am broadcast -a com.wireguard.android.action.SET_TUNNEL_DOWN -e tunnel <tunnel_name> com.wireguard.android

Note that you also need to enable "Allow remote control apps" under "Advanced" in the settings of the WireGuard app.

This has two caveats, though:

This does not appear to work when the WireGuard app is not running (even with battery optimizations disabled for the app, it only sometimes works).
We cannot configure the tunneled apps through intents.

While not great, we could work around 1. by starting the app, sending the intent, and then stopping the app. But that doesn't solve 2.

I'm questioning how much sense it makes to put more time into this. Maybe we should just bite the bullet and produce Android builds of wireguard-go ourselves to use directly instead of using the app…

baltpeter commented 1 year ago

Ok, 2. doesn't seem to be too bad, either. The configs are stored in /data/data/com.wireguard.android/files/<tunnel_name>.conf. We can edit and create those with root rights without issue.

So far, I haven't figured out how to get WireGuard to recognize config changes without restarting the app.

baltpeter commented 1 year ago

Hm, I had found that these work more reliably (but also not always) when the app is closed (via):

am broadcast -a com.wireguard.android.action.SET_TUNNEL_UP -n 'com.wireguard.android/.model.TunnelManager$IntentReceiver' -e tunnel <tunnel_name>
am broadcast -a com.wireguard.android.action.SET_TUNNEL_DOWN -n 'com.wireguard.android/.model.TunnelManager$IntentReceiver' -e tunnel <tunnel_name>

But then I restarted the phone to test whether they also work then, and now I can't control the tunnel via intents at all anymore, even when the app is in the foreground.

baltpeter commented 1 year ago

For future reference: I saw https://github.com/WireGuard/wireguard-android/blob/713947e432126e0e29dcf497960e5fa0f6301e2b/tunnel/src/main/java/com/wireguard/android/util/ToolsInstaller.java#L95 and https://github.com/WireGuard/wireguard-android/blob/713947e432126e0e29dcf497960e5fa0f6301e2b/ui/src/main/java/com/wireguard/android/preference/ToolsInstallerPreference.kt#L12 was excited that there may be an easy way to install the command line tools through the app, but alas:

The command line tools were not removed. If you are using the Go userspace backend (the app will indicate this in settings, together with the version information) they will not be available. The tools control the kernel backend and require root access. Thus they are only available if you are running a custom kernel with WireGuard support and have root access.

baltpeter commented 1 year ago

Just a thought: How annoying is installing a kernel with WireGuard support? Maybe we can make that a requirement?

baltpeter commented 1 year ago

Just a thought: How annoying is installing a kernel with WireGuard support? Maybe we can make that a requirement?

Looks like it's too annoying, unfortunately: https://dzx.fr/blog/how-to-compile-the-wireguard-kernel-module-for-lineageos/, https://chyen.cc/blog/posts/2020/02/02/htc-u11-wireguard-kernel-module.html

baltpeter commented 1 year ago

Hm, I had found that these work more reliably (but also not always) when the app is closed (via):
am broadcast -a com.wireguard.android.action.SET_TUNNEL_UP -n 'com.wireguard.android/.model.TunnelManager$IntentReceiver' -e tunnel <tunnel_name>
am broadcast -a com.wireguard.android.action.SET_TUNNEL_DOWN -n 'com.wireguard.android/.model.TunnelManager$IntentReceiver' -e tunnel <tunnel_name>
But then I restarted the phone to test whether they also work then, and now I can't control the tunnel via intents at all anymore, even when the app is in the foreground.

Aha! I think I found the problem: The broadcast only works with root rights.

baltpeter commented 1 year ago

Ok, it seems to be working pretty reliably now. Even after rebooting the phone or manually killing the WireGuard app, I can reliably start and stop the proxy using:

am broadcast -a com.wireguard.android.action.SET_TUNNEL_UP -n 'com.wireguard.android/.model.TunnelManager$IntentReceiver' -e tunnel <tunnel_name>
am broadcast -a com.wireguard.android.action.SET_TUNNEL_DOWN -n 'com.wireguard.android/.model.TunnelManager$IntentReceiver' -e tunnel <tunnel_name>

But only if I disable battery optimizations for the app. With those enabled, it only works if the app is in the foreground.

baltpeter commented 1 year ago

I don't entirely trust that method, though. It would be good if we could verify that the VPN connection is working.

There are methods that work from Java: https://stackoverflow.com/questions/28386553/check-if-a-vpn-connection-is-active-in-android, https://developer.android.com/training/monitoring-device-state/connectivity-status-type
But I would really prefer if we didn't need Frida for this.

Running ifconfig tun0 has exit code 0 if the VPN is enabled and exit code 1 if it isn't (https://stackoverflow.com/a/37983590). That's a great start. But unfortunately it doesn't tell us whether it's actually a WireGuard VPN (the phone may also have some other VPN running shrug):

ocean:/ # ifconfig tun0                                                                                                                                                         
tun0      Link encap:UNSPEC  
          inet addr:10.0.0.1  P-t-P:10.0.0.1  Mask:255.255.255.255 
          inet6 addr: fe80::b2d5:2cbd:951a:3d9c/64 Scope: Link
          UP POINTOPOINT RUNNING  MTU:1280  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0 
          TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 
          collisions:0 txqueuelen:500 
          RX bytes:0 TX bytes:396

A comment also suggests that there may be other interfaces that we would need to consider.

baltpeter commented 1 year ago

I asked ChatGPT. It suggested adb shell dumpsys net and adb shell dumpsys vpn. Neither of those work (Can't find service: net, Can't find service: vpn).

After I complained, it pointed me towards adb shell service list. That's actually super interesting. It gives me a list of services that I can query using dumpsys:

Found 239 services:
0   DockObserver: []
1   SurfaceFlinger: [android.ui.ISurfaceComposer]
2   SurfaceFlingerAIDL: [android.gui.ISurfaceComposer]
3   accessibility: [android.view.accessibility.IAccessibilityManager]
4   account: [android.accounts.IAccountManager]
5   activity: [android.app.IActivityManager]
6   activity_task: [android.app.IActivityTaskManager]
7   adb: [android.debug.IAdbManager]
8   adbroot_service: [android.adbroot.IADBRootService]
9   alarm: [android.app.IAlarmManager]
10  ambient_context: [android.app.ambientcontext.IAmbientContextManager]
11  android.frameworks.stats.IStats/default: [android.frameworks.stats.IStats]
12  android.hardware.bluetooth.audio.IBluetoothAudioProviderFactory/default: [android.hardware.bluetooth.audio.IBluetoothAudioProviderFactory]
13  android.hardware.drm.IDrmFactory/clearkey: [android.hardware.drm.IDrmFactory]
14  android.hardware.power.IPower/default: [android.hardware.power.IPower]
15  android.hardware.wifi.supplicant.ISupplicant/default: [android.hardware.wifi.supplicant.ISupplicant]
16  android.os.UpdateEngineService: [android.os.IUpdateEngine]
17  android.os.UpdateEngineStableService: [android.os.IUpdateEngineStable]
18  android.security.apc: [android.security.apc.IProtectedConfirmation]
19  android.security.authorization: [android.security.authorization.IKeystoreAuthorization]
20  android.security.compat: [android.security.compat.IKeystoreCompatService]
21  android.security.identity: [android.security.identity.ICredentialStoreFactory]
22  android.security.legacykeystore: [android.security.legacykeystore.ILegacyKeystore]
23  android.security.maintenance: [android.security.maintenance.IKeystoreMaintenance]
24  android.security.metrics: [android.security.metrics.IKeystoreMetrics]
25  android.service.gatekeeper.IGateKeeperService: [android.service.gatekeeper.IGateKeeperService]
26  android.system.keystore2.IKeystoreService/default: [android.system.keystore2.IKeystoreService]
27  android.system.suspend.ISystemSuspend/default: [android.system.suspend.ISystemSuspend]
28  app_binding: []
29  app_hibernation: [android.apphibernation.IAppHibernationService]
30  app_integrity: [android.content.integrity.IAppIntegrityManager]
31  app_search: [android.app.appsearch.aidl.IAppSearchManager]
32  appops: [com.android.internal.app.IAppOpsService]
33  appwidget: [com.android.internal.appwidget.IAppWidgetService]
34  attestation_verification: [android.security.attestationverification.IAttestationVerificationManagerService]
35  audio: [android.media.IAudioService]
36  auth: [android.hardware.biometrics.IAuthService]
37  autofill: [android.view.autofill.IAutoFillManager]
38  backup: [android.app.backup.IBackupManager]
39  battery: []
40  batteryproperties: [android.os.IBatteryPropertiesRegistrar]
41  batterystats: [com.android.internal.app.IBatteryStats]
42  binder_calls_stats: []
43  biometric: [android.hardware.biometrics.IBiometricService]
44  blob_store: [android.app.blob.IBlobStoreManager]
45  bluetooth_manager: [android.bluetooth.IBluetoothManager]
46  bugreport: [android.os.IDumpstate]
47  cacheinfo: []
48  carrier_config: [com.android.internal.telephony.ICarrierConfigLoader]
49  clipboard: [android.content.IClipboard]
50  color_display: [android.hardware.display.IColorDisplayManager]
51  companiondevice: [android.companion.ICompanionDeviceManager]
52  connectivity: [android.net.IConnectivityManager]
53  connmetrics: [android.net.IIpConnectivityMetrics]
54  consumer_ir: [android.hardware.IConsumerIrService]
55  content: [android.content.IContentService]
56  country_detector: [android.location.ICountryDetector]
57  cpuinfo: []
58  crossprofileapps: [android.content.pm.ICrossProfileApps]
59  dataloader_manager: [android.content.pm.IDataLoaderManager]
60  dbinfo: []
61  device_config: []
62  device_identifiers: [android.os.IDeviceIdentifiersPolicyService]
63  device_policy: [android.app.admin.IDevicePolicyManager]
64  device_state: [android.hardware.devicestate.IDeviceStateManager]
65  deviceidle: [android.os.IDeviceIdleController]
66  devicestoragemonitor: []
67  diskstats: []
68  display: [android.hardware.display.IDisplayManager]
69  dnsresolver: [android.net.IDnsResolver]
70  domain_verification: [android.content.pm.verify.domain.IDomainVerificationManager]
71  dreams: [android.service.dreams.IDreamManager]
72  dropbox: [com.android.internal.os.IDropBoxManagerService]
73  dynamic_system: [android.os.image.IDynamicSystemService]
74  emergency_affordance: []
75  ethernet: [android.net.IEthernetManager]
76  external_vibrator_service: [android.os.IExternalVibratorService]
77  file_integrity: [android.security.IFileIntegrityService]
78  fingerprint: [android.hardware.fingerprint.IFingerprintService]
79  font: [com.android.internal.graphics.fonts.IFontManager]
80  game: [android.app.IGameManagerService]
81  gfxinfo: []
82  gpu: [android.graphicsenv.IGpuService]
83  graphicsstats: [android.view.IGraphicsStats]
84  hardware_properties: [android.os.IHardwarePropertiesManager]
85  imms: [com.android.internal.telephony.IMms]
86  incident: [android.os.IIncidentManager]
87  incidentcompanion: [android.os.IIncidentCompanion]
88  input: [android.hardware.input.IInputManager]
89  input_method: [com.android.internal.view.IInputMethodManager]
90  inputflinger: [android.os.IInputFlinger]
91  installd: [android.os.IInstalld]
92  ions: [com.android.internal.telephony.IOns]
93  iphonesubinfo: [com.android.internal.telephony.IPhoneSubInfo]
94  ipsec: [android.net.IIpSecService]
95  isms: [com.android.internal.telephony.ISms]
96  isub: [com.android.internal.telephony.ISub]
97  jobscheduler: [android.app.job.IJobScheduler]
98  launcherapps: [android.content.pm.ILauncherApps]
99  legacy_permission: [android.permission.ILegacyPermissionManager]
100 lights: [android.hardware.lights.ILightsManager]
101 lineageglobalactions: [lineageos.app.ILineageGlobalActions]
102 lineagehardware: [lineageos.hardware.ILineageHardwareService]
103 lineagelivedisplay: [lineageos.hardware.ILiveDisplayService]
104 lineagetrust: [lineageos.trust.ITrustInterface]
105 locale: [android.app.ILocaleManager]
106 location: [android.location.ILocationManager]
107 location_time_zone_manager: []
108 lock_settings: [com.android.internal.widget.ILockSettings]
109 logcat: [android.os.logcat.ILogcatManagerService]
110 looper_stats: []
111 manager: [android.os.IServiceManager]
112 mdns: [android.net.mdns.aidl.IMDns]
113 media.audio_flinger: [android.media.IAudioFlingerService]
114 media.audio_policy: [android.media.IAudioPolicyService]
115 media.camera: [android.hardware.ICameraService]
116 media.camera.proxy: [android.hardware.ICameraServiceProxy]
117 media.extractor: [android.IMediaExtractorService]
118 media.metrics: [android.media.IMediaMetricsService]
119 media.player: [android.media.IMediaPlayerService]
120 media.resource_manager: [android.media.IResourceManagerService]
121 media.resource_observer: [android.media.IResourceObserverService]
122 media_communication: [android.media.IMediaCommunicationService]
123 media_metrics: [android.media.metrics.IMediaMetricsManager]
124 media_projection: [android.media.projection.IMediaProjectionManager]
125 media_resource_monitor: [android.media.IMediaResourceMonitor]
126 media_router: [android.media.IMediaRouterService]
127 media_session: [android.media.session.ISessionManager]
128 meminfo: []
129 memtrack.proxy: [android.hardware.memtrack.IMemtrack]
130 midi: [android.media.midi.IMidiManager]
131 mount: [android.os.storage.IStorageManager]
132 nearby: [android.nearby.INearbyManager]
133 netd: [android.net.INetd]
134 netd_listener: [android.net.metrics.INetdEventListener]
135 netpolicy: [android.net.INetworkPolicyManager]
136 netstats: [android.net.INetworkStatsService]
137 network_management: [android.os.INetworkManagementService]
138 network_score: [android.net.INetworkScoreService]
139 network_stack: [android.net.INetworkStackConnector]
140 network_time_update_service: []
141 network_watchlist: [com.android.internal.net.INetworkWatchlistManager]
142 notification: [android.app.INotificationManager]
143 oem_lock: [android.service.oemlock.IOemLockService]
144 otadexopt: [android.content.pm.IOtaDexopt]
145 overlay: [android.content.om.IOverlayManager]
146 pac_proxy: [android.net.IPacProxyManager]
147 package: [android.content.pm.IPackageManager]
148 package_native: [android.content.pm.IPackageManagerNative]
149 people: [android.app.people.IPeopleManager]
150 performance_hint: [android.os.IHintManager]
151 permission: [android.os.IPermissionController]
152 permission_checker: [android.permission.IPermissionChecker]
153 permissionmgr: [android.permission.IPermissionManager]
154 persistent_data_block: [android.service.persistentdata.IPersistentDataBlockService]
155 phone: [com.android.internal.telephony.ITelephony]
156 pinner: []
157 platform_compat: [com.android.internal.compat.IPlatformCompat]
158 platform_compat_native: [com.android.internal.compat.IPlatformCompatNative]
159 power: [android.os.IPowerManager]
160 powerstats: []
161 print: [android.print.IPrintManager]
162 processinfo: [android.os.IProcessInfoService]
163 procstats: [com.android.internal.app.procstats.IProcessStats]
164 profile: [lineageos.app.IProfileManager]
165 reboot_readiness: [android.scheduling.IRebootReadinessManager]
166 recovery: [android.os.IRecoverySystem]
167 resources: [android.content.res.IResourcesManager]
168 restrictions: [android.content.IRestrictionsManager]
169 role: [android.app.role.IRoleManager]
170 rollback: [android.content.rollback.IRollbackManager]
171 runtime: []
172 safety_center: [android.safetycenter.ISafetyCenterManager]
173 scheduling_policy: [android.os.ISchedulingPolicyService]
174 sdk_sandbox: [android.app.sdksandbox.ISdkSandboxManager]
175 search: [android.app.ISearchManager]
176 search_ui: [android.app.search.ISearchUiManager]
177 sec_key_att_app_id_provider: [android.security.keymaster.IKeyAttestationApplicationIdProvider]
178 secure_element: [android.se.omapi.ISecureElementService]
179 sensor_privacy: [android.hardware.ISensorPrivacyManager]
180 sensorservice: [android.gui.SensorServer]
181 serial: [android.hardware.ISerialManager]
182 servicediscovery: [android.net.nsd.INsdManager]
183 settings: []
184 shortcut: [android.content.pm.IShortcutService]
185 simphonebook: [com.android.internal.telephony.IIccPhoneBook]
186 slice: [android.app.slice.ISliceManager]
187 smartspace: [android.app.smartspace.ISmartspaceManager]
188 soundtrigger: [com.android.internal.app.ISoundTriggerService]
189 soundtrigger_middleware: [android.media.soundtrigger_middleware.ISoundTriggerMiddlewareService]
190 speech_recognition: [android.speech.IRecognitionServiceManager]
191 stats: [android.os.IStatsd]
192 statsbootstrap: [android.os.IStatsBootstrapAtomService]
193 statscompanion: [android.os.IStatsCompanionService]
194 statsmanager: [android.os.IStatsManagerService]
195 statusbar: [com.android.internal.statusbar.IStatusBarService]
196 storaged: [android.os.IStoraged]
197 storaged_pri: [android.os.storaged.IStoragedPrivate]
198 storagestats: [android.app.usage.IStorageStatsManager]
199 suspend_control: [android.system.suspend.ISuspendControlService]
200 suspend_control_internal: [android.system.suspend.internal.ISuspendControlServiceInternal]
201 system_config: [android.os.ISystemConfig]
202 system_server_dumper: []
203 system_update: [android.os.ISystemUpdateManager]
204 tare: [android.app.tare.IEconomyManager]
205 telecom: [com.android.internal.telecom.ITelecomService]
206 telephony.registry: [com.android.internal.telephony.ITelephonyRegistry]
207 telephony_ims: [android.telephony.ims.aidl.IImsRcsController]
208 testharness: []
209 tethering: [android.net.ITetheringConnector]
210 textclassification: [android.service.textclassifier.ITextClassifierService]
211 textservices: [com.android.internal.textservice.ITextServicesManager]
212 texttospeech: [android.speech.tts.ITextToSpeechManager]
213 thermalservice: [android.os.IThermalService]
214 time_detector: [android.app.timedetector.ITimeDetectorService]
215 time_zone_detector: [android.app.timezonedetector.ITimeZoneDetectorService]
216 tracing.proxy: [android.tracing.ITracingServiceProxy]
217 transparency: [com.android.internal.os.IBinaryTransparencyService]
218 trust: [android.app.trust.ITrustManager]
219 uimode: [android.app.IUiModeManager]
220 updatelock: [android.os.IUpdateLock]
221 uri_grants: [android.app.IUriGrantsManager]
222 usagestats: [android.app.usage.IUsageStatsManager]
223 usb: [android.hardware.usb.IUsbManager]
224 user: [android.os.IUserManager]
225 vcn_management: [android.net.vcn.IVcnManagementService]
226 vibrator_manager: [android.os.IVibratorManagerService]
227 virtualdevice: [android.companion.virtual.IVirtualDeviceManager]
228 voiceinteraction: [com.android.internal.app.IVoiceInteractionManagerService]
229 vold: [android.os.IVold]
230 vpn_management: [android.net.IVpnManager]
231 wallpaper: [android.app.IWallpaperManager]
232 wallpaper_effects_generation: [android.app.wallpapereffectsgeneration.IWallpaperEffectsGenerationManager]
233 webviewupdate: [android.webkit.IWebViewUpdateService]
234 wifi: [android.net.wifi.IWifiManager]
235 wifinl80211: [android.net.wifi.nl80211.IWificond]
236 wifip2p: [android.net.wifi.p2p.IWifiP2pManager]
237 wifiscanner: [android.net.wifi.IWifiScanner]
238 window: [android.view.IWindowManager]

And there's quite a few. vpn_management lists the configured VPNs, but doesn't show me which one is enabled:

adb shell dumpsys vpn_management
VPNs:
  0: com.wireguard.android

There is a recent commit from October 2022 that will add more information but that obviously doesn't help us. Also, it seems like the vpn_management service was only introduced in Android 12.

baltpeter commented 1 year ago

But that Reddit post also mentioned service call connectivity 5 i32 17.

I mean, that does indeed tell me whether the VPN is connected, but wow:

Result: Parcel(
  0x00000000: 00000000 00000001 00000011 ffffffff '................'
  0x00000010: 00000003 00500056 0000004e 00000000 '....V.P.N.......'
  0x00000020: 00000000 00000009 004f0043 004e004e '........C.O.N.N.'
  0x00000030: 00430045 00450054 00000044 00000009 'E.C.T.E.D.......'
  0x00000040: 004f0043 004e004e 00430045 00450054 'C.O.N.N.E.C.T.E.'
  0x00000050: 00000044 00000000 00000001 00000000 'D...............'
  0x00000060: ffffffff 00000000 00000000          '............    ')

Result: Parcel(
  0x00000000: 00000000 00000001 00000011 00000000 '................'
  0x00000010: 00000003 00500056 0000004e 00000000 '....V.P.N.......'
  0x00000020: 00000000 0000000c 00490044 00430053 '........D.I.S.C.'
  0x00000030: 004e004f 0045004e 00540043 00440045 'O.N.N.E.C.T.E.D.'
  0x00000040: 00000000 0000000c 00490044 00430053 '........D.I.S.C.'
  0x00000050: 004e004f 0045004e 00540043 00440045 'O.N.N.E.C.T.E.D.'
  0x00000060: 00000000 00000000 00000001 00000000 '................'
  0x00000070: ffffffff ffffffff                   '........        ')

baltpeter commented 1 year ago

But what if we call dumpsys connectivity?

That outputs a whole bunch of stuff, including:

  NetworkAgentInfo{network{108}  handle{467262165005}  ni{VPN CONNECTED extra: } Score(101 ; KeepConnected : 0 ; Policies : EVER_VALIDATED_NOT_AVOIDED_WHEN_BAD&IS_UNMETERED&IS_VPN&IS_VALIDATED)  created everValidated lastValidated  lp{{InterfaceName: tun0 LinkAddresses: [ 10.0.0.1/32 ] DnsAddresses: [ /10.0.0.53 ] Domains:  MTU: 1280 Routes: [ 0.0.0.0/0 -> 0.0.0.0 tun0 mtu 0,::/0 unreachable mtu 0,10.0.0.1/32 -> 0.0.0.0 tun0 mtu 0 ]}}  nc{[ Transports: WIFI|VPN Capabilities: NOT_METERED&INTERNET&NOT_RESTRICTED&TRUSTED&VALIDATED&NOT_ROAMING&FOREGROUND&NOT_CONGESTED&NOT_SUSPENDED&NOT_VCN_MANAGED LinkUpBandwidth>=12000Kbps LinkDnBandwidth>=30000Kbps TransportInfo: <VpnTransportInfo{type=1, sessionId=abc}> Uids: <{0-99999}> OwnerUid: 10160 AdminUids: [10160] UnderlyingNetworks: [106]]}  factorySerialNumber=3}

Most importantly, it actually has the name of the connected tunnel (<VpnTransportInfo{type=1, sessionId=abc}>). That's super helpful!

baltpeter commented 1 year ago

Another option, though more annoying:

> adb shell dumpsys wifi | grep -i wireguard

02-27 14:40:40.448  6229  6267 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Retrying handshake because we stopped hearing back after 15 seconds
02-27 14:40:40.448  6229  6267 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending handshake initiation
02-27 14:40:40.576  6229  6498 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Received handshake response
02-27 14:40:40.577  6229  6498 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending keepalive packet
02-27 14:41:08.241  6229  6267 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Retrying handshake because we stopped hearing back after 15 seconds
02-27 14:41:08.243  6229  6267 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending handshake initiation
02-27 14:41:08.267  6229  6267 D WireGuard/GoBackend/abc: Receiving cookie response from 10.0.0.68:51820
02-27 14:41:13.473  6229  6266 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Handshake did not complete after 5 seconds, retrying (try 2)
02-27 14:41:13.474  6229  6266 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending handshake initiation
02-27 14:41:13.508  6229  6267 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Received handshake response
02-27 14:41:13.508  6229  6267 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending keepalive packet
02-27 14:41:36.531  6229  6266 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Retrying handshake because we stopped hearing back after 15 seconds
02-27 14:41:36.532  6229  6266 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending handshake initiation
02-27 14:41:36.560  6229  6498 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Received handshake response
02-27 14:41:36.560  6229  6498 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending keepalive packet
02-27 14:42:05.047  6229  6266 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Retrying handshake because we stopped hearing back after 15 seconds
02-27 14:42:05.047  6229  6266 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending handshake initiation
02-27 14:42:05.074  6229  6498 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Received handshake response
02-27 14:42:05.074  6229  6498 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending keepalive packet
02-27 14:42:33.218  6229  6498 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Retrying handshake because we stopped hearing back after 15 seconds
02-27 14:42:33.219  6229  6498 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending handshake initiation
02-27 14:42:33.226  6229  6266 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Received handshake response
02-27 14:42:33.226  6229  6266 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending keepalive packet
02-27 14:42:33.218  6229  6498 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Retrying handshake because we stopped hearing back after 15 seconds
02-27 14:42:33.219  6229  6498 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending handshake initiation
02-27 14:42:33.226  6229  6266 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Received handshake response
02-27 14:42:33.226  6229  6266 D WireGuard/GoBackend/abc: peer(Gbgx…m9i0) - Sending keepalive packet

baltpeter commented 1 year ago

Unfortunately, the dumpsys connectivity outputs differs a fair bit between Android versions.

Here's Android 9:

  NetworkAgentInfo{ ni{[type: VPN[], state: CONNECTED/CONNECTED, reason: agentConnect, extra: (none), failover: false, available: true, roaming: false]}  network{101}  nethandle{437197393933}  lp{{InterfaceName: tun0 LinkAddresses: [10.0.0.1/32,]  Routes: [0.0.0.0/0 -> 0.0.0.0 tun0,::/0 unreachable,10.0.0.1/32 -> 0.0.0.0 tun0,] DnsAddresses: [10.0.0.53,] UsePrivateDns: false PrivateDnsServerName: null Domains:  MTU: 0}}  nc{[ Transports: WIFI|VPN Capabilities: NOT_METERED&INTERNET&NOT_RESTRICTED&TRUSTED&VALIDATED&NOT_ROAMING&FOREGROUND&NOT_CONGESTED&NOT_SUSPENDED Unwanted:  LinkUpBandwidth>=1048576Kbps LinkDnBandwidth>=1048576Kbps Uids: <{0-99999}> EstablishingAppUid: 10047]}  Score{101}  everValidated{true}  lastValidated{true}  created{true} lingering{false} explicitlySelected{false} acceptUnvalidated{false} everCaptivePortalDetected{false} lastCaptivePortalDetected{false} clat{null} }

But adb shell service call connectivity 5 i32 17 seems stable:

Result: Parcel(
  0x00000000: 00000000 00000001 00000011 00000000 '................'
  0x00000010: 00000003 00500056 0000004e 00000000 '....V.P.N.......'
  0x00000020: 00000000 00000009 004f0043 004e004e '........C.O.N.N.'
  0x00000030: 00430045 00450054 00000044 00000009 'E.C.T.E.D.......'
  0x00000040: 004f0043 004e004e 00430045 00450054 'C.O.N.N.E.C.T.E.'
  0x00000050: 00000044 00000000 00000001 00000000 'D...............'
  0x00000060: 0000000c 00670061 006e0065 00430074 '....a.g.e.n.t.C.'
  0x00000070: 006e006f 0065006e 00740063 00000000 'o.n.n.e.c.t.....'
  0x00000080: ffffffff                            '....            ')

baltpeter commented 1 year ago

Well, it would probably be possible to get the desired information through service (https://stackoverflow.com/a/53198696, https://gist.github.com/tniessen/ea3d68e7d572ed7c607b81d715798800). But they are very unstable (https://android.stackexchange.com/a/46523) and if something changes in the future, calling a random function by number is quite dangerous. :D

baltpeter commented 1 year ago

Just for the record, here's how to get this information using Frida:

var app_ctx = Java.use('android.app.ActivityThread').currentApplication().getApplicationContext();
var cm = Java.cast(app_ctx.getSystemService('connectivity'), Java.use('android.net.ConnectivityManager'));
cm.getNetworkCapabilities(cm.getActiveNetwork()).toString();

This will print something like:

"[ Transports: WIFI|VPN Capabilities: NOT_METERED&INTERNET&NOT_RESTRICTED&TRUSTED&VALIDATED&NOT_ROAMING&FOREGROUND&NOT_CONGESTED&NOT_SUSPENDED&NOT_VCN_MANAGED LinkUpBandwidth>=12000Kbps LinkDnBandwidth>=30000Kbps TransportInfo: <VpnTransportInfo{type=1, sessionId=abc}> Uids: <{0-99999}> AdminUids: [10160] UnderlyingNetworks: [112]]"

But unfortunately, for that to work, the app you're injecting into needs android.permission.ACCESS_NETWORK_STATE:

[moto g 7  power::Clock ]-> cm.getNetworkCapabilities(cm.getActiveNetwork()).toString();
Error: java.lang.SecurityException: ConnectivityService: Neither user 10114 nor current process has android.permission.ACCESS_NETWORK_STATE.
    at <anonymous> (frida/node_modules/frida-java-bridge/lib/env.js:124)
    at value (frida/node_modules/frida-java-bridge/lib/class-factory.js:1064)
    at e (frida/node_modules/frida-java-bridge/lib/class-factory.js:585)
    at apply (native)
    at value (frida/node_modules/frida-java-bridge/lib/class-factory.js:969)
    at e (frida/node_modules/frida-java-bridge/lib/class-factory.js:552)
    at <eval> (<input>:3)
    at eval (native)

baltpeter commented 1 year ago

Alright, I won't waste any more time on this. Before enabling the tunnel, we'll call ifconfig tun0 to assert that no VPN is connected already, afterwards we'll call it again to ensure that we are now connected to a VPN. That has to be enough.

baltpeter commented 1 year ago

Oh, how annoying. You can't just am broadcast -a com.wireguard.android.action.SET_TUNNEL_DOWN -n 'com.wireguard.android/.model.TunnelManager$IntentReceiver' to stop the currently running tunnel. You need to actually specify the tunnel to stop.

I really don't want to have to keep track of that…

baltpeter commented 1 year ago

kill $(pidof -s com.wireguard.android) would be a workaround. But the disadvantage is that the VPN will automatically be enabled again if you start the app the next time…

baltpeter commented 1 year ago

But the disadvantage is that the VPN will automatically be enabled again if you start the app the next time…

Oh dear, that gave me a horrible idea. :D If it does that, it has to remember the last tunnel somewhere. What if, instead of doing that intent call, we just modified that file and restarted WireGuard afterwards? :D

baltpeter commented 1 year ago

That configuration lives at /data/data/com.wireguard.android/files/datastore/settings.preferences_pb. It's a Protobuf (with this schema).

Reading and changing that wouldn't be too hard, but we would have to depend on protobufjs. I'm genuinely tempted to do that. @zner0L: What should I do? Intents with all the caveats listed above or this?

zner0L commented 1 year ago

What's the problem with just keeping track of the tunnel ourselves? Then we can properly shut them down without depending on some weird, unstable way of modifying the config. I mean, all of this seems a little bodgey to me and having read all these caveats I am wondering why we are not just using the HTTPToolkit app? Or does it have the same problems when it comes to automation?

baltpeter commented 1 year ago

I am wondering why we are not just using the HTTPToolkit app? Or does it have the same problems when it comes to automation?

Yes. Automating that also depends on intents.

In the end, the question is what is more stable? Intents or writing the config? I don't see that many caveats—there is just a lot of research and trial-and-error in this issue, but most of that won't actually be used in the end.

baltpeter commented 1 year ago

Yes. Automating that also depends on intents.

And I'm not sure whether we could filter on individual apps with that. Iirc, no.

zner0L commented 1 year ago

Ah, the main problem you see is that the app needs to to be in the foreground to respond to intents! I don't think this is that much of a problem. Also, I am pretty sure that the intents are more stable, since they are also used by the app to communicate with itself. The feature that the same VPN is started again after the app was stopped seems like something that could just go away. I don't think the intents will ever vanish completely, more like the API would change a little and then we don't need to change to whole process of interacting with the app. So, I like that way much more.

baltpeter commented 1 year ago

While looking for a way to programmatically disable battery optimizations, I happened to notice that Termux includes a package manager that has a wireguard-tools package already.

So, of course, I had to try that[^1]. Unfortunately, I didn't get far:

ocean:/ # /data/data/com.termux/files/usr/bin/wg-quick up ./mitmproxy.conf
Warning: `./mitmproxy.conf' is world accessible
[#] ip link add mitmproxy type wireguard
RTNETLINK answers: Operation not supported on transport endpoint
[#] ip link del mitmproxy
Cannot find device "mitmproxy"

wireguard-tools depend on the WireGuard kernel. Adding the link we need doesn't work, because ip doesn't know the type wireguard:

ocean:/ # ip link add dev wg0 type wireguard                                                                                                        
RTNETLINK answers: Operation not supported on transport endpoint

[^1]: Side note: I didn't fancy typing all those command on a phone. This script didn't work for me. But the binaries installed through pkg are in /data/data/com.termux/files/usr/bin/ and you can just run them from there. The only thing I found not to work was pkg (the env is set up correctly), so those commands still have to be run through the app.

baltpeter commented 1 year ago

Termux makes it pretty easy to build wireguard-go on Android. But even after a lot of fiddling, I didn't manage to connect. I'm not sure whether it's worth pursuing that further.

Some general notes on how far I got:

You need: pkg install git golang make
By default, it tries to create a socket in /var (https://github.com/WireGuard/wireguard-go/blob/21636207a6756120f56f836b06c086b627f7b911/ipc/uapi_unix.go#L27-L29) but that doesn't exist on Android. Luckily, that can be overridden with a linker flag: https://github.com/WireGuard/wireguard-android/blob/713947e432126e0e29dcf497960e5fa0f6301e2b/tunnel/tools/libwg-go/Makefile#L50
I built with: go build -tags linux -ldflags="-X golang.zx2c4.com/wireguard/ipc.socketDirectory=/data/data/com.termux/files/home/sockdir" -v -trimpath
Running sudo ./wireguard wg0, I can create a network interface that shows up under ifconfig wg0.
But I cannot get that interface to actually be used.

Running wg-quick up ./mitmproxy.conf (installed using pkg install wireguard-tools on the config mitmproxy spat out fails because it tries to create a new interface using the kernel implementation:

~ $ wg-quick up ./mitmproxy.conf
[$] su PATH=/data/data/com.termux/files/usr/bin:/system/bin wg-quick up ./mitmproxy.conf            Warning: `./mitmproxy.conf' is world accessible
[#] ip link add mitmproxy type wireguard
RTNETLINK answers: Operation not supported on transport endpoint
[#] ip link del mitmproxy                         Cannot find device "mitmproxy"

boringtun suggests an env var to get wg-quick to use a userspace implementation, but WG_QUICK_USERSPACE_IMPLEMENTATION=./wireguard-go/wireguard wg-quick up ./mitmproxy.conf had the same problem for me.
If I rename the config wg0.conf and move it to ~/.wireguard in order to get it to use the existing wg0 interface, it complains about the interface already existing (d'oh!):
```
~ $ wg-quick up wg0
[$] su PATH=/data/data/com.termux/files/usr/bin:/system/bin wg-quick up wg0
Error: wg0 already exists
```
Following the manual setup steps using wg doesn't work either:
- I can add an IP to the interface using sudo ip address add dev wg0 10.0.0.1/32.
- If I do wg setconf wg0 ./mitmproxy.conf, it complains about wg-quick-specific options:
```
~ $ wg setconf wg0 ./mitmproxy.conf
Line unrecognized: `Address=10.0.0.1/32'
Configuration parsing error
```
- If I remove those (Address and DNS), I get this:
```
~ $ sudo wg setconf wg0 mitmproxy-wg.conf
Unable to modify interface: Protocol not supported
```

Here's the shell logs in case I forgot something significant: wg-go-log1.txt, wg-go-log2.txt

baltpeter commented 1 year ago

Actually… Now that I'm implementing this: Is there even any reason to generate unique tunnel names each time?

Currently, whenever the user starts a tunnel, I'm creating a new tunnel config named appstraction-<timestamp>. But that has a few downsides:

To stop the tunnel later on, we need to remember its name. This is annoying but not a problem as long as our process is running (I'm just keeping it in a variable). But when the appstraction process exits, and the user starts a new process, that information is obviously lost. In my current implementation I throw an error in that case.
We leak tunnel configs. I'm currently only cleaning up the config when the tunnel is stopped cleanly through setProxy(null). But if you forget to do that, or just do multiple setProxy(config) calls without setProxy(null) in-between (no reason for that, you can just override the current tunnel), all those tunnel configs are left behind.

Considering this, I think it would be much nicer if we just used a static tunnel name (probably appstraction) and overrode that file whenever a new tunnel is started.

baltpeter commented 1 year ago

OK, I'm pretty happy with the implementation now. I haven't managed to make it fail in my testing. It also survived 100 passes of enabling and immediately disabling the tunnel without issues.

tweaselORG / meta

Investigate whether setting a proxy on Android is enough #19