v055c: Proxy Database migration failure

celzero / rethink-app

DNS over HTTPS / DNS over Tor / DNSCrypt client, WireGuard proxifier, firewall, and connection tracker for Android.

https://rethinkfirewall.com/

Apache License 2.0

2.82k stars 143 forks source link

v055c: Proxy Database migration failure #1276

Closed netizeni closed 4 months ago

netizeni commented 6 months ago

I switched from v0.5.5a to v0.5.5c today. While importing WireGuard configs, on 4th of 5th imported config, when trying to rename it in order to give it a descriptive location name, an error appeared:

03-12 17:56:22.339 21827 27904 I ProxyLogs: updating interface for config: 14, wg14
03-12 17:56:22.343 21827 27904 E ProxyLogs: Exception while parsing wg interface: UNIQUE constraint failed: ProxyApplicationMapping.uid, ProxyApplicationMapping.packageName, ProxyApplicationMapping.proxyId (code 1555 SQLITE_CONSTRAINT_PRIMARYKEY[1555])
03-12 17:56:22.343 21827 27904 E ProxyLogs: android.database.sqlite.SQLiteConstraintException: UNIQUE constraint failed: ProxyApplicationMapping.uid, ProxyApplicationMapping.packageName, ProxyApplicationMapping.proxyId (code 1555 SQLITE_CONSTRAINT_PRIMARYKEY[1555])
03-12 17:56:22.343 21827 27904 E ProxyLogs:     at android.database.sqlite.SQLiteConnection.nativeExecuteForChangedRowCount(Native Method)
03-12 17:56:22.343 21827 27904 E ProxyLogs:     at android.database.sqlite.SQLiteConnection.executeForChangedRowCount(SQLiteConnection.java:1179)
03-12 17:56:22.343 21827 27904 E ProxyLogs:     at android.database.sqlite.SQLiteSession.executeForChangedRowCount(SQLiteSession.java:756)
03-12 17:56:22.343 21827 27904 E ProxyLogs:     at android.database.sqlite.SQLiteStatement.executeUpdateDelete(SQLiteStatement.java:67)

After that, it wasn't possible to import any additional config or rename already imported. Two strange things followed up.

First, when switched to DNS part of app, beside usual three (System, Other and Rethink) DNS options, at the top appeared "WireGuard" and it was selected. Now, I can't reproduce it to appear again, so is this something intentionally and what this option does? That option seems the most logic one if we want to use DNS of VPN service, for which we imported WireGuard config. We could add it under "Other DNS" of course, but for example, Mullvad stating on their website, there's zero to none benefit of using their DoH when already connected.
Second, when returning to Proxy part of app, all previously imported configs got deleted somehow. I would like to search in logs in order to give more details, but I'm not sure what should I need to search for.

When connected, WireGuard proxy often failing and restarting on this version of app, which almost never happened on 0.5.5a. What are some log entries to search for in logs to debug this? As logs are quite big, I would rather share them only partially, as I'm not sure do they contain any Personally Identifiable Information (PII).

Lastly, one question regarding advanced WireGuard configuration, "Lockdown" and "Always-on" options, since it's a bit unclear to me. As all previous configs were deleted, there's only one WireGuard config now. Which option to enable in order to route all apps except a few through this VPN connection? Currently, Lockdown is enabled, always-on is not and all apps except those few are selected, so I assume this is the way? If both options are selected, these excluded apps become included as well, even that they aren't selected?

ignoramous commented 6 months ago

That db bug looks ominous. Thanks for the log, we'll take a look on priority.

First, when switched to DNS part of app, beside usual three (System, Other and Rethink) DNS options, at the top appeared "WireGuard" and it was selected. Now, I can't reproduce it to appear again, so is this something intentionally and what this option does?

When using WireGuard in "Simple" mode, Rethink uses WireGuard to also forward DNS. This is when you see the 4th non-changeable "WireGuard DNS" option.
When using in "Advanced" mode, Rethink uses an "Always-on" WireGuard (if any) to proxy queries to user-selected DNS.

when returning to Proxy part of app, all previously imported configs got deleted somehow. I would like to search in logs in order to give more details, but I'm not sure what should I need to search for.

Not surprising this happened given the db logs you shared. Looks like something funny is going on. Can you clarify what you mean by "returning to Proxy part" and "configs got deleted"? As in, all WireGuard configs are gone now? After an attempted rename post-upgrade?

WireGuard proxy often failing and restarting on this version of app, which almost never happened on 0.5.5a.

Are you going by the status shown in the UI (we think there's a bug there), or do you see frequently dropped connections? WireGuard itself is connectionless, btw. A "restart", on paper, should only happen when WireGuard configuration changes. It is worrying if you know for certain that restarts happen often.

As logs are quite big...

That they might be if you've enabled Verbose logs from Configure -> Settings -> Log Level. You can share them over email (mz at celzero dot com) and reference this GitHub issue. It may contain PII (think IPs and not secret keys or passwords or anything) Promise to only use it for debug / diagnosis and delete it right away.

Currently, Lockdown is enabled, always-on is not and all apps except those few are selected, so I assume this is the way? If both options are selected...

Lockdown means, apps part of the WireGuard configuration ("included apps") would never be routed outside of the WireGuard tunnel (regardless of whether the tunnel is connected or failing or disabled).
Always-on means, this WireGuard tunnel cannot be turned OFF. And ALL apps not routed by any other proxy or tunnel will be routed through an Always-on WireGuard tunnel (of which there could be many, and so, one is chosen at random).

As all previous configs were deleted...

Sounds terrible 😞 How many configurations?

netizeni commented 6 months ago

Thanks for the answer.

When using in "Advanced" mode, Rethink uses an "Always-on" WireGuard (if any) to proxy queries to user-selected DNS.

Sorry, could you please explain this a bit more detailed? I selected advanced mode in order to exclude some apps from going through the proxy connection, but seems like in that case the user has to use "Other DNS" which will slow down DNS requests (as mentioned on Mullvad website).

What would be the best way to set up Rethink app in order to have an always on proxy connection, all apps (except a few) and DNS requests going through it?

Can you clarify what you mean by "returning to Proxy part" and "configs got deleted"? As in, all WireGuard configs are gone now? After an attempted rename post-upgrade?

Honestly, I don't remember the exact "path" I took and what preceded deletion, because I didn't expect it. Basically, on Home tab, I went to "Proxy" and started importing WireGuard configs with QR code scanning. Fifth config was imported successfully, but when I tried to rename it, the error in OP appeared. I think I went to Home tab > DNS, checked that "Other DNS" is selected, and returned Home tab > Proxy and all configs were gone.

Are you going by the status shown in the UI (we think there's a bug there), or do you see frequently dropped connections? WireGuard itself is connectionless, btw. A "restart", on paper, should only happen when WireGuard configuration changes. It is worrying if you know for certain that restarts happen often.

Mostly by the status shown in the UI, but the reason why I started investigating it in the first place is I noticed when browsing, for example some forum, opening a thread page often takes 5+ or 10+ seconds, even that DNS logs are showing less than 5ms. On the other hand, sometimes it's instantly, so I thought WireGuard might be dropping a connection and reconnecting again, hence why a slow loading.

Unfortunately, I already deleted logs and reinstalled the app, as I will try setting it up again tomorrow. Log level wasn't changed, it was info. I remember there were six .txt files, the 6th being the biggest one, around 7MB.

Sounds terrible 😞 How many configurations?

Five or six, I think. Luckily, it wasn't too many, my plan was to add 15+. I understand this app is one of a kind, so I don't mind adding them again. :)

ignoramous commented 6 months ago

Sorry, could you please explain this a bit more detailed? I selected advanced mode in order to exclude some apps from going through the proxy connection, but seems like in that case the user has to use "Other DNS" which will slow down DNS requests (as mentioned on Mullvad website).

Yes.

In "Advanced" mode, the DNS upstream that resolves ALL queries is the user-selected DNS (DoH, DoT, ODoH, DNSCrypt etc). In "Simple" mode (you cannot exclude apps today: #1270), WireGuard's DNS is used.

What would be the best way to set up Rethink app in order to have an always on proxy connection, all apps (except a few) and DNS requests going through it?

You'll have to wait until we ship #1270.

If you're okay with DNS going to user-selected DNS (as opposed to WireGuard-configured DNS):

Use "Advanced" mode.
Enable "Lockdown" for the WireGuard profile.
Add apps as needed.

on Home tab, I went to "Proxy" and started importing WireGuard configs with QR code scanning.

So, only 0 or 1 WireGuard configurations prior to updating to v055c? Or, you had more but you deleted them? Or, they deleted on update?

for example some forum, opening a thread page often takes 5+ or 10+ seconds, even that DNS logs are showing less than 5ms. On the other hand, sometimes it's instantly, so I thought WireGuard might be dropping a connection and reconnecting again, hence why a slow loading.

Very much possible that we broke WireGuard. Will take a look: #1279

don't mind adding them again

You're kind. Thanks.

netizeni commented 6 months ago

So, only 0 or 1 WireGuard configurations prior to updating to v055c? Or, you had more but you deleted them? Or, they deleted on update?

On v055a I had 3 or 4 configs, but mostly used only one due to a slight annoyance to add apps again once config switched. Updating directly to v055c didn't work, as Obtainium showed some conflict message, so I used backup, uninstalled v055a and restored on a freshly installed v055c. Afterwards started adding configs with QR code scanning.

hussainmohd-a commented 6 months ago

Not 100% reproducible but reproduced twice by performing the below steps.

Upgrade the app from v055a to v055c
Restore the backup from v055a post upgrade
Try import/add multiple WireGuard config
Add some application to Orbot proxy
Edit WireGuard config, change the name and save

        android.database.sqlite.SQLiteConstraintException: UNIQUE constraint failed: ProxyApplicationMapping.uid, ProxyApplicationMapping.packageName, ProxyApplicationMapping.proxyId (code 1555 SQLITE_CONSTRAINT_PRIMARYKEY[1555])
    at android.database.sqlite.SQLiteConnection.nativeExecuteForChangedRowCount(Native Method)
    at android.database.sqlite.SQLiteConnection.executeForChangedRowCount(SQLiteConnection.java:1074)
    at android.database.sqlite.SQLiteSession.executeForChangedRowCount(SQLiteSession.java:756)
    at android.database.sqlite.SQLiteStatement.executeUpdateDelete(SQLiteStatement.java:66)
    at androidx.sqlite.db.framework.FrameworkSQLiteStatement.executeUpdateDelete(Unknown Source:2)
    at com.celzero.bravedns.database.ProxyApplicationMappingDAO_Impl.updateProxyForAllApps(Unknown Source:24)
    at com.celzero.bravedns.database.ProxyAppMappingRepository.updateProxyForAllApps(Unknown Source:2)
    at com.celzero.bravedns.service.ProxyManager.setProxyIdForAllApps(Unknown Source:117)
    at com.celzero.bravedns.service.WireguardManager.updateInterface(Unknown Source:237)
    at com.celzero.bravedns.service.WireguardManager.addOrUpdateInterface(Unknown Source:16)
    at com.celzero.bravedns.ui.activity.WgConfigEditorActivity.addWgInterface(Unknown Source:110)
    at com.celzero.bravedns.ui.activity.WgConfigEditorActivity.access$addWgInterface(Unknown Source:0)
    at com.celzero.bravedns.ui.activity.WgConfigEditorActivity$setupClickListeners$2$1.invokeSuspend(Unknown Source:51)
    at com.celzero.bravedns.ui.activity.WgConfigEditorActivity$setupClickListeners$2$1.invoke(SourceFile:0)
    at com.celzero.bravedns.ui.activity.WgConfigEditorActivity$setupClickListeners$2$1.invoke(SourceFile:0)
    at com.celzero.bravedns.ui.activity.WgConfigEditorActivity$io$1.invokeSuspend(Unknown Source:30)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(Unknown Source:11)
    at kotlinx.coroutines.DispatchedTask.run(Unknown Source:93)
    at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(Unknown Source:3)
    at kotlinx.coroutines.scheduling.TaskImpl.run(Unknown Source:2)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(Unknown Source:0)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(Unknown Source:14)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(Unknown Source:28)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(Unknown Source:0)

luckygitt commented 6 months ago

I also had very similar Wireguard problems when upgrading, notably all my confgs disappeared (and I also got error messages when trying to rename the tunnels but not the same as the OP. These problems seemed to disappear when I restarted the phone or restored the settings a second time (can't remember what exactly).

Regarding Wireguard Simple/Advanced (Lockdown and Alway On), I am none the wiser after reading Ignormaous' explanations! OK, Simple mode routes all (included apps) traffic, including DNS requests through the tunnel (did I get this right?) - that's what I would expect - why do we need more settings? Surely, anyone using a VPN tunnel doesn't want any traffic leaking outside of it? Shoudn't Lockdown be the default? Ditto regarding Always On - if the tunnel fails, shoudln't all traffic (or all apps part of the WireGuard configuration - included apps) fail with it?

To clarify, this is what I am trying to achieve - all (unblocked/allowed) Aps are routed through one Wireguard tunnel, also, DNS (in my case NexDNS). If the tunnel is down, fails or can't connect then nothing leaks/goes out to the internet.

Thanks, Lucky

hussainmohd-a commented 5 months ago

Not 100% reproducible but reproduced twice by performing the below steps.

Upgrade the app from v055a to v055c
Restore the backup from v055a post upgrade
Try import/add multiple WireGuard config
Add some application to Orbot proxy
Edit WireGuard config, change the name and save

        android.database.sqlite.SQLiteConstraintException: UNIQUE constraint failed: ProxyApplicationMapping.uid, ProxyApplicationMapping.packageName, ProxyApplicationMapping.proxyId (code 1555 SQLITE_CONSTRAINT_PRIMARYKEY[1555])
  at android.database.sqlite.SQLiteConnection.nativeExecuteForChangedRowCount(Native Method)
  at android.database.sqlite.SQLiteConnection.executeForChangedRowCount(SQLiteConnection.java:1074)
  at android.database.sqlite.SQLiteSession.executeForChangedRowCount(SQLiteSession.java:756)
  at android.database.sqlite.SQLiteStatement.executeUpdateDelete(SQLiteStatement.java:66)
  at androidx.sqlite.db.framework.FrameworkSQLiteStatement.executeUpdateDelete(Unknown Source:2)
  at com.celzero.bravedns.database.ProxyApplicationMappingDAO_Impl.updateProxyForAllApps(Unknown Source:24)
  at com.celzero.bravedns.database.ProxyAppMappingRepository.updateProxyForAllApps(Unknown Source:2)
  at com.celzero.bravedns.service.ProxyManager.setProxyIdForAllApps(Unknown Source:117)
  at com.celzero.bravedns.service.WireguardManager.updateInterface(Unknown Source:237)
  at com.celzero.bravedns.service.WireguardManager.addOrUpdateInterface(Unknown Source:16)
  at com.celzero.bravedns.ui.activity.WgConfigEditorActivity.addWgInterface(Unknown Source:110)
  at com.celzero.bravedns.ui.activity.WgConfigEditorActivity.access$addWgInterface(Unknown Source:0)
  at com.celzero.bravedns.ui.activity.WgConfigEditorActivity$setupClickListeners$2$1.invokeSuspend(Unknown Source:51)
  at com.celzero.bravedns.ui.activity.WgConfigEditorActivity$setupClickListeners$2$1.invoke(SourceFile:0)
  at com.celzero.bravedns.ui.activity.WgConfigEditorActivity$setupClickListeners$2$1.invoke(SourceFile:0)
  at com.celzero.bravedns.ui.activity.WgConfigEditorActivity$io$1.invokeSuspend(Unknown Source:30)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(Unknown Source:11)
  at kotlinx.coroutines.DispatchedTask.run(Unknown Source:93)
  at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(Unknown Source:3)
  at kotlinx.coroutines.scheduling.TaskImpl.run(Unknown Source:2)
  at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(Unknown Source:0)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(Unknown Source:14)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(Unknown Source:28)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(Unknown Source:0)

Fixed: https://github.com/celzero/rethink-app/issues/1311#issuecomment-2027094585

hussainmohd-a commented 4 months ago

Fix:https://github.com/celzero/rethink-app/commit/c4afb13da31f93046f36ea6c04f7e2b8ffebf3e4