[BUG] Unable to Sync Database stored on Google Drive

Ray-B commented 10 months ago

Checks

[X] I have read the FAQ section, searched the open issues, and still think this is a new bug.

Describe the bug you encountered:

Whenever you make a change to the local copy of the DB (e.g. add an entry) and save, the sync to the remote file on google drive fails, and only the local copy is updated with the change.

I've seen this bug crop up across multiple android devices (tablet and phone) when a KBDX file is stored/referenced on google drive.

Other Considerations of Note:

This behaviour appears to present itself on as many versions of the app that have bee released in the last year and on both android 12 and 13.
When tested, both devices that experience this problem had the Google Drive app installed.
Both devices reference the same KDBX file
Referencing a KBDX on google drive from a fresh app install, and then syncing appears to succeed, but subsequent closing and re-opening of the app results in sync failure.

Inspection of the log shows an exception on manual sync with a stack trace:

Important Note: I've obfuscated things, but its worth pointing out that this log entry did "leak" out my email address, google drive KBDX file name, and some other information. On a quick look, I am not sure if any of these are tokens or other important encoded values, so we should double check that too.

11/22/2023 11:12:08 PM:641 -- status message: Initializing...
11/22/2023 11:12:08 PM:642 -- status submessage: 
11/22/2023 11:12:08 PM:660 -- status message: Synchronizing cached database…
11/22/2023 11:12:08 PM:661 -- status submessage: Loading source file…
11/22/2023 11:12:08 PM:889 -- status submessage: Restoring source file…
11/22/2023 11:12:08 PM:890 -- gdrive://.U8-john.doe%[40gmail.com/.U8-kp-KP2A-1l0c8cfKdodmcmmdbSQellSAX-eIUuSu5/.U8-EXAMPLE-KEYPASS-DATABASE.kdbx-KP2A-10rny2SEiMSbLFMUpBQs9DTT6Iy0jwX5F](http://40gmail.com/.U8-kp-KP2A-1l0c8cfKdodmcmmdbSQellSAX-eIUuSu5/.U8-EXAMPLE-KEYPASS-DATABASE.kdbx-KP2A-10rny2SEiMSbLFMUpBQs9DTT6Iy0jwX5F) localVersionHash = 831C1552BD14FC35D1BDD57BDC37022B2E4FF0B070E593B7A61FEBA4F7F0D32611/22/2023 
11:12:09 PM:93 -- error getting file with for gdrive://.U8-john.doe%[40gmail.com/.U8-kp-KP2A-1l0c8cfKdodmcmmdbSQellSAX-eIUuSu5/.U8-EXAMPLE-KEYPASS-DATABASE.kdbx-KP2A-10rny2SEiMSbLFMUpBQs9DTT6Iy0jwX5F](http://40gmail.com/.U8-kp-KP2A-1l0c8cfKdodmcmmdbSQellSAX-eIUuSu5/.U8-EXAMPLE-KEYPASS-DATABASE.kdbx-KP2A-10rny2SEiMSbLFMUpBQs9DTT6Iy0jwX5F)
11/22/2023 11:12:09 PM:96 -- System.Exception: error getting file with for gdrive://.U8-john.doe%[40gmail.com/.U8-kp-KP2A-1l0c8cfKdodmcmmdbSQellSAX-eIUuSu5/.U8-EXAMPLE-KEYPASS-DATABASE.kdbx-KP2A-10rny2SEiMSbLFMUpBQs9DTT6Iy0jwX5F](http://40gmail.com/.U8-kp-KP2A-1l0c8cfKdodmcmmdbSQellSAX-eIUuSu5/.U8-EXAMPLE-KEYPASS-DATABASE.kdbx-KP2A-10rny2SEiMSbLFMUpBQs9DTT6Iy0jwX5F) ---> Java.IO.FileNotFoundException: error getting file with for gdrive://.U8-john.doe%[40gmail.com/.U8-kp-KP2A-1l0c8cfKdodmcmmdbSQellSAX-eIUuSu5/.U8-EXAMPLE-KEYPASS-DATABASE.kdbx-KP2A-10rny2SEiMSbLFMUpBQs9DTT6Iy0jwX5F](http://40gmail.com/.U8-kp-KP2A-1l0c8cfKdodmcmmdbSQellSAX-eIUuSu5/.U8-EXAMPLE-KEYPASS-DATABASE.kdbx-KP2A-10rny2SEiMSbLFMUpBQs9DTT6Iy0jwX5F)  at Java.Interop.JniEnvironment+InstanceMethods.CallVoidMethod (Java.Interop.JniObjectReference instance, Java.Interop.JniMethodInfo method, Java.Interop.JniArgumentValue* args) [0x0006e] in <6fb1725a77344b3e81b15cb69a959b5c>:0   at Java.Interop.JniPeerMembers+JniInstanceMethods.InvokeVirtualVoidMethod (System.String encodedMember, Java.Interop.IJavaPeerable self, Java.Interop.JniArgumentValue* parameters) [0x00036] in <6fb1725a77344b3e81b15cb69a959b5c>:0   at Keepass2android.Javafilestorage.GoogleDriveBaseFileStorage.UploadFile (System.String path, System.Byte[] data, System.Boolean writeTransactional) [0x00059] in <ef4fc53bed604ca380e5f749cd276ee6>:0   at keepass2android.Io.JavaFileStorage+JavaFileStorageWriteTransaction.CommitWrite () [0x0001c] in <483bd1ecfe364ef6a0dfb2f58ee9f0f5>:0    --- End of inner exception stack trace ---  at keepass2android.Io.JavaFileStorage+JavaFileStorageWriteTransaction.CommitWrite () [0x00036] in <483bd1ecfe364ef6a0dfb2f58ee9f0f5>:0   at keepass2android.Io.CachingFileStorage.UpdateRemoteFile (System.IO.Stream cachedData, KeePassLib.Serialization.IOConnectionInfo ioc, System.Boolean useFileTransaction, System.String hash) [0x00022] in <483bd1ecfe364ef6a0dfb2f58ee9f0f5>:0   at keepass2android.Io.CachingFileStorage.UpdateRemoteFile (KeePassLib.Serialization.IOConnectionInfo ioc, System.Boolean useFileTransaction) [0x00018] in <483bd1ecfe364ef6a0dfb2f58ee9f0f5>:0   at keepass2android.SynchronizeCachedDatabase.Run () [0x0009e] in <483bd1ecfe364ef6a0dfb2f58ee9f0f5>:0

Rampant Speculation:

The URI path is URL encoded. Probably nothing, but I have see bugs before where an encoded URL gets saved/passed and causes unexpected behaviour
"Error getting file with for gdrive" -> The path it logs looks funny. What is all the extra stuff appended? Is that a token-ized temporary path that is maybe no longer valid?

Not sure how busy contributors are - I don't have a dev env for this project set up - but if maintainers want help debugging or a PR, I am happy to do so, but it may take me some time as we have a new baby.

Thanks!

Describe what you expected to happen:

When attempting to reconcile/synchronise a remotely stored google drive KBDX file, it should succeed.

What version of Keepass2Android are you using?

1.09e-r7

Which version of Android are you on?

13

abend0c1 commented 9 months ago

I'm getting this issue too (just today). Same 1.09e-r7 version of the Android app. Also getting similar error using Keypass Password Safe Version 2.47 (64-bit) on Linux (Ubuntu 22-04.3LTS). On Linux I can list my gdrive keypass folder (and my other folders) but I notice that it does not contain a .kdbx file. Instead it contains two files called <33 gibberish characters>.tmp and one called <33 gibberish characters>. My guess is that something has accidentally deleted the .kdbx file and that these *.tmp files are the most recent backups (one is from today Jan 6, the other is a little older Jan 2). The one without the tmp extension is empty (0 bytes). Edit: If I rename one of the tmp files to my original kdbx file name then keypass can open the file OK, but when I save it the result is that the kdbx file disappears and a new tmp file contains the latest update. Seems to be some kind of GDrive API malfunction.

Ray-B commented 9 months ago

I'm getting this issue too (just today). Same 1.09e-r7 version of the Android app. Also getting similar error using Keypass Password Safe Version 2.47 (64-bit) on Linux (Ubuntu 22-04.3LTS). On Linux I can list my gdrive keypass folder (and my other folders) but I notice that it does not contain a .kdbx file. Instead it contains two files called <33 gibberish characters>.tmp and one called <33 gibberish characters>. My guess is that something has accidentally deleted the .kdbx file and that these *.tmp files are the most recent backups (one is from today Jan 6, the other is a little older Jan 2). The one without the tmp extension is empty (0 bytes). Edit: If I rename one of the tmp files to my original kdbx file name then keypass can open the file OK, but when I save it the result is that the kdbx file disappears and a new tmp file contains the latest update. Seems to be some kind of GDrive API malfunction.

@abend0c1

Might not be the same issue, however reading in the issues list, gdrive sync has been a pain for a while.

Can you turn on logging in keypass2android, then try to replicate the issue you're having and paste the logs here.

This will make it easier to tell if your issue is related, or completely different and requires a separate issue to be created.

Also post your android version and app version.

PhilippC commented 9 months ago

@Ray-B if you can collect a logcat, there might be a bit more information right before the logged error due to the e.printStackTrace();

try {
            fl = getDriveService(getAccount()).files().get(getGDriveId()).execute();
        } catch (Exception e) {
            e.printStackTrace();
            throw new FileNotFoundException("error getting file with for "+ this.getFullPath());
        }

The path it logs looks funny. What is all the extra stuff appended? Is that a token-ized temporary path that is maybe no longer valid?

Yes, I'm encoding GDrive-IDs in the path. The problem is that GDrive "looks like" a drive with a folder structure, but underneath, each file is referenced by an ID and the path is not even unique. I don't know a good way of dealing with this (considering cases like renamed files etc.)

Did you try to re-open the file?

abend0c1 commented 9 months ago

Hi Philipp I created a Junk entry in the Android app and then clicked the "Save" database icon and it responded with a "Could not save the source file..." message which was too quick to copy down completely. The trace log is as follows:

9/01/2024 4:52:32 PM:893 -- DatabaseSettingsActivity.OnPause 43 9/01/2024 4:52:32 PM:895 -- Timeout start 9/01/2024 4:52:32 PM:910 -- GroupActivity.OnStart 42 9/01/2024 4:52:32 PM:911 -- GroupActivity.OnResume 42 9/01/2024 4:52:32 PM:911 -- DatabaseIsUnlocked=True 42 9/01/2024 4:52:32 PM:911 -- Timeout cancel 9/01/2024 4:52:33 PM:281 -- DatabaseSettingsActivity.OnStop 43 9/01/2024 4:52:33 PM:284 -- DatabaseSettingsActivity.OnDestroyTrue 43 9/01/2024 4:52:41 PM:547 -- GroupActivity.OnPause 42 9/01/2024 4:52:41 PM:548 -- Timeout start 9/01/2024 4:52:41 PM:570 -- GroupActivity.OnCreate 44 9/01/2024 4:52:41 PM:571 -- GroupActivity:apptask= 44 9/01/2024 4:52:41 PM:574 -- Loaded task keepass2android.NullTask 9/01/2024 4:52:41 PM:575 -- Task in activity GroupActivity 44 changed to NullTask 9/01/2024 4:52:41 PM:606 -- GroupActivity.OnStart 44 9/01/2024 4:52:41 PM:607 -- GroupActivity.OnResume 44 9/01/2024 4:52:41 PM:607 -- DatabaseIsUnlocked=True 44 9/01/2024 4:52:41 PM:607 -- Timeout cancel 9/01/2024 4:52:42 PM:138 -- GroupActivity.OnStop 42 9/01/2024 4:52:47 PM:302 -- parsing autofillStructure... 9/01/2024 4:52:47 PM:302 -- Parsing done 9/01/2024 4:52:47 PM:303 -- cannot autofill 9/01/2024 4:52:57 PM:778 -- GroupActivity.OnPause 44 9/01/2024 4:52:57 PM:778 -- Timeout start 9/01/2024 4:52:57 PM:796 -- LockingActivity: OnActivityResult 9/01/2024 4:52:57 PM:797 -- AppTask keepass2android.NullTask in OnActivityResult 9/01/2024 4:52:57 PM:797 -- Task in activity GroupActivity 42 changed to NullTask 9/01/2024 4:52:57 PM:799 -- GroupActivity.OnStart 42 9/01/2024 4:52:57 PM:799 -- GroupActivity.OnResume 42 9/01/2024 4:52:57 PM:800 -- DatabaseIsUnlocked=True 42 9/01/2024 4:52:57 PM:800 -- Timeout cancel 9/01/2024 4:52:57 PM:878 -- Populating group view False 9/01/2024 4:52:58 PM:237 -- GroupActivity.OnStop 44 9/01/2024 4:52:58 PM:247 -- GroupActivity.OnDestroyTrue 44 9/01/2024 4:53:04 PM:676 -- GroupActivity.OnPause 42 9/01/2024 4:53:04 PM:677 -- Timeout start 9/01/2024 4:53:04 PM:699 -- EntryEditActivity.OnCreate 45 9/01/2024 4:53:04 PM:700 -- EntryEditActivity:apptask= 45 9/01/2024 4:53:04 PM:721 -- Loaded task keepass2android.NullTask 9/01/2024 4:53:04 PM:721 -- Task in activity EntryEditActivity 45 changed to NullTask 9/01/2024 4:53:04 PM:774 -- EntryEditActivity.OnStart 45 9/01/2024 4:53:04 PM:774 -- EntryEditActivity.OnResume 45 9/01/2024 4:53:04 PM:775 -- DatabaseIsUnlocked=True 45 9/01/2024 4:53:04 PM:775 -- Timeout cancel 9/01/2024 4:53:05 PM:247 -- GroupActivity.OnStop 42 9/01/2024 4:53:06 PM:911 -- parsing autofillStructure... 9/01/2024 4:53:06 PM:911 -- Parsing done 9/01/2024 4:53:06 PM:912 -- cannot autofill 9/01/2024 4:53:22 PM:262 -- status message: Initializing... 9/01/2024 4:53:22 PM:263 -- status submessage: 9/01/2024 4:53:22 PM:313 -- status message: Adding entry… 9/01/2024 4:53:22 PM:315 -- status message: Saving database… 9/01/2024 4:53:22 PM:317 -- status submessage: Checking target file for changes… 9/01/2024 4:53:23 PM:597 -- status submessage:

Then I deleted the Junk entry and clicked the "Synchronise database..." option and it gave the same error. The trace log for that is as follows:

9/01/2024 4:58:00 PM:274 -- DatabaseSettingsActivity.OnPause 51 9/01/2024 4:58:00 PM:275 -- Timeout start 9/01/2024 4:58:00 PM:299 -- GroupActivity.OnStart 49 9/01/2024 4:58:00 PM:300 -- GroupActivity.OnResume 49 9/01/2024 4:58:00 PM:301 -- DatabaseIsUnlocked=True 49 9/01/2024 4:58:00 PM:301 -- Timeout cancel 9/01/2024 4:58:00 PM:663 -- DatabaseSettingsActivity.OnStop 51 9/01/2024 4:58:00 PM:665 -- DatabaseSettingsActivity.OnDestroyTrue 51 9/01/2024 4:58:03 PM:217 -- status message: Initializing... 9/01/2024 4:58:03 PM:219 -- status submessage: 9/01/2024 4:58:03 PM:256 -- status message: Synchronizing cached database… 9/01/2024 4:58:03 PM:258 -- status submessage: Loading source file… 9/01/2024 4:58:03 PM:846 -- status submessage: Restoring source file… 9/01/2024 4:58:03 PM:852 -- gdrive://.U8-androidarmstrong% 40gmail.com/.U8-Keepass-KP2A-1V6nCKWiMAaB1dlLR514GsKCNh9WBIRvS/.U8-1M6Uag5PdvPktRugw4Godm3633kX4DdxC-KP2A-1w2e0CvGJihTpn9JQ4u47GEudwoDTkFWU localVersionHash = 24E50A79B59BEEC10AD56147346846ED1703B92D5543FA8EB14053443323A77C 9/01/2024 4:58:04 PM:267 -- error getting file with for gdrive://.U8-androidarmstrong% 40gmail.com/.U8-Keepass-KP2A-1V6nCKWiMAaB1dlLR514GsKCNh9WBIRvS/.U8-1M6Uag5PdvPktRugw4Godm3633kX4DdxC-KP2A-1w2e0CvGJihTpn9JQ4u47GEudwoDTkFWU 9/01/2024 4:58:04 PM:272 -- System.Exception: error getting file with for gdrive://.U8-androidarmstrong% 40gmail.com/.U8-Keepass-KP2A-1V6nCKWiMAaB1dlLR514GsKCNh9WBIRvS/.U8-1M6Uag5PdvPktRugw4Godm3633kX4DdxC-KP2A-1w2e0CvGJihTpn9JQ4u47GEudwoDTkFWU ---> Java.IO.FileNotFoundException: error getting file with for gdrive://.U8-androidarmstrong% 40gmail.com/.U8-Keepass-KP2A-1V6nCKWiMAaB1dlLR514GsKCNh9WBIRvS/.U8-1M6Uag5PdvPktRugw4Godm3633kX4DdxC-KP2A-1w2e0CvGJihTpn9JQ4u47GEudwoDTkFWU at Java.Interop.JniEnvironment+InstanceMethods.CallVoidMethod (Java.Interop.JniObjectReference instance, Java.Interop.JniMethodInfo method, Java.Interop.JniArgumentValue* args) [0x0006e] in

<6fb1725a77344b3e81b15cb69a959b5c>:0 at Java.Interop.JniPeerMembers+JniInstanceMethods.InvokeVirtualVoidMethod (System.String encodedMember, Java.Interop.IJavaPeerable self, Java.Interop.JniArgumentValue* parameters) [0x00036] in <6fb1725a77344b3e81b15cb69a959b5c>:0 at Keepass2android.Javafilestorage.GoogleDriveBaseFileStorage.UploadFile (System.String path, System.Byte[] data, System.Boolean writeTransactional) [0x00059] in :0 at keepass2android.Io.JavaFileStorage+JavaFileStorageWriteTransaction.CommitWrite () [0x0001c] in <483bd1ecfe364ef6a0dfb2f58ee9f0f5>:0 --- End of inner exception stack trace --- at keepass2android.Io.JavaFileStorage+JavaFileStorageWriteTransaction.CommitWrite () [0x00036] in <483bd1ecfe364ef6a0dfb2f58ee9f0f5>:0 at keepass2android.Io.CachingFileStorage.UpdateRemoteFile (System.IO.Stream cachedData, KeePassLib.Serialization.IOConnectionInfo ioc, System.Boolean useFileTransaction, System.String hash) [0x00022] in <483bd1ecfe364ef6a0dfb2f58ee9f0f5>:0 at keepass2android.Io.CachingFileStorage.UpdateRemoteFile (KeePassLib.Serialization.IOConnectionInfo ioc, System.Boolean useFileTransaction) [0x00018] in <483bd1ecfe364ef6a0dfb2f58ee9f0f5>:0 at keepass2android.SynchronizeCachedDatabase.Run () [0x0009e] in <483bd1ecfe364ef6a0dfb2f58ee9f0f5>:0 9/01/2024 4:58:04 PM:321 -- Populating group view False 9/01/2024 4:58:33 PM:993 -- GroupActivity.OnPause 49 9/01/2024 4:58:33 PM:994 -- Timeout start This time, the kdbx file has remained intact on gdrive (dated Jan 6) and no new tmp files are being created, so I have not completely reproduced the issue (the kdbx file disappeared last time and I had to manually recover it from a tmp file). Today the kdbx file is simply not being updated on gdrive (but my cached copy on Android is being updated ok). Cheers Andrew On Tue, 9 Jan 2024 at 10:58, Raymond ***@***.***> wrote: > I'm getting this issue too (just today). Same 1.09e-r7 version of the > Android app. Also getting similar error using Keypass Password Safe Version > 2.47 (64-bit) on Linux (Ubuntu 22-04.3LTS). On Linux I can list my gdrive > keypass folder (and my other folders) but I notice that it does not contain > a *.kdbx file. Instead it contains two files called <33 gibberish > characters>.tmp and one called <33 gibberish characters>. My guess is that > something has accidentally deleted the *.kdbx file and that these *.tmp > files are the most recent backups (one is from today Jan 6, the other is a > little older Jan 2). The one without the tmp extension is empty (0 bytes). > Edit: If I rename one of the tmp files to my original kdbx file name then > keypass can open the file OK, but when I save it the result is that the > kdbx file disappears and a new tmp file contains the latest update. Seems > to be some kind of GDrive API malfunction. > > @abend0c1 > > Might not be the same issue, however reading in the issues list, gdrive > sync has been a pain for a while. > > Can you turn on logging in keypass2android, then try to replicate the > issue you're having and paste the logs here. > > This will make it easier to tell if your issue is related, or completely > different and requires a separate issue to be created. > > — > Reply to this email directly, view it on GitHub > , > or unsubscribe > > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

abend0c1 commented 9 months ago

I did the same today on two different Android devices and now it all works perfectly. Adding, deleting and syncing work as expected. App version is 1.09e-r7. Android version 11 on phone, and 13 on tablet. Sorry for hijacking this issue :/

PhilippC / keepass2android