search

keepassium / KeePassium

KeePass-compatible password manager for iOS
https://keepassium.com
Other
1.21k stars 103 forks source link

Error: pCloud / OneDrive does not respond (on iPhone) #248

Closed chriswayg closed 2 years ago

chriswayg commented 2 years ago

First of all, I want to thank the developer for the excellent app. KeePassium makes it easy to switch from 1Password and the price is reasonable.

Description I am using pCloud to sync my KeePass database, but I get the error in KeePassium: "pCloud does not respond " or "database is unreachable".

The pCloud app has a folder MyData which contains the KeePass databae. This folder is set to "Make Available Offline".

There is no problem with Internet connectivity.

How to reproduce Steps to reproduce the behavior:

Expected behavior

Environment:

Screenshots pCloud - does not respond

keepassium commented 2 years ago

Thank you. To be clear, this is not KeePassium's issue. KeePassium asks for the database (in a standardized way, same for all clouds), but does not receive the file and thus shows the error. There is nothing I can do about it on KeePassium's side.

That said, the underlying error returned by the system can sometimes clarify what is going on. Can you attach a diagnostic log, please? (it is in KeePassium settings → Diagnostic Log)

chriswayg commented 2 years ago

I have now tried almost all the sync sources with "Full Support" and I am encountering database sync issues with almost all of them. DropBox seemed to work best, but is limited to 3 devices.

Most others allow 5 devices and pCloud is my favorite due to being based in Switzerland. My pCloud folder being set to "Make Available Offline" actually triggered the above issue initially, but even after changing this setting I still encountered sync problems.

I also tested OneDrive and I get "Couldn't communicate with a helper application" and "OneDrive does not respond" after about a day. I sent an email with the Diagnostic Log referencing this GitHub issue number, but I doubt that it provides much additional insight.

I have been testing KeePassium, Strongbox on macOS andiOS (where I did not encounter this issue yet), KeePassXC (on macOS & Windows), KeeWeb and KeePass2Android for a team implementation. After additional reading regarding sync issues from the support sections of these applications I now suspect that the cause could be KeePassXC which by default saves the database file using "safe saving".

I now changed the following in KeePassXC:

Settings > File Management:
    Enable: "Use Alternative Saving method" and
    Select: "Directly write to database file (dangerous)" in the dropdown.

"The reference to the database becomes invalid because of the way these Desktop clients save the database. They create a new temporary file and swap it in/rename it instead of updating it in place. This is sometimes known as Safe Saving ..." (from Strongbox troubleshooting)

Safe saving uses a temporary file in the same directory as your current database which is closed and written prior to deleting your existing db file and then renamed to your filename. This makes it a transaction that prevents data loss 99.9% of the time. - Unsafe saves use the temp directory to make the new database. Then your old db is deleted and the temp db moved in its place. (from the developer of KeePassXC) What "Safely save database files" means? · Issue #1724 · keepassxreboot/keepassxc · GitHub

Since the problems have been intermittent, I will need to test this for a few days (probably via pCloud) before I can tell for sure.

@keepassium Could you please comment, if you test for interoperability and compatibility of KeePassium with the above mentioned apps and possibly others?

Edit: I sent an additional email with the "pCloud does not respond" logs. Found no way to fix pCloud. - Will try again with OneDrive. I need to get one of these working reliably.

keepassium commented 2 years ago

I sent an email with the Diagnostic Log referencing this GitHub issue number, but I doubt that it provides much additional insight.

KeePassium's "... does not respond" is an umbrella message shown in several cases:

  1. Storage provider did not respond before the timeout (10 seconds by default)
  2. Storage provider's code could not be contacted by the system (system error code xpcConnectionInvalid)
  3. Storage provider failed to respond (system error code xpcConnectionInterrupted)
  4. Storage provider returned nonsense (system error code xpcConnectionReplyInvalid)

Here's the relevant part of your log:

11.030 (E) DatabaseLoader.swift:307 onDatabaseURLResolved(url:fileProvider:) Failed to open database document [error: pCloud does not respond.]

The leading number is the time since DB loading started (just above 10 s). There are no system errors in the log. That is, we are talking about pCloud timeout, case number 1.

This is a tough one, though… The response could be delayed for a variety of reasons:

I also tested OneDrive and I get "Couldn't communicate with a helper application" and "OneDrive does not respond" after about a day.

"Could not communicate with a helper application" is the system's description for all xpc* errors (numbers 2 to 4 above). They usually mean that OneDrive's module responsible for integration with Files (and other apps) has crashed. The most common reason is that it ran out of memory. (These background processes have very tight memory limits.) This, in turn, can happen if it has to work with a large file or if there is a queue of large files to upload (after a quick sequence of saves in KeePassium).

Does either "database over 2-3 MB" or "unreachable SMB share" fit in your case?

keepassium commented 2 years ago

@keepassium Could you please comment, if you test for interoperability and compatibility of KeePassium with the above mentioned apps and possibly others?

Initially I tested interoperability with KeePass and KeePassXC. Now I routinely test it with KeePassXC.

That said, the interoperability with a wide spectrum of other KeePass-compatible apps and libraries is routinely tested by the end users. If they report an issue and it happens to point to the other app, I report it:

chriswayg commented 2 years ago

@keepassium

Does either "database over 2-3 MB" or "unreachable SMB share" fit in your case?

It is a 6.7MB database with 1360 entries collected over many years. I did not re-import any binary attachments ('documents') from 1Password, as that would have brought the database to almost 50MB. I only use around 20 attachments with about 5KBytes each. I am not sure if there is any way to further compress it to reach anywhere near 3MB, apart from time-consuming pruning to test expired accounts one-by-one.

pCloud appeared to be very sluggish even inside their own iOS app. I suspect routing issues with the Philippines and possibly throttling of free accounts. This could explain the lack of reliable availability.

OneDrive seems to perform well now after I disabled "safe saving" in KeePassXC. Therefore I will likely stay with OneDrive.

(Generally internet quality is substandard in the Philippines, as even the nominal 50 to 100 Mbit speeds of our fiber connections do not provide stability or reliability. We experience vast shifts in bandwidth during peak hours, fluctuating ping times and unpredictable routing issues due to ISP provider non-interconnection. This is especially difficult for gamers, but I also encounter issues during testing of apps I develop. This probably could not be simulated easily in Germany where you are testing.)

For now I will increase the database timeout to 15 seconds to be able to deal with unreliable internet conditions better.

As for interoperability with KeePassXC, this seems to work generally well, after I enabled the 'Alternative Saving method'. I previously encountered duplicate database files which were probably due to this settingas well. (With Strongbox (macOS version) I encountered more problematic interoperability issues: corrupted (unfixable) Tags and disappearing TOTP after using their .PIF import method.)

Another feature I am concerned about is interoperability with KeeShare, which we are using for sharing a Group folder of items within the team. This works well among KeePassXC clients, but KeePassium does not handle this correctly after the initial sync. - I will post a separate issue, as using KeePassium leads to inconsistent data between users.

keepassium commented 2 years ago

It is a 6.7MB database with 1360 entries collected over many years. [...] I only use around 20 attachments with about 5KBytes each.

I would say all of this should have fit in 1.5 MB, but probably this depends on the content. In either case, 6.7 MB is large enough to take over 10 seconds to download.

OneDrive seems to perform well now after I disabled "safe saving" in KeePassXC. Therefore I will likely stay with OneDrive.

This is just a coincidence. "Safe saving" is known to affect pCloud, but not OneDrive.

For a 7MB database in OneDrive, there is an easy way to reproduce the "Could not communicate with a helper application":

If you give OneDrive the time to finish uploading the database, it won't crash on successive save. I can see how this becomes problematic with a 7MB file, though…

This probably could not be simulated easily in Germany where you are testing.

It's Luxembourg, not Germany. iOS has an in-built network limiter, which helps debugging some edge cases. But I don't see much point in testing third-party sync apps with a poor network connection: the results are rather predictable.

If you consider this solved, feel free to close.

chriswayg commented 2 years ago

You're correct., the database file should have been much smaller. I discovered that some attached document scans which I had deleted were still persisted in history. After some database maintenance & cleanup, I got it down to about 1.7MB. It does load faster when it works, but the underlying issue with "OneDrive does not respond" still recurs every day.

For me this bug is a showstopper and I do not consider it solved as it can take quite a lot of effort just to restore access to my database. Once the message "OneDrive does not respond" occurs, it is unpredictable and time consuming to get it to load the same database again.

I tried restarting KeePassium, restarting the iPhone, switching from Wifi (fiber) to LTE and back. I also opened OneDrive on the device and checked that it was synced and then switched back to KeePassium, which still showed the same error. There is no specific sequence of steps that will predictably restore access to the database in a quick manner. It is not just a one time error, but it persists for possibly half an hour until it suddenly works again (or not). - I was forced to remove the database from the KeePassium list multiple times and re-add it, when the database remained inaccessible for longer.

In summary:

When looking at https://account.live.com/consent/Manage I notice that two other apps I am testing: KeePass2Android and StrongBox have requested special permissions to directly access OneDrive from Android and iOS, whereas KeePassium is apparently using a different approach to access OneDrive. Either OneDrive is somehow misbehaving on iOS, or the way KeePassium is accessing the OneDrive cloud filesystem is not optimal.

Currently I cannot switch our team from 1Password to KeePassium until this is working more reliably.

KeePassium to OneDrive Log - Pastebin.com

chriswayg commented 2 years ago

Additional info - next day: On another iPhone (iPhone 11 with iOS 15.6) KeePassium is syncing without any issues using OneDrive with a different OneDrive account. The only other difference is that the iPhone 11 has FaceID enabled, whereas the iPhone XS Max has FaceID disabled. Since the issue appears after successful authentication, I do not see how this could influence the sync behavior.

I also noticed that both phones still showed yesterdays database date, as KeePassium did not refresh the database automatically, even though OneDrive (the app) had the updated file available. I think you mentioned somewhere that not refreshing the db automatically when loading KeePassium is by design, but I have to wonder if opening a database that is 10+ hours out of sync will not make sync conflicts more likely. Maybe I missed a setting where this can be changed.

Manually refreshing the database by sliding down the screen worked immediately on the iPhone 11, but took about 10 attempts and 5 minutes on the iPhone XS Max. No errors were displayed and I could find nothing relevant in the logs. Overall a frustrating experience on one device while ok on the other.

keepassium commented 2 years ago

For me this bug is a showstopper and I do not consider it solved

As I mentioned above, KeePassium is only a messenger here, showing you the error from the system. The bug happens outside of KeePassium and there is not much I can do about it.

I tried restarting KeePassium, restarting the iPhone, switching from Wifi (fiber) to LTE and back.

Try to reinstall OneDrive instead. (If you have several Microsoft apps installed, you might also need to reset OneDrive first in device settings → OneDrive → Clear Account Settings.)

I also opened OneDrive on the device and checked that it was synced and then switched back to KeePassium, which still showed the same error.

Cloud's integration with other apps (such as Files or KeePassium) is handled by a separate module (file provider) in OneDrive's installation. It runs in a separate background process, independently from OneDrive's main app. So it can be that integration with other apps is broken, but OneDrive app works just fine.

It is not just a one time error, but it persists for possibly half an hour until it suddenly works again (or not).

Hypothetically, this is when a system's watchdog timer restarts the crashed file provider module.

--

I notice that two other apps I am testing: KeePass2Android and StrongBox have requested special permissions to directly access OneDrive from Android and iOS, whereas KeePassium is apparently using a different approach to access OneDrive.

Of course is uses a different approach, that's the whole point! KeePassium does not have any special permission to cloud storage, it never asks you to log into your OneDrive account. Actually, it stays offline: How KeePassium stays offline and works with online storage

This ensures an important separation of concerns:

As a result of this separation, the user does not have to trust either of the parties.

after trying for multiple minutes to connect to my database via KeePassium, I launched Strongbox (iOS) on the same device and it synced with the same KeePass databse via OneDrive without issue.

Strongbox asks you to log into your OneDrive account and then connects to Microsoft servers directly. So of course it does not care whether OneDrive's file provider module is responsive or not. This ensures a more predictable connection, but has serious consequences in terms of privacy and security:

I also noticed that both phones still showed yesterdays database date, as KeePassium did not refresh the database automatically, even though OneDrive (the app) had the updated file available.

It does refresh file dates automatically. But, again, KeePassium uses OneDrive's file provider module, which runs independently from OneDrive main app. A more correct comparison would be to compare file dates in KeePassium and in the Files app.

Either OneDrive is somehow misbehaving on iOS, or the way KeePassium is accessing the OneDrive cloud filesystem is not optimal.

KeePassium uses the standard Apple-recommended approach to work with external storage. The approach and the code is the same for all storage, be it pCloud, OneDrive or even "On My iPhone". This approach works well for most users (otherwise we would not be here).

But yes, this approach depends on cloud provider's code which has its own quirks. When OneDrive/pCloud misbehaves, I can only explain why (see above), document most common issues and recommend ad-hoc solutions (see above). This was unexpected, definitely unwanted, but that's the price of that separation of concerns…

Currently I cannot switch our team from 1Password to KeePassium until this is working more reliably.

No problem, it's good there is a choice. This is less relevant for you, but I made synchronization fully functional in the free version exactly for this reason, so that everyone can test how it works in their specific environment.

craigo- commented 2 years ago

Would it be fair to say, then, that we as users might be better off directing our energies reporting these sorts of bugs to the cloud app provider (e.g. Microsoft, pcloud.com)?

And then if there is an upstream problem with the app’s integration with iOS, I’m sure they would have a much better chance of success getting Apple’s attention than we would.

keepassium commented 2 years ago

Would it be fair to say, then, that we as users might be better off directing our energies reporting these sorts of bugs to the cloud app provider (e.g. Microsoft, pcloud.com)?

@craigo- , yes. Though these might take months and years to get noticed, and by that time the issue might disappear by itself in the course of normal app evolution (case in point).

chriswayg commented 2 years ago

This ensures a more predictable connection, but has serious consequences in terms of privacy and security:

  • no separation of concerns: your password manager has full access to your OneDrive. ...

@keepassium thank you for your explanation of "separation of concerns". This is certainly an important issue and I appreciate that you have given thought to that. It is quite concerning with regards to privacy when OneDrive is also used for other files. The permissions given to Strongbox (and others) are very broad, as they basically give full control over all files and not just access to an app folder.

I am currently testing an instance of Nextcloud on Vultr to hopefully solve the availability and reliability issues of these other cloud services and to possibly improve privacy. At least, I will have more detailed technical control, if something does not work. Whether any VPS in a Five Eyes nation provides sufficient privacy protections is quite doubtful though.

The super-convenient 1Password subscription for a small team worked well for years in spite of our somewhat unstable internet, but I have serious privacy concerns having to trust one Canadian company with closed source security software having theoretical access to many critical credentials (I know they claim to have no access to my key/passphrase). Also their decision to abandon local vaults in version 8 is the last straw for me (its all in the Cloud from now on). (See for example the CLOUD Act: US, Canada to figure out rules on cops and Feds accessing people's data across borders • The Register.)

I will test this with Nextcloud for a few days and close the issue here, if it does not recur with Nextcloud.

keepassium commented 2 years ago

I will test this with Nextcloud for a few days and close the issue here, if it does not recur with Nextcloud.

Please avoid Nextcloud for iOS for sync. You won't experience this issue, but you will definitely encounter one of Nextcloud's other known issues. Some of them silent-data-loss level serious (Nextcloud might randomly overwrite remote files with their outdated cached versions).

Nextcloud issues are serious enough that I had to add WebDAV support directly to KeePassium (#247), relaxing the "strictly offline" stance. This was the only way to ensure reliable sync with Nextcloud/ownCloud/Seafile/Synology/QNAP. Currently available in beta, scheduled for release in a couple of weeks.

chriswayg commented 2 years ago

Please avoid Nextcloud for iOS for sync. You won't experience this issue, but you will definitely encounter one of Nextcloud's other known issues. Some of them silent-data-loss level serious (Nextcloud might randomly overwrite remote files with their outdated cached versions).

@keepassium Thanks for the heads-up about issues with Nextcloud. After reading about all the issues, I will certainly not use Nextcloud without Webdav.

Great that you are implementing Webdav support. I installed the KeePassium Beta (v1.36.115) and I am using it with Nextcloud 24.0.4 on a minimal 1GB Linode VPS in Japan with a ping of 75 to 100 ms depending on chosen ISP. Overall, its quite snappy and works online as well as offline.

I see you mention ownCloud/Seafile/Synology/QNAP. Are any of these less buggy or more recommended than Nexcloud? I remember testing Seafile in the past.

reinstall OneDrive instead. (If you have several Microsoft apps installed, you might also need to reset OneDrive first in device settings → OneDrive → Clear Account Settings.)

I have now reinstalled OneDrive as well and noticed some really buggy behavior before resetting the account. After clearing the account settings, it actually seems to perform well. I will be testing it for a few more days to see if it stays in sync.

Initial testing did not cause the same error "OneDrive does not respond", but the database was 3 hours out of sync and refused to refresh even after pulling down the KeePassium databases screen multiple times. It only refreshed after switching back and forth to the OneDrive app.

keepassium commented 2 years ago

I see you mention ownCloud/Seafile/Synology/QNAP. Are any of these less buggy or more recommended than Nexcloud? I remember testing Seafile in the past.

I cannot attest regarding their comparative "bugginess" (wow, that's actually a word! :) I mentioned them because all these apps are similar to Nextcloud, as they 1) struggle to sync in background and 2) use WebDAV to communicate with the server.

keepassium commented 2 years ago

Closing, as this is not a KeePassium's issue.