jstaf / onedriver

A native Linux filesystem for Microsoft OneDrive
GNU General Public License v3.0
1.86k stars 91 forks source link

Some files not updating or becoming corrupted #350

Closed pmoris closed 8 months ago

pmoris commented 10 months ago

Today I noticed that a few of my files were not updating properly and that their native software warned about potential file corruption.

For troubleshooting, I downloaded the respective files from the web version and also created a new mount point.

  1. Keepass .kdbx file:
    • I had a similar problem a few weeks ago (see https://github.com/jstaf/onedriver/issues/217#issuecomment-1669046058).
    • The file's metadata seems to be identical to the version online as well as the one in the new mount point, but the binary file contents differ according to diff.
    • The first time I attempted to open the file, keepass complained that the supplied password was wrong or that the file had been corrupted, but it did open successfully the second time. However, some of the password entries were clearly outdated (they had been updated on a Windows machine).
    • When opening the file, the log contains multiple entries (I think it tries to sync both when opening the file manually and when keepassXC tries to unlock it):
      Aug 21 09:19:58 fedora onedriver[12714]: 09:19:58 INF Not using cached item due to file hash mismatch, fetching content from API. id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Open path=/config/keepass.kdbx
      Aug 21 09:19:58 fedora onedriver[12714]: 09:19:58 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Fsync path=/config/keepass.kdbx
      Aug 21 09:20:21 fedora onedriver[12714]: 09:20:21 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Open path=/config/keepass.kdbx
      Aug 21 09:20:21 fedora onedriver[12714]: 09:20:21 INF Not using cached item due to file hash mismatch, fetching content from API. id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Open path=/config/keepass.kdbx
      Aug 21 09:20:23 fedora onedriver[12714]: 09:20:23 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Fsync path=/config/keepass.kdbx
      Aug 21 09:20:35 fedora onedriver[12714]: 09:20:35 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Open path=/config/keepass.kdbx
      Aug 21 09:20:35 fedora onedriver[12714]: 09:20:35 INF Not using cached item due to file hash mismatch, fetching content from API. id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Open path=/config/keepass.kdbx
      Aug 21 09:20:37 fedora onedriver[12714]: 09:20:37 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Fsync path=/config/keepass.kdbx
      Aug 21 09:20:37 fedora onedriver[12714]: 09:20:37 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Open path=/config/keepass.kdbx
      Aug 21 09:20:37 fedora onedriver[12714]: 09:20:37 INF Not using cached item due to file hash mismatch, fetching content from API. id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Open path=/config/keepass.kdbx
      Aug 21 09:20:37 fedora onedriver[12714]: 09:20:37 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=31 op=Fsync path=/config/keepass.kdbx
    • Opening the same file in the new mount point results in:
      Aug 21 09:37:13 fedora onedriver[12836]: 09:37:13 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=44 op=Open path=/config/keepass.kdbx
      Aug 21 09:37:13 fedora onedriver[12836]: 09:37:13 INF Found content in cache. id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=44 op=Open path=/config/keepass.kdbx
      Aug 21 09:37:13 fedora onedriver[12836]: 09:37:13 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=44 op=Fsync path=/config/keepass.kdbx
      Aug 21 09:37:43 fedora onedriver[12836]: 09:37:43 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=44 op=Open path=/config/keepass.kdbx
      Aug 21 09:37:43 fedora onedriver[12836]: 09:37:43 INF Found content in cache. id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=44 op=Open path=/config/keepass.kdbx
      Aug 21 09:37:43 fedora onedriver[12836]: 09:37:43 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=44 op=Fsync path=/config/keepass.kdbx
      Aug 21 09:38:13 fedora onedriver[12836]: 09:38:13 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=44 op=Open path=/config/keepass.kdbx
      Aug 21 09:38:13 fedora onedriver[12836]: 09:38:13 INF Found content in cache. id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=44 op=Open path=/config/keepass.kdbx
      Aug 21 09:38:13 fedora onedriver[12836]: 09:38:13 DBG  id=015XQ7CXTQV5BQMFQBDBCJNKAAZOFRDUPY nodeID=44 op=Fsync path=/config/keepass.kdbx
  2. Word document in .docx format:
    • Again the metadata seems identical, but opening the file in the original mount with Libre Office results in the warning:

      The file 'wgs-quote.docx' is corrupt and therefore cannot be opened. LibreOffice can try to repair the file. The corruption could be the result of document manipulation or of structural document damage due to data transmission. We recommend that you do not trust the content of the repaired document. Execution of macros is disabled for this document. Should LibreOffice repair the file?

    • The file in the new mount point works fine.
    • This file was also recently modified on a windows machine.
    • The log reads:
      Aug 21 09:42:59 fedora onedriver[12836]: 09:42:59 DBG  id=015XQ7CXTVUAMAV7N2JJAKQAER2XEUOYX7 nodeID=25 op=Open path=/quote.docx
      Aug 21 09:42:59 fedora onedriver[12836]: 09:42:59 INF Not using cached item due to file hash mismatch, fetching content from API. id=015XQ7CXTVUAMAV7N2JJAKQAER2XEUOYX7 nodeID=25 op=Open path=/quote.docx
      Aug 21 09:43:00 fedora onedriver[12836]: 09:43:00 DBG  id=015XQ7CXTVUAMAV7N2JJAKQAER2XEUOYX7 nodeID=25 op=Fsync path=/quote.docx

I believe the problem might be related to expired authentication credentials, because I had to log-in again using OTP in both my browser and thunderbird today as well. However, creating new files in the original mount point works fine; they become visible immediately in the web version.

Any help would be much appreciated!

ruuzia commented 10 months ago

So I just installed this program a few days ago, and the first thing I tested was

  1. Create a file locally
  2. Update the file remotely
  3. Open the file locally

I was disappointed to find the above messages showing it was attempting to fetch the latest changes but not actually working.

Anyways, it's just broken in the v0.14 caching system.

Fixed in the above pull request, or you can install v0.13 for now.

pmoris commented 10 months ago

Oh wow! Thanks a bunch for not only figuring out the issue, but also trying to solve it with a PR. I'll give it a spin.

massix commented 9 months ago

Hey @pmoris I've been testing out your patch for a couple of weeks now on my NixOS and I'm afraid that the problem is still there; less visible and less frequent but from time to time I have to restart the service in order to "force" the synchronization, something which was not happening with v0.13.0-2.. I don't know if there are other underlying issues, I just wanted to let you know :)

Thanks a lot for your effort!

philmck commented 9 months ago

In case it helps, I've had the same problem and I have lost data from my KeePassXC database as a result. :-(

Worse still, I have been unable to recover them from the OneDrive file history. This utility is very dangerous in its current state!

As far as I can tell, onedriver (installed in Linux Mint) correctly shows the latest modification date and time on my kbdx file in OneDrive, and bidirectional synching SEEMS to be working if you just look as file sizes and dates. But if I try to open the latest database in Linux, an old version of the file seems to be loaded. It then seems to delete the file in OneDrive and replace it with that old version, erasing the history in the process.

I'm uninstalling it for now because it's too dangerous, but if I can help with diagnostic information let me know.

massix commented 9 months ago

In case it helps, I've had the same problem and I have lost data from my KeePassXC database as a result. :-(

Worse still, I have been unable to recover them from the OneDrive file history. This utility is very dangerous in its current state!

As far as I can tell, onedriver (installed in Linux Mint) correctly shows the latest modification date and time on my kbdx file in OneDrive, and bidirectional synching SEEMS to be working if you just look as file sizes and dates. But if I try to open the latest database in Linux, an old version of the file seems to be loaded. It then seems to delete the file in OneDrive and replace it with that old version, erasing the history in the process.

I'm uninstalling it for now because it's too dangerous, but if I can help with diagnostic information let me know.

Version 0.13.0-2 is quite stable, I've been using it for weeks without issues. I suggest you to rollback if you can, while waiting for a fix..

jstaf commented 8 months ago

Should be fixed in 0.14.1 I think. I ended up doing an expanded version of @grassdne's changes after catching another bug with hashing of file streams (this is why some files would constantly re-download and never sync because the hashes never matched). Sorry for the wait!

Please let me know/reopen if the issue is still popping up.