Regression in 0.14.0 fills entire disk

[danigamba]

After the latest released (from Suse build for Ubuntu 23.04) onedriver filled the entire disk with 110 GB of cache files. In the logs I have thousands of lines like this

2023-07-14T09:58:01.457105+02:00 dani-x1carb onedriver[1969]: #033[90m09:58:01#033[0m #033[32mINF#033[0m Migrating file content. #033[36mkey=#033[0m014EHT2EKL3FHG2NRIMFF

Let me know if I can contribute with additional details for debugging this issue

[jstaf]

Ah - yeah I knew this would happen to someone out there. Sorry about that :grimacing:

The biggest problem with this app in 0.13.x was that it was really "expensive" in terms of both disk and memory. The db I used when I originally wrote this app (boltdb) doesn't have the ability to read only part of a file, it had to read the whole file into memory when you're reading/writing it (this could make onedriver semi-demanding on your PC depending on how big the files you were opening were). Likewise, once a file was downloaded, it kept things basically forever in the db, which would cause the cache to just grow forever slowly over time (or super-quickly if you were storing videos in OneDrive).

The best way to fix the memory issue is just to store files as... regular files. Then we don't need to use memory at all. But in order to switch to this, we need to migrate all of the file contents out of the db one last time which temporarily requires twice the space until the migration completes and the old db file storage is cleaned up. (The really annoying thing is that the db can't be compacted and made smaller without copying it and doubling the space used.)

As a workaround, you can "just delete" the onedriver.db file (or delete and readd the mountpoint in the UI). Counter-intuitively, this is actually safe to do - unless you just uploaded like several gigabytes of files in the last 10 seconds and onedriver isn't done uploading them yet, the Microsoft OneDrive servers already have the latest version of all of your files and onedriver will just redownload the latest versions when you access them again.

---

Re: dealing with why the cache is so big in the first place:

I am hoping to have a setting where we can pick a maximum size/maximum age of files in the cache (so you don't have to clean up things manually like this) in a future version.

Likewise, 99% of the files get downloaded in the first place because the GNOME file browser (Nautilus) simply can't help itself. Nautilus literally cannot be told to not download every file in a folder when you open that folder because it really, really wants to make thumbnails/preview images of everything in that folder. I've filed a bug report on this with GNOME but it's been ignored for 3+ years. However, I think I have a new idea on how to stop this, and was hoping to get that into a future version as well. (Should make everything much faster and decrease the amount of disk used because the file browser is no longer downloading things like crazy to make preview images.)

[dclark16]

Hi Jstaf,

First of all, thank you so much for all your hard work on this project.

Since version 14, how can we recover the space taken up by the "content" directory, is it safe to just delete everything in the directory? Baloo is indexing the contents of the "OneDriver", it has downloaded 200GB+ and subsequently eat all the space in my home area, which is fine but KDE really doesn't do well once the home area runs out of space.

Kind regards!

[jstaf]

Yeah you can just delete the whole directory and it will recreate whatever files it needs to. Also created #358 for the baloo issue (I keep forgetting that this is an issue for KDE people and it would be nice to auto-fix it for people).