search

plexdrive / plexdrive

Plexdrive mounts your Google Drive FUSE filesystem (optimized for media playback)
MIT License
1.76k stars 206 forks source link

file resize error on building cache/bolt file for the first time #362

Open tcowing opened 4 years ago

tcowing commented 4 years ago

I'm setting plexdrive up against a Google Drive library with lots of files (~20TB). During the initial scan to build the bolt file plexdrive always halts after a few hours with the following error:

[USR/SBIN/PLEXDRIVE] [2020-05-04 01:04] DEBUG : file resize error: truncate /home/seed/.config/plexdrive/cache.bolt: invalid argument [USR/SBIN/PLEXDRIVE] [2020-05-04 01:04] INFO : Unmounting path /mnt/plexdrive [USR/SBIN/PLEXDRIVE] [2020-05-04 01:04] DEBUG : Closing cache file [USR/SBIN/PLEXDRIVE] [2020-05-04 01:04] WARNING: Could not update/save objects: file resize error: truncate /home/seed/.config/plexdrive/cache.bolt: invalid argument

I'm running plexdrive with the following flags:

/usr/sbin/plexdrive mount -v 4 --refresh-interval=1m --chunk-check-threads=8 --chunk-load-threads=8 --chunk-load-ahead=4 --max-chunks=100 --fuse-options=allow_other,read_only --config=/home/seed/.config/plexdrive --cache-file=/home/seed/.config/plexdrive/cache.bolt /mnt/plexdrive

I'm out of ideas at this point but I'm happy to help debug. Just point me in the right direction.

Todd

plexdrive.log

tcowing commented 4 years ago

Oh and there's plenty of room on the filesystem. It's not that either. I'm running Ubuntu 18.04.

felixbuenemann commented 4 years ago

Ist it a 32-Bit system? 2GB sounds suspiciously like a 32-Bit limit. It is strange that it gets so large though, my cache.bolt is just 8 MB with a similar amount of data.

felixbuenemann commented 4 years ago

It would also be helpful, if you could check how many objects there are in your google drive, as I'd like to understand why the cache.bolt gets so large in the first place.

I also checked and bolt db uses mmap, so it could be problematic with more than 2GB on 32-Bit systems, but it could also be some internal data structure that uses unsigned 32-Bit integers.

felixbuenemann commented 4 years ago

From the log it looks like there are many files that are not media. You could try mounting a subfolder instead. Go to drive in the browser and surf to the folder below which your media is located and copy the folder id from the browser's address bar, then specify it as '--root-node-id' when starting plexdrive.

I am not sure if that also allows files outside that folder to be ignored for the cache.bolt, but it is worth a try.

tcowing commented 4 years ago

Hmmm....I should have thought to check if I was a running on a 32-bit instance (the VM is an Amazon Lightsail instance). I was hoping that was it since it's an easy fix but uname -m reports x86_64. Still...you're right about it being suspiciously close to the 2GB limit.

I'm not actually sure how many files I have. It's actually an media directory filled with rclone encrypted files that I'm running plexdrive against, if that makes any difference. That's basically the only data in this user account. I don't believe rclone encryption slices up the files into chunks though so that's probably not making a huge number of files. Is there an easy way to tell how many files there are using the Google Admin console or somewhere in the reports? I looked around a little but it wasn't obvious. I'm only mildly familiar with Google tools and I wouldn't call them user friendly. I'll dig around a little deeper and see if there's an easy way to get a total count. I did find a couple threads (recent ones) of people complaining this feature is missing but that doesn't necessarily mean anything.

I set plexdrive to have it's root-node to a folder with probably a total of no more ~10k files in it. I'll report back when the initial scan is complete.

tcowing commented 4 years ago

This time it ran until it hit almost exactly 4.0GB then stopped with the same error message. Strange. Do you want me to run it against a small folder with only a few files in it?

felixbuenemann commented 4 years ago

I wonder why it is getting so large in the first place. My plexdrive cache.bolt is 8 MB with around 10k files (including directories).

It is possible that plexdrive still has to go through all the changesets fir the drive, ecen if a subdirectory is mounted via root node id. I would have to read through the code an APyi reference to say for sure.

triwing commented 1 year ago

I know this issue is nearly 3 years old, but I also ran into this one. AFAIK you'll face this problem with a 32bit version of plexdrive, if your cache.bolt exceeds 2gb.

Just switch to a 64bit version and it'll keep on working.

Ran into this on a raspberry pi 4 with plexdrive-linux-arm7 and switched over to plexdrive-linux-arm64.