search

gboudreau / Greyhole

Greyhole uses Samba to create a storage pool of all your available hard drives, and allows you to create redundant copies of the files you store.
http://www.greyhole.net
GNU General Public License v3.0
263 stars 34 forks source link

Locked share #140

Closed zarmstrong closed 9 years ago

zarmstrong commented 9 years ago

I'm getting a bunch of messages in the greyhole log about locked shares. I can't seem to figure out the cause. Reboots, stopping other services, etc don't help. It's always the same files.

Jul 23 10:18:19 INFO write: Now working on task ID 36997065: write Media/TVShows/The Walking Dead/Season 04/The Walking Dead - S04E15 - Us.mkv.cover.jpg [2439448] Jul 23 10:18:19 INFO write: File created: Media/TVShows/The Walking Dead/Season 04/The Walking Dead - S04E15 - Us.mkv.cover.jpg - 43.3KB [2440040] Jul 23 10:18:19 DEBUG write: Loading metafiles for Media/TVShows/The Walking Dead/Season 04/The Walking Dead - S04E15 - Us.mkv.cover.jpg ... [2440184] Jul 23 10:18:19 DEBUG write: Got 0 metadata files. [2440504] Jul 23 10:18:19 DEBUG write: 0 metafiles loaded. [2440392] Jul 23 10:18:19 DEBUG write: File /mnt/snapdisk4/landingzone/TVShows/The Walking Dead/Season 04/The Walking Dead - S04E15 - Us.mkv.cover.jpg is locked by another process. Will wait until it's unlocked to work on any file in this share. [2440808] Jul 23 10:18:19 INFO write: Now working on task ID 36997066: write Media/TVShows/The Walking Dead/Season 04/The Walking Dead - S04E15 - Us.mkv [2438504] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2438368] Jul 23 10:18:19 INFO write: Now working on task ID 36997085: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E13 - TAC in the Day.mkv.cover.jpg [2437056] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2436912] Jul 23 10:18:19 INFO write: Now working on task ID 36997086: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E06 - Ditch Day.mkv [2435568] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2435440] Jul 23 10:18:19 INFO write: Now working on task ID 36997087: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E05 - Gayborhood.mkv.cover.jpg [2434128] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2433984] Jul 23 10:18:19 INFO write: Now working on task ID 36997088: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E10 - Trivia Pursuits.mkv.cover.jpg [2432664] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2432520] Jul 23 10:18:19 INFO write: Now working on task ID 36997089: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E12 - Peyote It Forward.mkv [2431184] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2431048] Jul 23 10:18:19 INFO write: Now working on task ID 36997090: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E09 - Wedding Thrashers.mkv.cover.jpg [2429720] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2429568] Jul 23 10:18:19 INFO write: Now working on task ID 36997091: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E08 - Blood Drive.mkv [2428176] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2428040] Jul 23 10:18:19 INFO write: Now working on task ID 36997092: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E11 - The Slump.mkv.cover.jpg [2426648] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2426504] Jul 23 10:18:19 INFO write: Now working on task ID 36997093: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E06 - Ditch Day.mkv.cover.jpg [2425088] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2424944] Jul 23 10:18:19 INFO write: Now working on task ID 36997094: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E07 - Gramps DeMamp is Dead.mkv [2423520] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2423376] Jul 23 10:18:19 INFO write: Now working on task ID 36997095: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E11 - The Slump.mkv [2421952] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2421824] Jul 23 10:18:19 INFO write: Now working on task ID 36997096: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E13 - TAC in the Day.mkv [2420416] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2420280] Jul 23 10:18:19 INFO write: Now working on task ID 36997097: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E04 - Menergy Crisis.mkv [2418960] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2418824] Jul 23 10:18:19 INFO write: Now working on task ID 36997098: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E12 - Peyote It Forward.mkv.cover.jpg [2417520] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2417368] Jul 23 10:18:19 INFO write: Now working on task ID 36997099: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E09 - Wedding Thrashers.mkv [2416024] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2415888] Jul 23 10:18:19 INFO write: Now working on task ID 36997100: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E05 - Gayborhood.mkv [2414552] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2414416] Jul 23 10:18:19 INFO write: Now working on task ID 36997101: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E04 - Menergy Crisis.mkv.cover.jpg [2413096] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2412952] Jul 23 10:18:19 INFO write: Now working on task ID 36997102: write Media/TVShows/Workaholics/Season 05/Workaholics - S05E07 - Gramps DeMamp is Dead.mkv.cover.jpg [2411608] Jul 23 10:18:19 INFO write: Share is locked because another file operation is waiting for a file handle to be released. Skipping. [2411456] Jul 23 10:18:19 DEBUG sleep: Nothing to do... Sleeping. [2409816]

FatzBomb75 commented 9 years ago

I had this problem recently and as it turned out for me, Samba had crashed failing to close the handles to the files. My solution was copying the files to a new destination, deleting the original files and moving the copied files back to the original location (restating the system in between). This could work by just renaming the copied folder back to the original name.

Also, don't forget after doing this, you may have to set your file permissions back to their original values, so take note of that first.

I hope this helps, Keith

FatzBomb75 commented 9 years ago

I should add it was only the files in the landing zone in my case which didn't allow greyhole to move them to the storage drives. I can't remember the command off hand, but you can copy only the files that are not symbolic links as well as delete those files via terminal as root.

Doing it this way would prevent you from having to recreate the entire greyhole pool through fsck. A little googling should net you the command.

gboudreau commented 9 years ago

It is indeed a stuck file write operation. Samba was closed (probably crashed) before it could close the file handle, and Greyhole now thinks it is still open. You can simply do:

touch "/mnt/samba/Media/TVShows/The Walking Dead/Season 04/The Walking Dead - S04E15 - Us.mkv.cover.jpg"

Provided that you have your shares locally mounted.

gboudreau commented 9 years ago

Also, if it's not too late, could you zip the /var/spool/greyhole folder, and send it to me at guillaume@greyhole.net ? Thanks.

zarmstrong commented 9 years ago

Thanks for the suggestions. I tried touching the file both on the samba share and in the LZ. Didn't seem to have any affect at all.

I then moved ALL the files erroring in the log out of the LZ. This seemed to fix it. I'm letting it catch up before I put these files back in place, though this is probably an unnecessary precaution.

zarmstrong commented 9 years ago

Sorry, I guess it's too late. That path is empty.

zarmstrong commented 9 years ago

I'm actually able to reproduce it, but not reliably. I checked /var/spool/greyhole its empty save for a directory named mem (also empty).

Is there something I can do to enable some debugging that might help?

The workaround is move all the files listed in the error out of there, then back into the LZ after the error clears.

gboudreau commented 9 years ago

Yes, I looked into it, but was never able to reproduce, but I'm pretty sure I found the issue: it's that we saw the start of a write operation, and logged that into the Greyhole database, but never received the end-of-write signal, and thus that write operation stays locked forever. Try the new 0.9.62 version. When you're able to reproduce the issue, the fix is to either reboot, or execute this:

sudo greyhole --boot-init

That command is executed on boot, thus why a reboot should also fix the issue. It marks all file writes operations as complete, so make sure you're not writing to a file on samba when executing it; safer to just reboot.

SteveByerly commented 8 years ago

Just noting that I had the same issue. I had renamed a in the LZ. Reboot fixed the issue as noted above.