search

gboudreau / Greyhole

Greyhole uses Samba to create a storage pool of all your available hard drives, and allows you to create redundant copies of the files you store.
http://www.greyhole.net
GNU General Public License v3.0
262 stars 34 forks source link

Critical error during FSCK - CRITICAL fsck: Fatal error! $file_metafiles is now a string #132

Closed martincjarvis closed 9 years ago

martincjarvis commented 9 years ago

During every FSCK operation the daemon stops with the following log message:

Feb 05 07:51:50 CRITICAL fsck: Fatal error! $file_metafiles is now a string: 'array (  '/media/45dfde82-c4f5-452b-a361-f5d135e47a23/.greyhole/Storage/Videos/TV/The Andromeda Strain (2008)/Season 01/The Andromeda Strain (2008) - S01E02 - Part 2 - SD.TV.en.srt' =>   stdClass::__set_state(array(     'path' => '/media/45dfde82-c4f5-452b-a361-f5d135e47a23/.greyhole/Storage/Videos/TV/The Andromeda Strain (2008)/Season 01/The Andromeda Strain (2008) - S01E02 - Part 2 - SD.TV.en.srt',     'is_linked' => false,     'state' => 'OK',  )),)'.

I'm using greyhole 0.9.60 and have recently migrated several drives (using the --going process).

gboudreau commented 9 years ago

Can you change log_level to DEBUG, restart the service, and try the fsck again, to see what gets logged ?

martincjarvis commented 9 years ago

Thanks for the response. I'm afraid that shortly after I opened the issue, I started getting another error during a rename operation that's preventing the fsck operation from running!

After the greyhole daemon startup and "is OK," messages the log records:

Feb 10 19:49:53 DEBUG read_smb_spool: Processing Samba spool...
Feb 10 19:50:07 DEBUG read_smb_spool: Found 10 new tasks in spool.
Feb 10 19:50:07 INFO rename: Now working on task ID 124: rename Storage/Videos/Adult/Movies/Shaun of the Dead (2004) R 720p Bluray (1280x560) -> Storage/Videos/Adult/Movies/Shaun of the Dead (2004)
Feb 10 19:50:07 INFO rename: Directory renamed: /media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/Movies/Shaun of the Dead (2004) R 720p Bluray (1280x560) -> /media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/Movies/Shaun of the Dead (2004)
Feb 10 19:50:08 DEBUG rename: Loading metadata files for (dir) Storage/Videos/Adult/Movies/Shaun of the Dead (2004) ...
Feb 10 19:50:08 DEBUG rename: Loading metafiles for Storage/Videos/Adult/Movies/Shaun of the Dead (2004)/Shaun of the Dead (2004).mkv ...
Feb 10 19:50:08 ERROR rename: PHP Fatal Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 729786368 bytes); BT: greyhole[L4388]

I suspect that something's gone wrong with my installation, but don't know what I can do about repairing it...or have I just had some bad luck?

gboudreau commented 9 years ago

Memory errors are normally triggered when trying to rename very large folders, but I doubt that's the case here... Anyway, you can increase the maximum memory that Greyhole can use, temporarily, in /etc/greyhole.conf: https://github.com/gboudreau/Greyhole/blob/master/greyhole.example.conf#L230 Use 1G or something, to make sure it works, and lower it back to the default (comment-out the memory_limit line) later, once Greyhole was able to catch up with everything it had to do.

martincjarvis commented 9 years ago

Ok, I had to raise the allowed memory several times (up to 7G in the end) and got a little further each time. Unfortunately, I then ran out of memory (I've only got 7,7G usable). It looks like there was a set of folder renames (probably triggered by couchpotato), none of the folders seemed particularly big.

Since, the renames weren't important, I've cleared out the tasks so I can get the fsck running again. It now errors with:

Feb 14 20:41:34 WARN fsck: PHP Warning [2]: is_file(): open_basedir restriction in effect. File(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/.gh_metastore/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg) is not within the allowed path(s): (/) in /usr/bin/greyhole on line 4364; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) => greyhole[L5185] get_metafile_data_filename(Storage,Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg) => greyhole[L4354] get_metafile_data_filenames(Storage,Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,1) => greyhole[L4364] is_file(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/.gh_metastore/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg) [2332328]
Feb 14 20:41:34 WARN fsck: PHP Warning [2]: is_file(): open_basedir restriction in effect. File(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/.gh_metastore/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg) is not within the allowed path(s): (/) in /usr/bin/greyhole on line 4364; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) => greyhole[L5185] get_metafile_data_filename(Storage,Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg) => greyhole[L4354] get_metafile_data_filenames(Storage,Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,1) => greyhole[L4364] is_file(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/.gh_metastore/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg) [2332496]
Feb 14 20:41:34 DEBUG fsck: Found /media/f4ec69a1-354b-4c02-9e1b-e2ccb9efe94f/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg [2325904]
Feb 14 20:41:34 DEBUG fsck: Found /media/a4abeb37-5dc2-4786-9c78-d5690602958b/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg [2327544]
Feb 14 20:41:34 WARN fsck: PHP Warning [2]: is_link(): open_basedir restriction in effect. File(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg) is not within the allowed path(s): (/) in /usr/bin/greyhole on line 6383; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) => greyhole[L5203] get_file_inodes(Storage,Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,array (  '/media/f4ec69a1-354b-4c02-9e1b-e2ccb9efe94f/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg' =>   stdClass::__set_state(array(     'path' => '/media/f4ec69a1-354b-4c02-9e1b-e2ccb9efe94f/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg',     'is_linked' => false,     'state' => 'OK',  )),  '/media/a4abeb37-5dc2-4786-9c78-d5690602958b/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg' =>   stdClass::__set_state(array(     'path' => '/media/a4abeb37-5dc2-4786-9c78-d5690602958b/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg',     'is_linked' => false,     'state' => 'OK',  )),)) => greyhole[L6383] is_link(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg) [2336496]
Feb 14 20:41:34 DEBUG fsck: Loading metafiles for Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg ... [2325864]
Feb 14 20:41:34 WARN fsck: PHP Warning [2]: is_file(): open_basedir restriction in effect. File(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/.gh_metastore/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg) is not within the allowed path(s): (/) in /usr/bin/greyhole on line 4364; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) => greyhole[L5216] get_metafiles(Storage,Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,1) => greyhole[L4377] get_metafiles_for_file(Storage,Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,1,,1) => greyhole[L4384] get_metafile_data_filename(Storage,Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg) => greyhole[L4354] get_metafile_data_filenames(Storage,Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,1) => greyhole[L4364] is_file(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/.gh_metastore/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg) [2336112]
Feb 14 20:41:34 DEBUG fsck:   Got 0 metadata files. [2326984]
Feb 14 20:41:34 WARN fsck: PHP Warning [2]: Invalid argument supplied for foreach() in /usr/bin/greyhole on line 5276; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2331936]
Feb 14 20:41:34 WARN fsck: PHP Warning [2]: Invalid argument supplied for foreach() in /usr/bin/greyhole on line 5291; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2331824]
Feb 14 20:41:34 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5305; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2332232]
Feb 14 20:41:34 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5305; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2338848]
Feb 14 20:41:34 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5305; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2338536]
Feb 14 20:41:34 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5305; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2337416]
Feb 14 20:41:34 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5305; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2337328]
Feb 14 20:41:34 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5305; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2336848]
Feb 14 20:41:34 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5339; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2338152]
Feb 14 20:41:34 WARN fsck: PHP Warning [8]: Undefined variable: original_file_path in /usr/bin/greyhole on line 5340; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2338184]
Feb 14 20:41:34 WARN fsck: PHP Warning [2]: mb_strpos(): Empty delimiter in /usr/bin/greyhole on line 1952; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) => greyhole[L5340] real_file_is_locked() => greyhole[L3891] string_contains(,) => greyhole[L1952] mb_strpos(,) [2333312]
Feb 14 20:41:34 DEBUG read_smb_spool: Processing Samba spool... [3237856]
Feb 14 20:41:36 WARN fsck: PHP Warning [8]: Undefined variable: original_file_path in /usr/bin/greyhole on line 5353; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2338112]
Feb 14 20:41:36 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5353; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2338120]
Feb 14 20:41:36 WARN fsck:   A file copy with a different file size than the original was found: /media/f4ec69a1-354b-4c02-9e1b-e2ccb9efe94f/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg is 20,793 bytes. Original:  is 0 bytes. [2329648]
Feb 14 20:41:36 DEBUG fsck:   Moved copy from /media/f4ec69a1-354b-4c02-9e1b-e2ccb9efe94f/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg to trash: /media/f4ec69a1-354b-4c02-9e1b-e2ccb9efe94f/.greyhole/.gh_trash/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E12 - Combustion - SD.TV-thumb.jpg [2333776]
Feb 14 20:41:36 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5355; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2338800]
Feb 14 20:41:36 WARN fsck: PHP Warning [8]: Undefined variable: original_file_path in /usr/bin/greyhole on line 5355; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E12 - Combustion - SD.TV-thumb.jpg,link,landing_zone,Storage,) [2338800]
Feb 14 20:41:36 ERROR fsck: PHP Fatal Error: Cannot unset string offsets; BT: greyhole[L5358]  [2332464]

Thanks for your patience with debugging this.

gboudreau commented 9 years ago

Very weird errors you're getting... The open_basedir is set to /, which should allow the PHP process to access all files below that root; no idea why it complains about that... Anyway, you can try to change the 1st line of /usr/bin/greyhole from #!/usr/bin/php -d open_basedir=/ to #!/usr/bin/php

martincjarvis commented 9 years ago

Thanks again, but I still see very similar errors:

Feb 15 11:33:58 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5353; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E04 - Cheating - SD.TV.tbn,link,landing_zone,Storage,) [2433320]
Feb 15 11:33:58 WARN fsck:   A file copy with a different file size than the original was found: /media/b1c76e1f-f98f-4579-a487-056137a9535c/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E04 - Cheating - SD.TV.tbn is 15,519 bytes. Original:  is 0 bytes. [2425120]
Feb 15 11:33:58 DEBUG fsck:   Moved copy from /media/b1c76e1f-f98f-4579-a487-056137a9535c/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E04 - Cheating - SD.TV.tbn to trash: /media/b1c76e1f-f98f-4579-a487-056137a9535c/.greyhole/.gh_trash/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E04 - Cheating - SD.TV.tbn [2428200]
Feb 15 11:33:58 WARN fsck: PHP Warning [8]: Undefined variable: expected_file_size in /usr/bin/greyhole on line 5355; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E04 - Cheating - SD.TV.tbn,link,landing_zone,Storage,) [2433504]
Feb 15 11:33:58 WARN fsck: PHP Warning [8]: Undefined variable: original_file_path in /usr/bin/greyhole on line 5355; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E04 - Cheating - SD.TV.tbn,link,landing_zone,Storage,) [2433616]
Feb 15 11:33:58 ERROR fsck: PHP Fatal Error: Cannot unset string offsets; BT: greyhole[L5358]  [2427104]

Do you have any further suggestions?

martincjarvis commented 9 years ago

I've tried restarting the greyhole service and I'm now back to the original error:

Feb 15 13:49:58 DEBUG fsck: Found /media/b1c76e1f-f98f-4579-a487-056137a9535c/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E08 - Corby - SD.TV.en.srt [2353360]
Feb 15 13:49:58 WARN fsck: PHP Warning [2]: is_link(): open_basedir restriction in effect. File(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E08 - Corby - SD.TV.en.srt) is not within the allowed path(s): (/) in /usr/bin/greyhole on line 6383; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3492] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting,Storage,) => greyhole[L5079] gh_fsck(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,Storage,) => greyhole[L5081] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E08 - Corby - SD.TV.en.srt,link,landing_zone,Storage,) => greyhole[L5203] get_file_inodes(Storage,Videos/Adult/TV/QI Quite Interesting/Season 03,QI - S03E08 - Corby - SD.TV.en.srt,array (  '/media/b1c76e1f-f98f-4579-a487-056137a9535c/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E08 - Corby - SD.TV.en.srt' =>   stdClass::__set_state(array(     'path' => '/media/b1c76e1f-f98f-4579-a487-056137a9535c/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E08 - Corby - SD.TV.en.srt',     'is_linked' => false,     'state' => 'OK',  )),)) => greyhole[L6383] is_link(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E08 - Corby - SD.TV.en.srt) [2362400]
Feb 15 13:49:58 DEBUG fsck: Found /media/620abb76-3b98-4e1a-a04a-eed3739447d4/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E08 - Corby - SD.TV.en.srt [2356224]
Feb 15 13:49:58 CRITICAL fsck: Fatal error! $file_metafiles is now a string: 'array (  '/media/b1c76e1f-f98f-4579-a487-056137a9535c/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E08 - Corby - SD.TV.en.srt' =>   stdClass::__set_state(array(     'path' => '/media/b1c76e1f-f98f-4579-a487-056137a9535c/.greyhole/Storage/Videos/Adult/TV/QI Quite Interesting/Season 03/QI - S03E08 - Corby - SD.TV.en.srt',     'is_linked' => false,     'state' => 'OK',  )),)'. [2356680]

Thanks for your patience.

gboudreau commented 9 years ago

At this point, I would need remote access to be able to help you further. Please create a new user, give it sudo access, and allow it to login using SSH remotely (setup port forwarding as needed). Send me the needed details to connect by opening a new support ticket here: https://greyhole.freshdesk.com

gboudreau commented 9 years ago

Follow up: a bad drive was causing those issues (/media/b563c756-e9b5-48d0-9de8-33791311196f). It had I/O errors. A real (ext4) fsck of the drive, or removing it from the Greyhole pool, if it can't be fixed, should resolve all those errors.

martincjarvis commented 9 years ago

Thanks again for this. I've run the real fsck check and fix and kicked off a greyhole fsck which has been running for over two hours (which is much longer than before). I'll confirm when it's done, but it looks like the issue is resolved. Thanks

martincjarvis commented 9 years ago

Sorry, the fsck has been running for days and is now failing again:

Feb 25 19:44:43 DEBUG fsck_file: Loading metafiles for Storage/Videos/Adult/TV/The Walking Dead/Season 04 (file copy) ... [2327960]
Feb 25 19:45:46 WARN fsck_file: PHP Warning [2]: file_get_contents(): content truncated from 2343588837 to 2147483647 bytes in /usr/bin/greyhole on line 4388; BT: greyhole[L3273] run() => greyhole[L3249] execute_next_task() => greyhole[L3387] gh_fsck_file(/media/c20c4014-679b-453e-af58-96417cb9ffc1/lz/Videos/Adult/TV/The Walking Dead,Season 04 (file copy),,metastore,Storage) => greyhole[L5216] get_metafiles(Storage,Videos/Adult/TV/The Walking Dead,Season 04 (file copy),1) => greyhole[L4377] get_metafiles_for_file(Storage,Videos/Adult/TV/The Walking Dead,Season 04 (file copy),1,,1) => greyhole[L4388] file_get_contents(/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole/.gh_metastore/Storage/Videos/Adult/TV/The Walking Dead/Season 04 (file copy)) [2345931656]
Feb 25 19:45:52 ERROR fsck_file: PHP Fatal Error: Allowed memory size of 7516192768 bytes exhausted (tried to allocate 2147483776 bytes); BT: greyhole[L1691]  [6640936344]

The 7G allowed memory usage is still in effect and i've done a real fsck on the /media/b563c756-e9b5-48d0-9de8-33791311196f drive.

Have you any ideas on how to move forward? I've reopened your SSH access to the server so you can access logs, etc..

gboudreau commented 9 years ago

Your fsck-ed drive was not used. Your log says:

Feb 25 19:44:20 WARN daemon: Warning! It seems the partition UUID of /media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole changed. This probably means this mount is currently unmounted, or that you replaced this drive and didn't use 'greyhole --replace'. Because of that, Greyhole will NOT use this drive at this time. [2276840]

I fixed that problem like this:

greyhole --replace=/media/b563c756-e9b5-48d0-9de8-33791311196f/.greyhole

And your out-of-memory issue is due to a huge (2.2G!) metadata file!! This file is supposed to contain a couple bytes of JSON text, not binary data!

You can find the files that have this problem like this:

find /media/*/.greyhole/.gh_metastore -type f -size +4k

You'll need to move those binary files out of the .gh_metastore folders. Then try greyhole --fsck again.

martincjarvis commented 9 years ago

All fixed now! Thanks for your patience.