SyncHandler error unrecoverable

[GoogleCodeExporter]

I setup a blob-based camlistore using kv as the indexmechanism.
After uploading many files... I see this error from SyncHandler:

(my kv index is 1.66 GB in size, and I have about 180GB of blobs)

Client:

2014/11/29 22:07:42 FILES: 
Total: {files:14300 bytes:116928633465} 
Skipped: {files:3067 bytes:20376547894} 
Uploaded: {files:10996 bytes:88601455180}  
BLOBS: [uploadRequests=[blobs=1536618 bytes=91381065531] uploads=[blobs=1534331 
bytes=91249330012]] 
Digested: 10498 
last upload: I:\Photos\2010\2010_12_31_MOV\P1020623.MOV

2014/11/29 22:07:42 Error uploading I:\Photos\2010\2010_12_31_MOV\P1020745.MOV:
open I:\Photos\2010\2010_12_31_MOV\P1020745.MOV: The request could not be 
performed because of an I/O device error.

Server:

2014/11/30 20:17:47 [SyncHandler /bs/ -> /index/] queue delete of 
sha1-d61f85fc5f0d0906c355b4859e1074fedb9a96c2 error: Error, off: 
0x6163323362636c0, EOF
2014/11/30 20:18:14 [SyncHandler /bs/ -> /index/] queue delete of 
sha1-d6f0b77ed7466251de73e2bc557bfe1a98d5d8b7 error: Error, off: 
0x6163323362636c0, EOF
2014/11/30 20:18:41 [SyncHandler /bs/ -> /index/] error copying 
sha1-52d4e6fdec92db682912c43d26db0e4ae0dc5469: Read data has unexpected digest 
974cae93acd9fa61198a03697bde69e75bb81a03
2014/11/30 20:18:52 [SyncHandler /bs/ -> /index/] queue delete of 
sha1-d70691d9a411afb860fd506a87dbd46c25bd6d96 error: Error, off: 
0x6163323362636c0, EOF
2014/11/30 20:19:47 [SyncHandler /bs/ -> /index/] queue delete of 
sha1-d63db8092ad4d94be2ddc611571145da9e27c4dc error: Error, off: 
0x6163323362636c0, EOF
2014/11/30 20:19:54 [SyncHandler /bs/ -> /index/] queue delete of 
sha1-d703b40df7b38270cd588af9cdc2442205745539 error: Error, off: 
0x6163323362636c0, EOF
2014/11/30 20:20:03 [SyncHandler /bs/ -> /index/] queue delete of 
sha1-d636c1174e216e0cdaa53a3c7722b8e0645c0822 error: Error, off: 
0x6163323362636c0, EOF
2014/11/30 20:20:15 Got SIGINT: shutting down
2014/11/30 20:20:15 Closing kvfile database O:\Camlistore\data\blobs\index.kv
Timeout shutting down. Exiting uncleanly.
Terminate batch job (Y/N)? Y

O:\Camlistore>Y
'Y' is not recognized as an internal or external command,
operable program or batch file.

O:\Camlistore>camlistore.bat

O:\Camlistore>windows\camlistored.exe -configfile 
O:\Camlistore\config\server-config.json
2014/11/30 20:20:23 Starting camlistored version 2014-11-18-21a61da; Go go1.3.3 
(windows/amd64)
2014/11/30 20:20:24 Starting to listen on http://localhost:3179
2014/11/30 20:20:28 [SyncHandler /bs/ -> /index/] Added 201081 pending blobs 
from sync queue to pending list
2014/11/30 20:20:28 Caught panic installer handlers: error instantiating 
handler for prefix "/sync/", type "sync": Error reading sync queue to memory: 
Error, off: 0x6163323362636c0, EOF
Error parsing config: Caught panic: error instantiating handler for prefix 
"/sync/", type "sync": Error reading sync queue to memory: Error, off: 
0x6163323362636c0, EOF

Original issue reported on code.google.com by laramie....@gmail.com on 1 Dec 2014 at 6:37

[GoogleCodeExporter]

I don't know about the root cause of the problem (some kv corruption?), but you 
ended up with a corrupted index, so you'd have to at least restart camlistored 
with -reindex.
Could you please retry with leveldb instead of kv as your indexer?

Original comment by mathieu....@gmail.com on 2 Dec 2014 at 1:13

• Changed state: HelpWanted
• Added labels: Type-Defect

[GoogleCodeExporter]

Original comment by mathieu....@gmail.com on 2 Dec 2014 at 3:16

• Added labels: Storage-Sorted

[GoogleCodeExporter]

Is there a simple way to reindex from kv to leveldb?

Original comment by laramie....@gmail.com on 8 Dec 2014 at 5:33

[GoogleCodeExporter]

This issue has moved to https://camlistore.org/issue/555

Original comment by bradfitz on 14 Dec 2014 at 11:39

• Changed state: Moved
• Added labels: IssueMoved, Restrict-AddIssueComment-Commit