Refactor to use Cuttlefish

isaacs commented 10 years ago

The logging output is all completely changed, since Cuttlefish does things in a single big task queue, rather than a bunch of independent steps.

But, I'm seeing about a 10-20% speed increase, once I figured out how to work around joyent/node-manta#152.

If it meets with your approval, you can probably nudge the logging output around easily enough using the events that Cuttlefish gives you. I'd also recommend adding a flag to enable the timingDebug option in Cuttlefish, it's really useful for debugging issues.

isaacs commented 10 years ago

Fixes #4

bahamas10 commented 10 years ago

Thanks for the pr... i'll have free time around the holidays to test this and make sure everything works as expected... so far it looks good though! I'm glad to see the command line options have been unified with node-manta style

bahamas10 commented 10 years ago

I have more testing to do, but i'm noting some of the issues i have found...

1.

--dry-run and --version are ineffective now

2

error happens if a file being sent is 0 bytes

~/temp/foo $ echo -n > empty
~/temp/foo $ echo > not-empty
~/temp/foo $ b
~/node-manta-sync $ node manta-sync.js ~/temp/foo ~~/stor/foo
building local file list...
local file list built, 2 files found

send empty...
send not-empty...
not-empty sent
not-empty sent
    UploadTimeoutError: request took too long to send data
    at ClientRequest.onResponse (/Users/dave/dev/forks/node-manta-sync/node_modules/manta/node_modules/restify/lib/clients/http_client.js:133:38)
    at ClientRequest.g (events.js:175:14)
    at ClientRequest.EventEmitter.emit (events.js:95:17)
    at HTTPParser.parserOnIncomingClient (http.js:1663:21)
    at HTTPParser.parserOnHeadersComplete [as onHeadersComplete] (http.js:122:23)
    at CleartextStream.socketOnData (http.js:1556:20)
    at CleartextStream.read [as _read] (tls.js:502:12)
    at CleartextStream.Readable.read (_stream_readable.js:320:10)
    at EncryptedStream.write [as _write] (tls.js:366:25)
    at doWrite (_stream_writable.js:219:10)
File: empty
Task: [object Object]
done!
1 sent, 0 matched, 0 deleted, 1 errors
Synced completed in 45399 ms

TritonDataCenter / node-manta-sync

Refactor to use Cuttlefish #5

1.

2