wvanbergen / node-vertica

Pure javascript Vertica database client. Except it is written in CoffeeScript.
MIT License
44 stars 30 forks source link

Pass through callbacks in CopyInResponse method #39

Closed themadcreator closed 10 years ago

themadcreator commented 10 years ago

This enables users to receive notification when a COPY query has consumed the current data. This can be used to stream bulk data efficiently using a COPY query.

Example:

# `buffered` is a stream-like object that can emit 'data', 'done', and 'error' events and can be resumed when more data is needed.
copyIn = (transfer, success, fail) ->
  buffered.on 'data',  (data) -> transfer(data, -> buffered.resume())
  buffered.on 'done',  ()     -> success()
  buffered.on 'error', (err)  -> fail(err)
  buffered.resume()

# Now we can stream in pipe-separated values into the DB
db.copy("COPY #{schema.table} (#{schema.columns.join(',')}) FROM STDIN DELIMITER '|' NULL '' ABORT ON ERROR", copyIn)

NOTES:

I regenerated the src using coffeescript 1.7.1, but some of the javascripts resources appeared to update even though I did not modify the coffee source.

wvanbergen commented 10 years ago

Looks good. Any chance you can add a functional test to https://github.com/wvanbergen/node-vertica/blob/master/test/functional/copy_test.coffee ?

themadcreator commented 10 years ago

Unit tests added.

wvanbergen commented 10 years ago

Looks like the unit test failed on node 0.8:

  1) Vertica.Connection#copy should accept callbacks in a data handler function and call them in order:
     Uncaught 
  AssertionError: 
    at Query.copySQL [as callback] (/home/travis/build/wvanbergen/node-vertica/test/functional/copy_test.coffee:81:14)
    at Query.onReadyForQuery (/home/travis/build/wvanbergen/node-vertica/src/query.coffee:70:12)
    at process.startup.processNextTick.process._tickCallback (node.js:245:9)

Are you using any node 0.10 specific features?

wvanbergen commented 10 years ago

If this can only be supported in node 0.10 or higher, you can wrap the test in if require('semver').gte(process.version, '0.10.0') (see the test below). I'd prefer it if it also works in 0.8 though.

themadcreator commented 10 years ago

Updated. You'll notice that in order for the callbacks to be called, we must use the signature:

socket.write(buffer, callback)

instead of

socket.write(buffer, null, callback)

because node v0.8.* does not check for the second signature type.

themadcreator commented 10 years ago

Also, I'm not sure what vertica versions you run your tests on, but the copy_test failure tests return an error code of V1001 for me, not 08000 (we are on vertica 6). Also, your error messages appear to be custom, so running these mocha tests won't work on any vertica instance other than yours.

themadcreator commented 10 years ago

Oops -- left in debugger statement accidentally.

wvanbergen commented 10 years ago

The tests run against Vertica 7 (because that's the one available as community edition, so I can use it for an open source project like this). I guess the error codes have changed between those versions?

themadcreator commented 10 years ago

I see. If I make more changes, I'll set one of those up.