search

TritonDataCenter / dragnet

event stream analysis
MIT License
11 stars 4 forks source link

dn index-scan of file with --counters produces error #36

Open davepacheco opened 8 years ago

davepacheco commented 8 years ago 
dn index-scan --counters --index-config=/assets/poseidon/stor/dragnet/muskie_logs/indexconfig.json --interval=hour --before=2016-01-12T17:00:00.000Z --after=2016-01-11T23:00:00.000Z input < $MANTA_INPUT_FILE > stdout

where the "input" datasource is:

# dn datasource-show -v input
DATASOURCE           LOCATION                                                   
input                file://dev/stdin                                           
    dataFormat: "json"
    timeField:  "time"

produces:

/dragnet/bin/dn:915
        stream.vsHead().vsWalk(function (s) {
               ^
TypeError: Object #<PassThrough> has no method 'vsHead'
    at dnPrintCounters (/dragnet/bin/dn:915:9)
    at PassThrough.<anonymous> (/dragnet/bin/dn:964:4)
    at PassThrough.EventEmitter.emit (events.js:117:20)
    at _stream_readable.js:919:16
    at process._tickCallback (node.js:419:13)

The reason is that we need to wrap this PassThrough using vstream: https://github.com/joyent/dragnet/blob/91039bfbd09d7d7095a13793ab133850a2e1fac5/lib/datasource-file.js#L702

the way we do in these places: https://github.com/joyent/dragnet/blob/91039bfbd09d7d7095a13793ab133850a2e1fac5/lib/datasource-file.js#L203 https://github.com/joyent/dragnet/blob/91039bfbd09d7d7095a13793ab133850a2e1fac5/lib/datasource-file.js#L637

We should also add a test for this.