micnews / megafunnel

artisanal big data analytics
MIT License
11 stars 0 forks source link

filter output with regular expression? #1

Open dominictarr opened 10 years ago

dominictarr commented 10 years ago

An easy low hanging fruit would be to filter the output with a regular expression. when running the /query path you could pass in a url encoded regular expression which could be used to filter particular output. If you designed your data well, this could be used to cheaply filter the out the data you don't need in a view, before it is even sent to the view engine.

@kesla @hij1nx

heapwolf commented 10 years ago

Idea: If an output stream emits raw csv lines, and the headers are well known, a through stream could split ecah line line.split(','), enumerate and match. For example...

var criteria = { 
  eventName: 'scroll', 
  x: function(val) { return val > 500; } 
};

readStream()
  .pipe(filter(headers, criteria))
  .on('data', function(d) {
    console.log(d) // => an array of values symmetrical to the headers that satisfies the criteria. 
  })
dominictarr commented 10 years ago

Also, am currently working on the subset of SQL that will work well: SELECT, WHERE, GROUP BY but not JOIN.

will123195 commented 10 years ago

sick