scramjetorg / scramjet

Public tracker for Scramjet Cloud Platform, a platform that bring data from many environments together.
https://www.scramjet.org
MIT License
253 stars 20 forks source link

CSVParse() step issue? #100

Closed kenzik closed 3 years ago

kenzik commented 3 years ago

Should we expect the step function in the CSVParse() options to pass the result down the chain?

const { StringStream } = require('scramjet');
const fs = require('fs');
const args = process.argv.slice(2);

// seq.csv
//
// line
// 002
// 003
// 004
// 005

const opts = { 
  header: true,
  delimiter: ','
}

if(args[0]==='step') {
  opts['step'] = (results, parser) => { return results };
}

if(args[0]==='step-worker') {
  opts['step'] = (results, parser) => { return results };
  opts['worker'] = true;
}

console.log('Running with opts: ', opts);

StringStream
  .from(fs.createReadStream('./seq.csv') )
  .CSVParse(opts)
  .map( async (l) => {
    return await l;
   })
  .each( (l) => {
    console.log(l);
  })

node main - as expected node main step - nada node main step-worker - nada

Repo: https://github.com/kenzik/scramjet-csvparse-step

Goal is to dynamically inject a field in each row in the CSV to use later in the map function.

kenzik commented 3 years ago

Checked doing the same with only papaparse and had same issue. May not be supported. Not a scramjet issue.