mafintosh / csv-parser

Streaming csv parser inspired by binary-csv that aims to be faster than everyone else
MIT License
1.41k stars 134 forks source link

is mapValues called on skipped lines? #222

Open Stasmo opened 1 year ago

Stasmo commented 1 year ago

Documentation Is:

Please Explain in Detail...

I am a little unclear on whether the mapValues function is called for skipped lines.

I assumed that it would not be called for skipped lines, but when writing some tests it seems like the mapValues function is being called for a skipped line.

My test data:

t,u,v,w
a,101,1.02,2.02
b,101,1.03,2.03

My csv-parser.Options :

  opts: {
    headers: [ 'id', 'timestamp', 'x', 'y' ],
    skipLines: 1,
    mapHeaders: [Function: mapHeaders],
    mapValues: [Function: mapValues]
  }

The error I see in my mapValues function when I run my tests.

    Error during date parsing: {
      filePath: '/xxx/tests/custom-headers.csv',
      header: 'timestamp',
      value: 'u',
      index: 1,
      opts: {
        headers: [ 'id', 'timestamp', 'x', 'y' ],
        skipLines: 1,
        mapHeaders: [Function: mapHeaders],
        mapValues: [Function: mapValues]
      }
    }

Your Proposal for Changes

I think it's fair to assume that the mapValues function would skip the number of lines defined by the skipLines option.

If this is not the intent, and you don't agree that it's fair to assume that the mapValues function would not be run on skipped lines, then please close this ticket.

If it is the intent of the parser to skip these lines then either I am doing something wrong or there's a bug.

If the library was designed to run the mapValues function on skipped lines but you agree that it is a fair assumption that it shouldn't, then we should update the documentation to let users know that mapValues will be run on skipped lines.

Thanks!

Stasmo commented 1 year ago

Doesn't look like there are any tests for mapValues that use the skipLines option.