Incomplete csv file

[natcohen]

(see attached for files)

I have the test.json (attached named test.json.txt for Github compatibility issue) that I would like to transform to test.csv (the output is attached). Here is what I do:

import fs from 'node:fs';
import { Transform } from '@json2csv/node';

(async ()=>{
  const input = fs.createReadStream(`test.json`, { encoding: 'utf8' });
  const output = fs.createWriteStream(`test.csv`, { encoding: 'utf8' });
  const parser = new Transform({ fields }); // This is the list of 49 headers
  input.pipe(parser).pipe(output);

  await new Promise((resolve, reject) => {
    parser
      .on('end', () => resolve())
      .on('error', (err) => reject());
  });
})()

The csv file is generated but while the json has 684 objects, the csv has only 628 lines, it always stops at the same object. If I print lines like this:

let counter = 0;
await new Promise((resolve, reject) => {
    parser
      .on('line', (line) => {
        counter += 1;
        console.log(counter);
        console.log(line);
      })
      .on('end', () => resolve())
      .on('error', (err) => reject());
  });

Each and every line gets displayed and the counter is 684 (as expected!). How can this happen? Why is it not writing everything in the file?

I use version 6.1.3 of @json2csv/node on Node 16.18.0 on a Mac m1.

[juanjoDiaz]

That's odd.

The end event indicates that the reading part of the Transform is done. You should use the finish event to know when the write part is done.

Also, you probably want to wait until the output is done writing.

  input.pipe(parser).pipe(output);

  await new Promise((resolve, reject) => {
    parser.on('error', (err) => reject());
    output.on('finish', (err) => resolve());
  });
})()

Let me know if it was that. Otherwise, I'll look deeper into it 🙂

[natcohen]

That worked!!! Didn't realize I should have listened to the output and not the end of the parser.

Thank you very much!