ironSource / parquetjs

fully asynchronous, pure JavaScript implementation of the Parquet file format
MIT License
345 stars 173 forks source link

ParquetTransformer stream doesn't emit error #116

Open annfomenko opened 3 years ago

annfomenko commented 3 years ago

Please add error catcher to _transform method of ParquetTransformer class:

_transform(row, encoding, callback) {
     if (row) {
      this.writer.appendRow(row).then(callback).catch((error) => callback(error));
    } else {
      callback();
    }
}

It allows to listen error on the stream object.

IvanOfThings commented 3 years ago

@annfomenko I think your suggestion also solves the problem of founding a error when applying schema in stream process.

I would make the PR but contributors agreement is not available anymore.

Based on the current code the _transform function should look like this:

  _transform(row, encoding, callback) {
    if (row) {
      this.writer
        .appendRow(row)
        .then((d) => callback(null, d), callback)
        .catch((error) => callback(error));
    } else {
      callback();
    }
  }

Edited: It is an already open PR to solve that --> https://github.com/ironSource/parquetjs/pull/46