All values in stream _transform row are either null or undefined

[ochart2]

I return new parquet.ParquetTransformer(schema);

and then I write to the stream as normal... all the values for the document are either null or undefined when inspecting them in ParquetTransformer _transform method.

What is the correct way to use streams with parquetjs?

[ochart2]

Got it working. Thanks for the lib!

[jdhankins]

@ochart2, could you upload your solution? Thanks for your help.

[ochart2]

Looks something like this:

import { Transform } from 'stream';
import { ParquetSchema, ParquetTransformer } from 'parquetjs';

const types: any = {
    string: { compression: 'GZIP', type: 'UTF8' },
    int: { compression: 'GZIP', type: 'INT32', encoding: 'RLE', bitWidth: 7 },
    date: { compression: 'GZIP', type: 'TIMESTAMP_MILLIS', encoding: 'RLE', bitWidth: 7 },
};

const schema = new ParquetSchema({
    a: types.string,
    b: types.int,
    c: types.date,
});

export const newParquetStream = function (): Transform {
    return new ParquetTransformer(schema);
};

I was having issues reading the parquet file from Redshift... Good luck, let me know how you fare.