Closed joscha closed 2 months ago
Thanks for the PR @joscha. In this case I think it's best to keep the example as-is: the change adds a lot of noise, diluting the example's value. Its focus should remain highlighting the avsc
-specific parts: 'metadata'
and contents of 'data'
.
There are also many ways to use streams. For example, I typically use stream.finished
rather than explicit promise-wrapping.
the change adds a lot of noise, diluting the example's value
maybe we can change it to add your stream.finished
part? Then at least it can be copy & pasted and callbacks are actually fired before the script ends, like in the current example?
There are also many ways to use streams. For example, I typically use
stream.finished
rather than explicit promise-wrapping.
I actually wasn't able to get the encoder to work with finished
. I kept on getting a:
Promise resolution is still pending but the event loop has already resolved
It did work with the decoder.
My code roughly looked like this:
const codec: 'zstd' | 'null' = 'zstd';
const encoder = Avro.createFileEncoder('companies.avro', registry['model.Company'], {
codec,
codecs: {
...Avro.streams.BlockEncoder.defaultCodecs(),
zstd: (buf, cb) => {
const checksum = crc32(buf);
compress(buf)
.then(deflated => {
cb(null, Buffer.concat([deflated, checksum]));
})
.catch(cb);
}
}
});
for (const company of companies) {
sanitizeEnumValues(company);
encoder.write(company);
}
encoder.end()
await finished(encoder)
~and I wasn't able to figure out my mistake. I assumed it was potentially some flushing issue from the compression? Do you have an idea?~ added a encoder.end()
and it worked.
The current example won't emit anything as the node script will be finished before the event callbacks are triggered. I updated the example to make that more obvious and easier to copy & paste into an existing project. It also surfaces errors as Promise rejections, the old example didn't attach to the
'error'
event.