mtth / avsc

Avro for JavaScript :zap:
MIT License
1.29k stars 148 forks source link

docs: update decoder example #472

Closed joscha closed 2 months ago

joscha commented 2 months ago

The current example won't emit anything as the node script will be finished before the event callbacks are triggered. I updated the example to make that more obvious and easier to copy & paste into an existing project. It also surfaces errors as Promise rejections, the old example didn't attach to the 'error' event.

mtth commented 2 months ago

Thanks for the PR @joscha. In this case I think it's best to keep the example as-is: the change adds a lot of noise, diluting the example's value. Its focus should remain highlighting the avsc-specific parts: 'metadata' and contents of 'data'.

There are also many ways to use streams. For example, I typically use stream.finished rather than explicit promise-wrapping.

joscha commented 2 months ago

the change adds a lot of noise, diluting the example's value

maybe we can change it to add your stream.finished part? Then at least it can be copy & pasted and callbacks are actually fired before the script ends, like in the current example?

There are also many ways to use streams. For example, I typically use stream.finished rather than explicit promise-wrapping.

I actually wasn't able to get the encoder to work with finished. I kept on getting a:

Promise resolution is still pending but the event loop has already resolved

It did work with the decoder.

My code roughly looked like this:

  const codec: 'zstd' | 'null' = 'zstd';
  const encoder = Avro.createFileEncoder('companies.avro', registry['model.Company'], {
    codec,
    codecs: {
      ...Avro.streams.BlockEncoder.defaultCodecs(),
      zstd: (buf, cb) => {
        const checksum = crc32(buf);
        compress(buf)
          .then(deflated => {
            cb(null, Buffer.concat([deflated, checksum]));
          })
          .catch(cb);
      }
    }
  });

  for (const company of companies) {
    sanitizeEnumValues(company);
    encoder.write(company);
  }
  encoder.end()
  await finished(encoder)

~and I wasn't able to figure out my mistake. I assumed it was potentially some flushing issue from the compression? Do you have an idea?~ added a encoder.end() and it worked.