mtth / avsc

Avro for JavaScript :zap:
MIT License
1.27k stars 147 forks source link

Update `snappy` examples in wiki for `snappy` 7.x.x (async) #436

Closed thomastoye closed 1 year ago

thomastoye commented 1 year ago

BlockDecoder

The wiki for BlockDecoder (link) has an example for using snappy. It uses Buffer#slice, which is deprecated. It also mixes up async (callback-based) and sync (throwing an Error) error handling. I propose to update the example in the wiki to

const crc32 = require('buffer-crc32');
const snappy = require('snappy');

const blockDecoder =   new avro.streams.BlockDecoder({
    codecs: {
      null: (buf, cb) => cb(null, buf),
      snappy: async (buf, cb) => {
        // Avro appends checksums to compressed blocks.
        const len = buf.length
        const checksum = buf.subarray(len - 4, len)

        try {
          const inflated = await snappy.uncompress(buf.subarray(0, len - 4))

          if (!checksum.equals(crc32(inflated))) {
            // We make sure that the checksum matches.
            throw new Error('invalid checksum')
          }

          cb(null, inflated)
        } catch (err) {
          cb(err)
        }
      },
    },
  })

BlockEncoder

The wiki for BlockEncoder (link) relies on the old snappy sync behaviour. snappy.compress is now an async call (snappy 7.x.x). I propose to update the example to

const blockEncoder =   new avro.streams.BlockEncoder(someType, {
    writeHeader: 'auto',
    codec: 'snappy',
    codecs: {
      null: (buf, cb) => {
        cb(null, buf)
      },
      snappy: async (buf, cb) => {
        try {
          // Avro requires appending checksums to compressed blocks.
          const checksum = crc32(buf)
          const deflated = await snappy.compress(buf)

          const block = Buffer.alloc(deflated.length + 4)
          deflated.copy(block)
          checksum.copy(block, deflated.length)
          cb(null, block)
        } catch (err) {
          cb(err)
        }
      },
    },
  })

(Note that both examples also include the null codec, which is technically not needed for the example, but helps as an extra example on how to implement codecs)

mtth commented 1 year ago

Thanks @thomastoye; examples updated.