feat: export `Tokenizer`, document how it can be used

Closes: https://github.com/rvagg/cborg/issues/110

What do you reckon @achingbrain?

Here's my full quick example work-up, part of which I included in the README:

import { encode, decode, Tokenizer, Type } from 'cborg'

class CustomTokeniser extends Tokenizer {
  next () {
    const nextToken = super.next()
    if (nextToken.type === Type.bytes) {
      throw new Error('Unsupported type: bytes')
    }
    return nextToken
  }
}

function customDecode (data, options) {
  options = Object.assign({}, options, {
    tokenizer: new CustomTokeniser(data, options)
  })
  return decode(data, options)
}

function rt (obj) {
  console.log('--\nplaying with:', obj)
  const byts = encode(obj)
  console.log(Buffer.from(byts).toString('hex'))
  let decoded = decode(byts)
  console.log('round-trip:', decoded)
  decoded = customDecode(byts)
  console.log('custom round-trip:', decoded)
}

rt([1, true, null, 'string'])
rt({ hello: 'world', b: true, f: 1.101, i: 101, ni: -101, byt: Uint8Array.from([1, 2, 3, 4]) })

Of course you have access to Token and Type too so you can do what you want with the token stream, including turning bytes into base64 strings kind of like the Go json encoder does transparently 🤮.

class Base64Tokeniser extends Tokenizer {
  next () {
    const nextToken = super.next()
    if (nextToken.type === Type.bytes) {
      return new Token(Type.string, Buffer.from(nextToken.value).toString('base64'))
    }
    return nextToken
  }
}

Plus you could do the reverse too, on encode since the json encoder here can be overridden in various ways to perform transformations, like we use for dag-json. Whatever floats your boat. At least with access to the token stream you can halt before you waste time processing something big that's going to error later.

Hopefully enough flexibility here to do creative things without making too many holes in your foot in the process.

rvagg / cborg

feat: export `Tokenizer`, document how it can be used #111