manipulating existing codecs / versioning

[harrysolovay]

Feature idea: utils for manipulating existing codecs for the sake of versioning.

Let's say we're using the following codec to model an event.

v1.ts

export const $superhero = $.object(
  $.field("pseudonym", $.str),
  $.optionalField("secretIdentity", $.str),
  $.field("superpowers", $.array($.str)),
)

In v2, perhaps we'd like to remove the superpowers field. However, our users may still want to interact with previous events, hence we must keep the original codec around. Yes, we could define the new version as an entirely new codec.

v2.ts

export const $superhero = $.object(
  $.field("pseudonym", $.str),
  $.optionalField("secretIdentity", $.str),
- $.field("superpowers", $.array($.str)),
)

However, we'll encounter quite a bit of duplication, especially if bumping is frequent. Perhaps there's a simpler path?

v2.ts

import { $superhero as $superheroV1 } from "./v1.ts"

export const $superhero = $.migration($superheroV1, [
  $.deleteField("superpowers"),
])

This would be especially useful in cases where the means and validity of change is dubious.

import { $superhero as $superheroV1 } from "./v1.ts"

export const $superhero = $.migration($superheroV1, [
  $.renameField("superpowers", "powers"),
])

In most cases, we'd be unable to determine whether the rename was actually a field deletion followed by a field addition. If modeled with a migration codec factory + instructions, we could know how to transform previous versions.

Some potential use cases:

• Simplifying FRAME metadata codec definitions
• Modeling smart contract upgrades in TS (enables preservation of old storage locations / aka., no need to spend any funds on explicit storage migrations –– structural sharing instead)

[tjjfvi]

I don't think "migrations" are a problem we should solve in scale. Scale should be a minimal language for defining codecs, and these kinds of migrations add a lot of conceptual complexity and do not scale very well.

Changes like the "$.renameField" are cross-compatible, as they encode in the exact same way, so there's no reason to keep the old codecs around.

If one really wants to have multiple versioned codecs, most of the time, the changes will be additive. In this case, you can simply write

const $superheroV1 = $.object(
  $.field("pseudonym", $.str),
  $.optionalField("secretIdentity", $.str),
  $.field("superpowers", $.array($.str)),
)

const $superheroV2 = $.object(
  $superheroV1,
  $.field("defeated", $.array($supervillain)),
)

Modeling smart contract upgrades in TS (enables preservation of old storage locations / aka., no need to spend any funds on explicit storage migrations –– structural sharing instead)

The only way this would work is if the storage was a versioned union in the first place, i.e.

// v1
const $superhero = $.taggedUnion("version", [
  $.variant("1", $superheroV1),
])

// v2
const $superhero = $.taggedUnion("version", [
  $.variant("1", $superheroV1),
  $.variant("2", $superheroV2),
])

At this point, though, there are other, likely simpler, solutions. For example, if we went back to allowing $.option($.field(...)) to represent $.optionalField(...), we could have

const $extensible = $.option($.never);

// v1
const $superhero = $.object(
  $.field("pseudonym", $.str),
  $.optionalField("secretIdentity", $.str),
  $.field("superpowers", $.array($.str)),
  $extensible,
)

// v2
const $superhero = $.object(
  $.field("pseudonym", $.str),
  $.optionalField("secretIdentity", $.str),
  $.field("superpowers", $.array($.str)),
  $.option($.object(
    $.field("defeated", $.array($supervillain)),
    $extensible,
  )),
)

[harrysolovay]

these kinds of migrations add a lot of conceptual complexity and do not scale very well.

This is a very good point.

Changes like the "$.renameField" are cross-compatible

How would you feel about a utility that does this compatibility check for the sake of versionless API development?

if we went back to allowing $.option($.field(...)) to represent $.optionalField(...)...

Is this something you've been considering?