Change backend encoding to include type+version information?

[alecthomas]

As we start thinking about data migration more and more, one idea that came up was to encode the type+version along with values.

Currently a data type like:

data User { // v1
  name String
  age Int
}

Will be encoded as JSON like so:

{"name":"Alice","age":30}

But if the data structure is changed to:

data User { // v2
  username String
}

The old version of the struct will not be able to be decoded into the new version, and will fail with an arbitrary decoding error.

If we instead encoded the type+version into the value:

{
  "ver": "v1",
  "t": "module.User",
  "d": {
    "name": "Alice",
    "age": 30
  }
}

We would be able to give a much more useful error, or perform automatic migrations via inline "migration functions".

[deniseli]

Notes:

• Require mapping funcs to translate from old to new data versions
• Formalize the 3 step shuffle. Build in 1st class support for FTL for doing those data migrations
  • Don't need to keep knowledge of previous states forever if we have mapping funcs from old to new state
  • Alex: when you're building a module, say module A call to module B, does A include what version of B is compiled against?
  • A: yes needed
  • If you're exposing multiple versions of the same verb, roll out the new version and then triage inbound calls to call the right one. Should we consider limiting the number of versions live simultaneously?
  • A: foundational to FTL from the start, based on unison. Content hash based. Content hash of a function is its address. For internal RPC boundaries, that's a good approach. We can garbage collect old ones. As long as you're always building against latest, the old ones will die off. Real issue is persistent state.
  • Juho: how urgent?
  • A: yes, something, causes problems all the time. e.g. they rename a topic. This is a problem if they don't just wipe the db. We'll need to disallow all backwards breaking changes
  • Can we detect backward-breaking database changes?
  • A: dont have first class db support yet, so no
  • We might need to materialize data updates if we have the past->next mapping functions for data records. e.g. when a version gets garbage collected, we automatically materialize the next version of each record
  • 2 distinct types of migrations: schema change for db, backfill / actual migration of underlying data
  • We need a solution for the SQL too - manifesting the db schema into FTL before we can solve that problem too
  • Do we want an FTL ORM layer to abstract this away and hold onto new definition. At some point, developer can come in, hit big red button, halt the world, and switch to the new thing. Or shadow mode where you can read-only on one v while shuffling read-write to the other.
  • Dhanji liked ORM idea, but then we need to write an ORM in every single runtime lang, and we're competing with projects we don't want to compete with
  • Don't need a full ORM, just the parts that manage the schemas. Data access pane. Opinionated about what are the SQLC generated classes, sort of, but FTL-native. Figure out how we send this to the actual ORM, then someone builds a hook to the actual ORM.
    • Custom data source + connection for JVM. If you build your own ORM, all the other ones won't be able to use it, since they can only use data source and connection. Nontrivial amt of work.
  • If we model DB queries as verbs (e.g. SQL as a runtime) (exciting!! +!!) Define some parameters to your DB query. Could extract that as a verb, call that verb from your module no matter the language. Then you get all the benefits of verbs. Input params and return types would be FTL types. Full integration.
  • If this is a real remote call, how do you handle txns?
  • A: would need to inline. Run on the same host as the module.
  • Need impl for each runtime?
  • Wait no can we single-home the DB access? but that thing would need access to all dbs
  • but we can have multiple queries per txn
  • inlining is easiest solution. Need templates to inline per language, but that shouldn't be very complicated. Here is the data structure, iterate, then populate.
  • It might be a simple ORM but it's still an ORM. Developers would have to choose that instead of their favorite tool.
  • hard for us to add features without us being an ORM. Hard to avoid. If we don't have that requirement, we can't add anything to begin with.
  • We're not writing an ORM with a massive runtime in each language. Codegen small stubs in each lang?
  • What about impl-ing as a small sidecar thing? Local to the machine but different language? Is that too much to cross lang boundary?
    • We already know these are special SQL verbs, so we know they need to be generated inline. Matter of codegenning in each language.
  • sqldelite from cash is very similar to sqlc - sqlc is not unique
  • new FTL logo = stack of turtles?