How can we bundle Unison/codebase definitions with a ucm release?

[aryairani]

This is relevant to both IOSource definitions and base lib.

[ChrisPenner]

Let's make sure we're clear about the issues we want to solve:

• pulling base is pretty slow, and this is an annoyance all users encounter when creating new codebases
• base is necessary to do basically anything, so we should make it as readily available as possible (GHC ships base with the compiler for example)
• Since there's a relationship between ucm version and base version (via builtins) it would be good to ensure that UCM is using a base version it's compatible with

Additionally:

• IOSource needs to be compiled the first time it's used on a given UCM startup, which unnecessarily slows things down. It'd be nice if we could just bundle the compiled versions somehow.

One option we discussed:

• Bundle a codebase with all of the definitions from the corresponding base release as a data-file with UCM, the definitions would be included, but there wouldn't be any names in the root namespace. Then we use this as a starting place for every new codebase, it'll appear to be a "clean" codebase, but any pull of base will be fast since all the definitions are pre-loaded. How do we provide names for IOSource without just polluting the global namespace though?

[ChrisPenner]

I looked into using lift in Template Haskell to lift the compilation of the IOSource typechecked unison file to compile-time, but in order to derive Lift for a typechecked unison file I need to derive it for all dependencies of that type; which includes ShortByteString and that depends on GHC.Prim.ByteArray#, which only got a Lift instance 4 months ago so it won't be available in our base.

Here's the broken attempt at lifting: https://github.com/unisonweb/unison/tree/cp/lifted-unison-file https://gitlab.haskell.org/ghc/ghc/-/commit/83f5841b32ef11c54ff1e24dd5b92bc3c77f6250

So that particular approach is probably a dead-end for now at least, it also required littering the codebase with a ton of lift derivations, but that's maybe not a big deal. I may look into just bundling a codebase next.

[ChrisPenner]

I looked a bit into a codebase bundling approach, I think we could use this to bundle a "kick-start" codebase with definitions from base, but I'm not sure how it relates to the IOSource problem, which as I understand it right now is this:

• IOSource is used for a few things:
  • It contains the code and names with which to execute builtins.mergeio
  • It is used to lookup the current references for things like IsTest metadata by name (so code which uses it is less fragile)
• Due to the way we currently look up these refs means that the first time we lookup the value of a reference by name we actually compile IOSource in the background of whatever we're doing (often running tests or displaying docs) which artificially slows those things down.

We also have this problem:

• Doc literals (E.g. {{ hi }} desugar into names rather than references, these names need to always be in-scope somehow, (usually by having a copy of base), but if you don't have base in scope you get weird errors.

[ChrisPenner]

I tried another approach where I just lift the derived references for the specific things we need.

This was still a pain because we can't lift Hash or ByteString, but I was able to middle-man through Text to get through the lift.

It works, but it's extremely unergonomic for development, easy to make a mistake, requires a bit of unsafe code, and we'd still need to parse the IOSource code at runtime anyways to get a code lookup for the Runtime to use, so this isn't worth it I don't think. This broken version is here