Extensions: consider integration with language native package managers

[sipsma]

With the release of the Go client SDK, use is as easy as importing a package, just like you would any other. In all likelihood, the other languages will start with a similar approach.

The problem is when we introduce extensions, in our current plans this will break and instead:

1. Codegen (i.e. via go generate) will create the client SDKs
2. dagger.json will manage dependencies+versions

The reason for this is that extensions need to be importable cross-language. It's obviously not possible to import Python code into Go; that's where our magic tooling comes into place.

There's a decent chance this approach that skirts around language native package managers (i.e. go modules and npm) is totally fine. We can do all sorts of things to make this as nice+painless as possible. But it will still always not be "language-native" and will be at least a small learning curve. It also rules out use of stuff like Dependabot and other dependency management tooling.

---

There are technically paths to integrating with language native package managers while also supporting imports of other languages. The precise details will vary from lang-to-lang and there are multiple possible approaches, but here's a sketch of one possibility in Go:

1. Use our vanity URL go.dagger.io/dagger to point to servers running in our cloud
2. That server sees the path you are trying to import, can identify that it points to an extension (i.e. maybe a git repo, maybe a universe extension, etc.)
   • Our backend will grab that extension, generate the client bindings on the fly into a git repo and then respond to the request with that
   • This of course will all be built on dagger, so we get nice caching and stuff
   • E.g. you might import go.dagger.io/github.com/foo/my-cool-extension-written-in-typescript, which hits our servers and results in Go bindings for the typescript extension being generated on the fly and returned to you.
     • I think this will work even w/ Google's default proxying of everything through their servers, shouldn't need to have users disable that.

Open questions:

1. How would you do an import of a locally defined extension?
   • May still need local codegen to support these use cases
2. How does versioning work from one package manager to another?
   • I.e. if your extension is written in go, can we always translate the version of your module to a reasonable version of an npm package for its client bindings?
3. Is the same approach of importing from urls with dynamically generated content feasible in every other package manager we might need to support (npm, pip, etc.)?

Pros:

1. Language-native. Nothing new to learn if you already know how to use the language.
   • It's highly debatable, but I think there's a good chance this would be the best possible DX (ignoring all the practical concerns of implementing it).
2. Integration with existing tooling like Dependabot
3. Avoids questions of how we are going to cleanly translate from the current "just import the SDK" experience to the future codegen experience.

Cons:

1. Lots of work multiplied by each language we support.
2. Don't have control over the experience of dependency management anymore (i.e. the interface we are making w/ dagger project in the CLI and whatever we do similarly in the browser). We have to defer to each language's dependency management and accept the problems that come with it.
3. Not sure how extending types from a different extension (i.e. extend Directory) works in general in every language.
   • Languages that support type-classes or equivalent concepts (i.e. Rust) could support this I think, but not sure how it'd work in Go for example. Can't add methods to a type from a different package.
   • Vague idea: Maybe something clever can be done with the codegen clients to work around this though. Like maybe in Go there could by a just .With(...) method on types that accepts an interface that can be implemented from other code generated packages. Like the optional pattern but baked even further into the functionality of our codegen.
     • Would need to figure out the same in every other language too though.
   • Crazy+almost-certainly-bad idea: have the cloud backend generate the entire api in a single import statement. Would require registering your project (and all of its schemas) in our cloud first, which is a big hurdle to just import a library

[sipsma]

Ancient issue, no longer relevant