[FEATURE] Wash Dev v2

[thomastaylor312]

This proposal is to detail a suggested overhaul of the developer experience for those writing components for wasmCloud (although this experience would technically work for anyone working on components). Currently this is more focused on the inner dev loop for components and not providers, although what is defined here could be easily extended to providers in the near-to-medium term future.

Background

This proposal requires a bit of background context to explain the why behind it. One of the key things to understand about how wasmCloud as a project approaches design problems is that we focus on separation of concerns. This is why non-functional requirements are entirely externalized and we focus on developers writing their code while platform engineers and SREs "wire it in" to their platform with whatever dependencies, configuration, and credentials they see fit. This separation of concerns is a key feature of wasmCloud. To restate; devs should rarely care about specific dependencies they need, leaving that to their platform. This means developers should rarely be editing wadm manifests. This key detail is critical for understanding the direction in this proposal.

Key goals

With that in mind, the main thrust of the proposal is that wash dev becomes the "platform" for local devs, meaning that it should handle wiring up dependencies. So the main outcomes of this work should be:

1. By default, a developer can just run their component in wasmCloud with all of their dependencies getting hooked up to their component using a single command. They should not need to create a Wadm manifest as that is something that is done by a platform.
2. A user should be able to override these default dependencies (providers) and their configuration on a project and a global level (e.g. when a company has different defaults they use for their projects)

This proposal is intentionally scoped as small as possible so we can use it as a good foundation on which to build. I currently see this evolving in 3 stages:

• Stage 1: For a single component, everything gets hooked up automatically and run in wasmCloud (this is the scope of this proposal.) Any changes to the component source code will result in recompiling the component and updating the running version with the new changes.
• Stage 2: Expand experience to be able to handle a collection of components and providers that operate together. This will go hand-in-hand with a feature of wash build to compile multiple components or providers with one command.
• Stage 3: Add ability to specify a limited subset of things to build and run with wash dev while the other parts of the project fallback to configured defaults

Please keep in mind this proposal is only focused on that first stage.

User experience

The user experience should be as simple as this:

$ wash new component foobar
$ cd foobar
$ wash dev
Starting wasmCloud and Wadm
Building component...success!
Wiring up the bits and bobs
Launching component
Component is now available on http://localhost:8080
Component is listening for messages
Watching for changes...

The user can then edit their code and on save, the component is rebuilt and redeployed if the build is successful.

Supported automatic linking

By default, all wasi-cloud interfaces and any other common wasmCloud interfaces (like the control interface and wadm) will be supported. After build, wash will introspect the component and figure out which interfaces are being imported and exported. It will then consult its list of built-ins or configured options and generate a wadm manifest on the fly for the built component.

By default we will use the following providers:

• HTTP server and client
• NATS KV
• NATS Messaging
• Blobstore-FS (Though if we can get a NATS object store that would be nice)
• Postgres? (We'd need some sort of in memory thing to do this by default. This is not required for this first batch of work)

Once they are ready, we will also support the control interface provider and the wadm provider.

In addition to the providers, wash dev should also start the NATS secrets implementation to support secrets

NOTE: Although not part of this phase of work, when we support running a whole repo of components and providers (stage 2), wash should automatically fulfill those as well (such as linking two components that export/import an interface from another component—you can see where this gets pretty exciting and cool.)

Saving generated manifests

wash dev should have a flag, such as --output-manifest </path/to/file.yaml> that will output the generated manifest to a file. Depending on the implementation of the command wash dev SHOULD delete the application before shutting down, so this flag would allow the user to save a generated manifest for later consistent use. This generated manifest could be a candidate for deploying to production if only wasmCloud first-party providers are needed.

Configuring links and defaults

For most people starting off, these defaults should suffice. However, there will be many needs to override and/or configure the different dependencies. It can also be used to specify custom dependencies not built in to wash dev by default. I propose that this be done at a project level using the wasmcloud.toml file and at a global level using a new dev.toml file located in the wash dir. I am not married to this file name or location. It could be folded into an existing file as well. All that matters is that the data is structured properly.

This will be done by adding a new section to the wasmCloud toml that looks like something below. The global config will follow the same format. This is meant mostly to be a guideline and it may differ when actually implemented. Functionality of the config is documented inline:

[dev]
# This is an optional field that, when set, is the escape hatch. Wash will only use the manifest 
# you actually create rather than generating anything or using overrides. Wash will still watch the
# current directory, build the component, and then delete the component (so wadm reloads it) 
manifest = "wadm.yaml"
# Configuration passed to the component
config = { foo = "bar" }

# This section contains all overrides for any specific package or interface Wash should validate that
# each item is unique (e.g. if you specify a global wasi:keyvalue in one and then try to specify
# wasi:keyvalue/atomic in another). We don't have this be a map of the package name because you can 
# link an interface multiple times via link name
[dev.overrides]
# This sets config globally for keyvalue
"wasi:keyvalue@0.2.0" = { config = { "extra" = "config" } }
# This sets config only for a specific interface
"wasi:http/incoming-handler@0.2.0" = { config = { address = "0.0.0.0:9999" } }
# This overrides the provider to use. If set to a specific provider, the user would be responsible
# for ensuring the needed server was set up (e.g. Redis). Depending on how this involves, we may add
# support for some kind of hooks that could help with setup of needed dependencies
"wasi:messaging" = { uri = "oci://my.customreg.io/custom-messaging:0.1.1" } # This could also be a file as well
# An example of a custom provider or component as well as setting secrets. Secrets will have a
# special syntax where if the value is prefaced with `$ENV:` it will load the secret from and
# environment variable.
"acme:mycustom" = { uri = "file://./path/to/provider", config = { some = "config" }, secrets = { password = "supasecret", another = "$ENV:MY_SECRET_VAR"} }

# This is a section only for advanced users where they need to use a link name other than "default".
# In this example, this would be the link name of "foobar"
[dev.overrides.foobar]
"wasi:keyvalue@0.2.0" = { config = { "different" = "config" } 

Running in production

None of this affects how the component will eventually be run in production. Because everything is behind an interface, the finished component can be handed to a platform team who will create (most likely through automatic generation) a Wadm manifest or deploy it through any other process desired. wash dev is purely for local development.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this has been closed too eagerly, please feel free to tag a maintainer so we can keep working on the issue. Thank you for contributing to wasmCloud!

[vados-cosmonic]

Definitely not stale -- working on this now!

Note that wash dev is now available in the latest version of wash, v0.34.0

[EDIT] "not" -> "now"