Modulizing Spin.toml files

[seungjin]

My spin.toml is getting bigger and bigger (A couple of hundred lines). If I can implement some sort of 'include' feature that allows me to modularize spin.toml's content into separate TOML files, it would be more organized and productive.

[seungjin]

Spin plugin for this?

[itowlson]

@seungjin A plugin is an interesting idea. It could be something the user just remembers to run, or I think @cardoso designed the spin let plugin (https://developer.fermyon.com/hub/preview/plugin_spinlet) to allow hooks so possibly it could be configured to automagically combine "includes" on spin let build or spin let up. But I'm not sure exactly what hooks spinlet provides.

@fibonacci1729 @michelleN I think you some ideas about this - do you have time to weigh in?

[fibonacci1729]

@seungjin I like this idea! I'm curious to hear more about your usecase and how you would like to use such a feature.

Are you thinking something like a new component section key (i.e. include) in the spin.toml that allows you to modularize components out into their own manifests, for example:

In spin.toml:

spin_manifest_version = "1"
name = "my-app"
version = "0.1.0"
trigger = { type = "http", base = "/" }

[[component]]
include = "path/to/component.toml"

Where path/to/component.toml looks like:

id = "my-component"
source = "path/to/wasm"
[trigger]
route = "/..."

# ... other component specific sections ...

[seungjin]

@fibonacci1729 Exactly what I meant. Originally, I was thinking it's somewhat like a cargo workspace. Basically, it's the same concept, as you say.

My spin.toml is currently 170 lines and growing very fast. I can see it easily expanding to a few thousand lines.

So, having a dedicated spin config for each component with a root spin file that includes statements would give me a more organized and manageable development experience.

Currently, my structure is as follows: having one single spin file for dozens (possibly hundreds of components) is a pain to manage.

# tree -L 2
.
├── assets
│   ├── x.css
│   └── yew-app
├── bots
│   ├── rssbot_nyt (*)
│   └── rssbot_wsj (*)
├── Cargo.lock
├── Cargo.toml
├── common
│   ├── Cargo.lock
│   ├── Cargo.toml
│   ├── src
│   └── target
├── my
│   ├── cert_checker (*)
│   ├── geoip (*)
│   ├── health (*)
│   ├── ifconfig (*)
│   ├── weather (*)
│   └── whoami (*)
├── root
│   ├── Cargo.lock
│   ├── Cargo.toml
│   ├── src
│   └── target
├── spin.toml
├── templates
│   ├── 404.hbs
│   ├── 500.hbs
│   ├── cert_checker
│   ├── ifconfig
│   └── root
└── todos.md

(*): Individual cargo project/wasm component. Actually I have more than above and growing fast.

[cardoso]

@seungjin A plugin is an interesting idea. It could be something the user just remembers to run, or I think @cardoso designed the spin let plugin ([developer.fermyon.com/hub/preview/plugin_spinlet]

@itowlson yep, that's exactly the sort of use case I was looking to explore first

[fibonacci1729]

@cardoso do you think this usecase is something that would be a good fit for spin let? If so, because I'm not super familiar with the implementation of spin let do you have an idea for how this might work/look as part of that experience?

Alternatively, since 2.0 is just around the corner, a few of us have been discussing baking "manifest modularity" into the manifest redesign. Something (hand-wavy) that looks like what was mentioned above. I'm going to start a proposal to further discuss this approach in the next few weeks.

[seungjin]

Spin 2 is out. Is this feature in Spin 2? I am converting/improving my project to spin 2.0. It the feature is in spin 2. love to see it.

[seungjin]

Seems not there yet. https://github.com/fermyon/spin/blob/40bbead627ef6c8f61706516e5f11f1122a33f21/crates/manifest/src/schema/v2.rs

[itowlson]

Correct: we weren't able to deliver everything in the time available, and we felt we could do this in a later minor version without breaking backward compatibility.

[seungjin]

I can wait! Thanks!!

[itowlson]

Okay, looking to define this more closely, particularly in the light of Spin 2 manifest changes.

Requirements

• Put all the manifest stuff related to a subset of the application (triggers, components, variables) into a separate TOML file and have that merged into the main spin.toml at runtime.
• Child TOMLs must be allowed to go in subdirectories.
  • If a subdirectory tree is moved, a child TOML that refers to stuff only within that subdirectory shouldn't need to change - only the reference from the main to the child should need to change.
• Validation errors arising from child TOMLs should point to the child TOML.
  • For example, if a child TOML specifies an illegal component name, the error message should tell you which file needs fixing. If two child TOMLs declare the same component name, the error message should tell you which two were involved. This is important because the user doesn't want to hunt through dozens of child TOML files.

What have I missed?

Challenges

This is going to need more than naive text inclusion, because:

• If a child TOML declares variables, those need to be merged into a single [variables] map. We can't interleave this the way we do triggers and components.
• We will need to be smart about working directories (for build) and relative references (for files, source and watch). We'll have to fix these up to be relative to the child TOML.
• Mapping validation errors back to the place of origin feels tricky. Some things (like component IDs) could be done by validating the child TOML during load, but things like duplicate IDs will need to be caught during merge. Detecting these as we merge each child in will at least give us one of the files in conflict - not necessarily a location in the file but hopefully child files will be short! - but not the second. Maybe that's enough?

Any others that come to mind?

Planning

• Is there a subset of this feature that could give us most of the benefit for minimal effort?
  • For example, suppose all IDs had to be top level, e.g. you could write [component.cart] definition = "cart/mod.toml" but not write an entire trigger-plus-component in a child file. (This would also mean a child could not include its own child files). I'm not sure this delivers though, since I imagine triggers and components will often want to go together in child files.
  • Or we could punt on the validation and accept that error messages are going to provide limited information in the first drop.

Other ideas for how to approach it?

[lann]

We have discussed the possibility of adding a "component manifest" format for standalone reusable "library" component projects. It may be that the same idea could serve this purpose as well with the component projects appearing in subdirectories rather than separate repos.

[seungjin]

After Spin 2 and getting an idea of component model, things changed. "Component manifest" format for reusable "library" component makes much sense to me. Thanks!