This PR refactors Bud's architecture to what I hope will be the final major iteration for the foreseeable future.
Stepping back a minute, over the last several months, I've taken a significant detour from v0.4 Hosted Docs. So what happened?
Well, I've spent much of this time trying to figure out the right architecture to allow developers to bring their own generators. In the same way that Bud turns a directory of controllers into routes on your web server, you should be able to build your own "controller generator" that does something entirely different (e.g. GraphQL generator anyone? 🙃)
I was planning on deferring this whole custom generator business until after the framework was more feature-complete, but it's come up while trying to implement syntax transformers in the views. I want to be able to render Markdown on the documentation site but I didn't want to bake a Markdown renderer into Bud itself. You should be able to configure and even swap out the Markdown renderer with something else. Same goes with Tailwind support or React views. So that's why I'm building custom generators six versions earlier than planned 😂
I was pretty uncompromising in my goals for custom generators:
Should have the same power as the core generators (in fact core generators could be replaced!)
Should be built into bud run and bud build, no extra command should be necessary.
Should support live reloads just like the rest of Bud.
These goals led to some challenges in coming up with a solution:
Triggering project-specific Go code from the Bud binary: Go is a static, compiled language with very limited options for dynamically loading code. In order to be able to run custom code, either written by or imported by the application developer, we need a way to compile this custom code separately and communicate with it from Bud. This requirement led to the rest of the complexity.
Side note: Go's plugin mode actually fit the bill for this problem, but I lost trust in it after failing to compile a package that the default compiler had no problem with. It's also quite slow. It seems like it doesn't use the build cache nearly as aggressively as the default compiler.
Sharing the generator cache: As Bud grows larger, the generator cache has become increasingly important. Without a cache, there's a significant amount of duplicate work happening as the generators request the results of other generators triggering regenerations. I wanted a way to share the cached results across processes and bonus points if the cache could persist on disk.
It can't just be a part of the application binary: I was originally thinking maybe I could just squeeze this generated code into the user's application binary, but then you realize "oh, you need the generators to build the application binary in the first place".
After a couple of tries, I landed on the following design. I'm sharing this now because I'm fairly confident in the flexibility of this design going forward:
As a quick recap, Bud's workflows are the following:
bud generate: Generate the glue code into your app's bud/ directory. You don't typically run this directly.
bud build: Typically used for production. Essentially just runs bud generate, but where file embedding is set to true, live reload is disabled, and minify is set to true.
bud run: Typically used in development. Generates the glue code, runs your app, watches for changes, and re-triggers builds upon change.
Here how the various components fit together:
Bud CLI: This is the bud binary. It's responsible for triggering and syncing code generators, setting up the development server, watching files for changes and triggering hot reloads.
BudFS: A library within the Bud CLI. Handles all code generation and syncing the results to the bud/ directory.
Bud Server: The development server that's internal within the Bud CLI. Hot reloads are placed here so the application server can be restarted without disconnecting the browser's event source connection. We also place V8 in here during development to save about 1 second off recompiling after each file change.
AFS: Short for the Application File Server. The AFS is a plugin binary that lives in your application's bud/ directory. Conceptually, you can think of the AFS like a mounted remote directory (e.g. FUSE or an FTP server), but entirely local. BudFS generates, builds then connects to the AFS. BudFS then walks the AFS, which actually runs the core generators and custom generators together. The AFS is only used to support development and is not something you'd deploy.
BudDB: Right now this is just an SQLite3 database used for sharing a cache between Bud and the AFS in a concurrency-safe way. I initially considered using shared memory or doing what Go does with their GOCACHE directory, but SQLite ended up being the perfect tool for this job. The need to share the cache might diminish over time since the generators in BudFS are quite distinct from the generators in AFS, but I have a feeling I'll find other needs for this SQLite database over time.
App: This is your generated application server. This is what gets built into bud/app. During development, the app uses Bud Server to evaluate JS (for server-side rendering) and for opening client-side files upon request. This second feature will shift to AFS (as shown in the sketch) and may go away entirely because ESBuild is so fast. In production, there's no connection to AFS or Bud Server, it's all embedded.
Browser: This is what the end user sees. The browser makes requests to the App to render HTML and request JS. It's also connected to the local Bud Server which is how we trigger the browser to update itself when files change.
This may all seem very convoluted and confusing, but it gives us this dynamic loading capability we need while providing us with live reload. It's also surprisingly fast. Live reloads take under 50ms for frontend changes and under a second for backend changes. Initial startup time is somewhere between 250-600ms. This is all with synchronous syncs, no caching between syncs, uncached Go binaries, and roundtrips to evaluate JS code.
I still want to do a bit of cleanup after this PR and do some more testing. Once I'm confident, I'll cut a new release. You shouldn't notice any major differences, likely just different delays for backend changes. I removed the import hasher called imhash that helped us avoid rebuilding previously built binaries. This made prior changes nearly instant, but fresh changes were slower. This now makes fresh changes faster, but prior changes slower.
In the coming months, you should notice a re-focus towards new code generators (new features!) and getting the website up!
Okay, with all that said, I just want to say, thank you for your continued interest in Bud! I know it's been slow, but I hope these steps will set Bud up on the right path to eventually being your go to tool for painlessly building ambitious full-stack apps for your business. 2023 is going to be an exciting year for Bud. I really hope you'll come along for the ride.
This PR refactors Bud's architecture to what I hope will be the final major iteration for the foreseeable future.
Stepping back a minute, over the last several months, I've taken a significant detour from v0.4 Hosted Docs. So what happened?
Well, I've spent much of this time trying to figure out the right architecture to allow developers to bring their own generators. In the same way that Bud turns a directory of controllers into routes on your web server, you should be able to build your own "controller generator" that does something entirely different (e.g. GraphQL generator anyone? 🙃)
I was planning on deferring this whole custom generator business until after the framework was more feature-complete, but it's come up while trying to implement syntax transformers in the views. I want to be able to render Markdown on the documentation site but I didn't want to bake a Markdown renderer into Bud itself. You should be able to configure and even swap out the Markdown renderer with something else. Same goes with Tailwind support or React views. So that's why I'm building custom generators six versions earlier than planned 😂
I was pretty uncompromising in my goals for custom generators:
bud runandbud build, no extra command should be necessary.These goals led to some challenges in coming up with a solution:
Triggering project-specific Go code from the Bud binary: Go is a static, compiled language with very limited options for dynamically loading code. In order to be able to run custom code, either written by or imported by the application developer, we need a way to compile this custom code separately and communicate with it from Bud. This requirement led to the rest of the complexity.
Side note: Go's plugin mode actually fit the bill for this problem, but I lost trust in it after failing to compile a package that the default compiler had no problem with. It's also quite slow. It seems like it doesn't use the build cache nearly as aggressively as the default compiler.
Sharing the generator cache: As Bud grows larger, the generator cache has become increasingly important. Without a cache, there's a significant amount of duplicate work happening as the generators request the results of other generators triggering regenerations. I wanted a way to share the cached results across processes and bonus points if the cache could persist on disk.
It can't just be a part of the application binary: I was originally thinking maybe I could just squeeze this generated code into the user's application binary, but then you realize "oh, you need the generators to build the application binary in the first place".
After a couple of tries, I landed on the following design. I'm sharing this now because I'm fairly confident in the flexibility of this design going forward:
As a quick recap, Bud's workflows are the following:
bud generate: Generate the glue code into your app'sbud/directory. You don't typically run this directly.bud build: Typically used for production. Essentially just runsbud generate, but where file embedding is set to true, live reload is disabled, and minify is set to true.bud run: Typically used in development. Generates the glue code, runs your app, watches for changes, and re-triggers builds upon change.Here how the various components fit together:
budbinary. It's responsible for triggering and syncing code generators, setting up the development server, watching files for changes and triggering hot reloads.bud/directory.bud/directory. Conceptually, you can think of the AFS like a mounted remote directory (e.g. FUSE or an FTP server), but entirely local. BudFS generates, builds then connects to the AFS. BudFS then walks the AFS, which actually runs the core generators and custom generators together. The AFS is only used to support development and is not something you'd deploy.GOCACHEdirectory, but SQLite ended up being the perfect tool for this job. The need to share the cache might diminish over time since the generators in BudFS are quite distinct from the generators in AFS, but I have a feeling I'll find other needs for this SQLite database over time.bud/app. During development, the app uses Bud Server to evaluate JS (for server-side rendering) and for opening client-side files upon request. This second feature will shift to AFS (as shown in the sketch) and may go away entirely because ESBuild is so fast. In production, there's no connection to AFS or Bud Server, it's all embedded.This may all seem very convoluted and confusing, but it gives us this dynamic loading capability we need while providing us with live reload. It's also surprisingly fast. Live reloads take under 50ms for frontend changes and under a second for backend changes. Initial startup time is somewhere between 250-600ms. This is all with synchronous syncs, no caching between syncs, uncached Go binaries, and roundtrips to evaluate JS code.
I still want to do a bit of cleanup after this PR and do some more testing. Once I'm confident, I'll cut a new release. You shouldn't notice any major differences, likely just different delays for backend changes. I removed the import hasher called imhash that helped us avoid rebuilding previously built binaries. This made prior changes nearly instant, but fresh changes were slower. This now makes fresh changes faster, but prior changes slower.
In the coming months, you should notice a re-focus towards new code generators (new features!) and getting the website up!
Okay, with all that said, I just want to say, thank you for your continued interest in Bud! I know it's been slow, but I hope these steps will set Bud up on the right path to eventually being your go to tool for painlessly building ambitious full-stack apps for your business. 2023 is going to be an exciting year for Bud. I really hope you'll come along for the ride.