search

cloudflare / workers-sdk

โ›…๏ธ Home to Wrangler, the CLI for Cloudflare Workersยฎ
https://developers.cloudflare.com/workers/
Apache License 2.0
2.43k stars 601 forks source link

๐Ÿš€ Feature Request: support deployments with dynamically generated wrangler.toml (or runtime config in another format) #5579

Open IgorMinar opened 3 months ago

IgorMinar commented 3 months ago

Describe the solution

NuxtHub, Nuxt, and Nitro would like to be able to hide / abstract away wrangler.toml configuration for full-stack app deployments via wrangler pages deploy and Pages CI.

In the ideal scenario, Nitro (powering Nuxt and NuxtHub) would as part of the build generate a config file (wrangler.toml or more likely a subset of it stored in a simpler format), and this configuration would be picked up by wrangler when the application is deployed to pages.

The configuration supported in this way is limited to JS runtime config (compat flags and date and bindings config). This would enable to frameworks to completely manage runtime configuration on behalf of the user without exposing them to often unnecessary deployment configuration.

Without any other changes, the only way to approximate this behavior now is by the framework generating wrangler.toml and writing it into the source directory of the project, which increases noise for developers and creates a potential for clobbering developer's modifications.

// @pi0 @atinux

penalosa commented 3 months ago

This mostly works today, by generating a wrangler.toml file in the source directory of the project during the build. I'm not sure I'm quite clear on the specific ask here:

I wonder if wrangler.toml is the wrong abstraction for frameworks to use here, since it sounds like there's a need for much greater control over the output than users would want/need directly. Maybe supporting frameworks outputting a _worker.bundle file directly might be the way to go here? That would allow them to fully customise the wrangler.toml parsing and metadata construction behaviour, as well as completely bypass Wrangler's bundling & module rules.

pi0 commented 3 months ago

Supporting _worker.bundle would be fantastic for Nitro-based deployments.

IgorMinar commented 3 months ago
  • Increasing noise for developers: Frameworks can generate a dynamic wrangler.toml file during the build, which users would never see

@penalosa you'd see it if you were doing direct deploy rather than using the pages CI to deploy.

IgorMinar commented 3 months ago

Maybe supporting frameworks outputting a _worker.bundle file directly might be the way to go here?

That's what I'm thinking as well. The build output will then become environment specific (unlike today when it's environment-agnostic since _worker.bundle is created during deployment based on env selection), but that's ok.

pi0 commented 3 months ago

This mostly works today, by generating a wrangler.toml file in the source directory of the project during the build. I'm not sure I'm quite clear on the specific ask here:

Just made an experimental change to try this (https://github.com/unjs/nitro/pull/2355) as a short-time hotfix.

Here is the simple repo for a nitro app: https://github.com/pi0/nitro-app-cf

If you want to try the CI locally, you can use pnpm install > CI=1 pnpm build and notice it will generate a top-level wrangler.toml file (it is dynamically generated by nitro)

From build logs, notice that No Pages config file found is thrown before build command.

build.log ```txt 2024-04-10T23:26:44.40083Z Cloning repository... 2024-04-10T23:26:45.07849Z From https://github.com/pi0/nitro-app-cf 2024-04-10T23:26:45.079085Z * branch 4aeb7435e3d33a368d2c18cf6a326bded14c048f -> FETCH_HEAD 2024-04-10T23:26:45.079195Z 2024-04-10T23:26:45.108142Z HEAD is now at 4aeb743 add node_compat 2024-04-10T23:26:45.108869Z 2024-04-10T23:26:45.203127Z 2024-04-10T23:26:45.20368Z Using v2 root directory strategy 2024-04-10T23:26:45.229732Z Success: Finished cloning repository files 2024-04-10T23:26:46.815361Z 2024-04-10T23:26:46.902351Z โœ˜ [ERROR] No Pages config file found 2024-04-10T23:26:46.902616Z 2024-04-10T23:26:46.902701Z 2024-04-10T23:26:46.906712Z ๐Ÿชต Logs were written to "/root/.config/.wrangler/logs/wrangler-2024-04-10_23-26-46_607.log" 2024-04-10T23:26:46.917517Z No wrangler.toml file found 2024-04-10T23:26:47.077817Z Detected the following tools from environment: pnpm@8.7.1, nodejs@18.17.1 2024-04-10T23:26:47.862979Z Installing project dependencies: pnpm install 2024-04-10T23:26:48.570437Z Lockfile is up to date, resolution step is skipped 2024-04-10T23:26:48.60017Z Progress: resolved 1, reused 0, downloaded 0, added 0 2024-04-10T23:26:48.671753Z Packages: +396 2024-04-10T23:26:48.672033Z ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2024-04-10T23:26:49.601184Z Progress: resolved 396, reused 0, downloaded 137, added 137 2024-04-10T23:26:50.604868Z Progress: resolved 396, reused 0, downloaded 392, added 391 2024-04-10T23:26:50.768457Z Progress: resolved 396, reused 0, downloaded 396, added 396, done 2024-04-10T23:26:50.971695Z .../esbuild@0.20.2/node_modules/esbuild postinstall$ node install.js 2024-04-10T23:26:51.083311Z .../esbuild@0.20.2/node_modules/esbuild postinstall: Done 2024-04-10T23:26:51.19905Z 2024-04-10T23:26:51.199352Z devDependencies: 2024-04-10T23:26:51.199573Z + nitropack <- nitropack-nightly 2.10.0-28546505.8e88dd41 2024-04-10T23:26:51.199723Z 2024-04-10T23:26:51.204207Z 2024-04-10T23:26:51.204433Z > @ prepare /opt/buildhome/repo 2024-04-10T23:26:51.204553Z > nitro prepare 2024-04-10T23:26:51.20467Z 2024-04-10T23:26:52.053025Z Done in 3.8s 2024-04-10T23:26:52.134518Z Executing user command: npm run build 2024-04-10T23:26:52.932187Z 2024-04-10T23:26:52.932464Z > build 2024-04-10T23:26:52.932676Z > nitro build 2024-04-10T23:26:52.932811Z 2024-04-10T23:26:53.65194Z [success] [nitro] Generated public dist 2024-04-10T23:26:53.680071Z [info] [nitro] Building Nitro Server (preset: `cloudflare-pages`) 2024-04-10T23:26:54.820655Z [success] [nitro] Nitro Server built 2024-04-10T23:26:54.849683Z โ”œโ”€ dist/_worker.js/chunks/routes/index.mjs (745 B) (380 B gzip) 2024-04-10T23:26:54.849938Z โ”œโ”€ dist/_worker.js/chunks/routes/index.mjs.map (818 B) (294 B gzip) 2024-04-10T23:26:54.850091Z โ”œโ”€ dist/_worker.js/chunks/runtime.mjs (204 kB) (46.2 kB gzip) 2024-04-10T23:26:54.850315Z โ”œโ”€ dist/_worker.js/chunks/runtime.mjs.map (4.44 kB) (826 B gzip) 2024-04-10T23:26:54.850501Z โ””โ”€ dist/_worker.js/index.js (144 B) (148 B gzip) 2024-04-10T23:26:54.850652Z ฮฃ Total size: 210 kB (47.8 kB gzip) 2024-04-10T23:26:54.854427Z [success] [nitro] You can preview this build using `npx wrangler pages dev dist/` 2024-04-10T23:26:54.854736Z [success] [nitro] You can deploy this build using `npx wrangler pages deploy dist/` 2024-04-10T23:26:54.895632Z Finished 2024-04-10T23:26:54.896085Z Found _worker.js in output directory. Uploading. 2024-04-10T23:26:55.760071Z Attaching additional modules: 2024-04-10T23:26:55.766192Z โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” 2024-04-10T23:26:55.766466Z โ”‚ Name โ”‚ Type โ”‚ Size โ”‚ 2024-04-10T23:26:55.766566Z โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค 2024-04-10T23:26:55.766656Z โ”‚ chunks/routes/index.mjs โ”‚ esm โ”‚ 0.73 KiB โ”‚ 2024-04-10T23:26:55.766783Z โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค 2024-04-10T23:26:55.766916Z โ”‚ chunks/runtime.mjs โ”‚ esm โ”‚ 199.15 KiB โ”‚ 2024-04-10T23:26:55.767054Z โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ 2024-04-10T23:26:55.811841Z โœจ Compiled Worker successfully 2024-04-10T23:26:55.881448Z Found _routes.json in output directory. Uploading. 2024-04-10T23:26:55.893425Z Validating asset output directory 2024-04-10T23:26:56.574643Z Deploying your site to Cloudflare's global network... 2024-04-10T23:27:00.7183Z Uploading... (0/1) 2024-04-10T23:27:01.670487Z Uploading... (1/1) 2024-04-10T23:27:01.670967Z โœจ Success! Uploaded 1 files (1.41 sec) 2024-04-10T23:27:01.671156Z 2024-04-10T23:27:02.030702Z โœจ Upload complete! 2024-04-10T23:27:04.14264Z Success: Assets published! 2024-04-10T23:27:06.252786Z Success: Your site was deployed! ```
penalosa commented 3 months ago

@pi0 Because the wrangler.toml file is generated during the build, build-time environment variables won't be injected (which is what that initial error message refers to). However, bindings, compatibility flags, and runtime environment variables should all work for the deployed project

pi0 commented 3 months ago

@penalosa After some trial and error (and some pointers re wrangler validation thanks to @atinux) I found with a default name (any value is that ok?) and pages_build_output_dir (tested here) it only works with after build.

But it has the issue that if a framework wants to auto-enable a certain flag (like nodejs_als) it makes the bindings defined by user in UI, not work anymore and disabled because as I could understand, the auto-generated wrangler.toml becomes the only new source of trust. It makes it essentially a breaking change for our users when they configure bindings through UI.

Would _worker.bundle and _build.metadata.json in https://github.com/cloudflare/workers-sdk/pull/5580 allow to support both user config and framework auto generated config combined?

(btw thank you so much for the quick response โค๏ธ)

IgorMinar commented 3 months ago

@pi0 oof, I was hoping that we could avoid this because this will result in weird "source of truth" issues. This is why we design the wrangler.toml support in Pages to override and not augment the dash config.

Now I understand some of your comments. It seems that what you really want is an ability to intercept the runtime config creation in wrangler and augment it. Is that right?

penalosa commented 3 months ago

@pi0 we intentionally lock the dashboard in cases like this (where we would have to merge config from a file and from the UI), because it's not always clear how that merging should work. For instance, what if a user added some compatibility flags via the dashboard? In the Nuxt case ideally that list would be merged with the Nuxt provided list so that nodejs_compat was preserved, but that's not the only way to do that merging (potentially in another scenario the user would want UI flags to override config flags entirely). Because of that potential for conflict, when Pages config is managed via a config file we lock the dashboard UI to prevent confusing states.

However, what the linked PR would give you is a way to manage the merging of user config (from a file) and framework config yourself, within the framework build command. This would let frameworks individually decide on the specific logic they want to use for merging, although it wouldn't let users edit things through the dashboard.