cloudflare / workers-sdk

⛅️ Home to Wrangler, the CLI for Cloudflare Workers®
https://developers.cloudflare.com/workers/
Apache License 2.0
2.41k stars 592 forks source link

Added `d1 export --local` support #6073

Closed geelen closed 5 days ago

geelen commented 2 weeks ago

This uses a new miniflare-only debug pragma PRAGMA miniflare_d1_export to trigger the dump as D1 provides no in-Worker API (which Miniflare uses) to generate exports.

I'm open to other ways to implement this.

What this PR solves / how to test

Fixes limitation with the previous iteration of the export functionality. Previously, all d1 export --local commands would fail.

Author has addressed the following

changeset-bot[bot] commented 2 weeks ago

🦋 Changeset detected

Latest commit: 6f94501465ed021ccc24d0c4c9436df295596086

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages | Name | Type | | ------------------------------- | ----- | | miniflare | Minor | | wrangler | Minor | | @cloudflare/pages-shared | Patch | | @cloudflare/vitest-pool-workers | Patch |

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions[bot] commented 2 weeks ago

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9691836885/npm-package-wrangler-6073

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/6073/npm-package-wrangler-6073

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9691836885/npm-package-wrangler-6073 dev path/to/script.js
Additional artifacts: ```sh npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9691836885/npm-package-create-cloudflare-6073 --no-auto-update ``` ```sh npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9691836885/npm-package-cloudflare-kv-asset-handler-6073 ``` ```sh npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9691836885/npm-package-miniflare-6073 ``` ```sh npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9691836885/npm-package-cloudflare-pages-shared-6073 ``` ```sh npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/9691836885/npm-package-cloudflare-vitest-pool-workers-6073 ``` Note that these links will no longer work once [the GitHub Actions artifact expires](https://docs.github.com/en/organizations/managing-organization-settings/configuring-the-retention-period-for-github-actions-artifacts-and-logs-in-your-organization).

wrangler@3.62.0 includes the following runtime dependencies:

Package Constraint Resolved
miniflare workspace:* 3.20240620.0
workerd 1.20240620.1 1.20240620.1
workerd --version 1.20240620.1 2024-06-20

Please ensure constraints are pinned, and miniflare/workerd minor versions match.

geelen commented 5 days ago

For my own education, does this PR mean that PRAGMA miniflare_d1_export(?,?,?); is part of our public API now? I don't think we want it to be so. How do we prevent people from writing that in their own queries?

It's part of miniflare's implementation of D1, but not present anywhere else. It's not accessible from wrangler d1 execute --local since you can't pass parameters but technically if someone wrote it in a worker and passed in at least two params they'd get a sql dump in local dev (and an error in production).

We could refactor dumpSql in this PR to use the public D1 API rather than the internal DO one (and then wrangler export would just send lots of queries to the D1 rather than a single debug pragma), but that would mean it's much harder to reuse the specific dumpSql code that's present in D1's internal codebase. And so I've erred on the side of making it easier to keep them consistent and accept leaking an implementation detail.

Ideally, Miniflare would make it possible to do an end-run around the D1 shim and hit an object directly with an RPC call, but as far as I can find out that's not possible.