search

cloudflare / workers-sdk

⛅️ Home to Wrangler, the CLI for Cloudflare Workers®
https://developers.cloudflare.com/workers/
Apache License 2.0
2.56k stars 659 forks source link

🐛 BUG: `[7501]` error after a successfully complete migration... #3421

Open uceumice opened 1 year ago

uceumice commented 1 year ago

Which Cloudflare product(s) does this pertain to?

D1

What version of Wrangler are you using?

3.1.0

What operating system are you using?

Linux (ubuntu-latest/github-actions)

Describe the Bug

I have got a strange error when running migrations on recently created, empty D1 database. Both migrations listed below where applied and I can confirm it inspecting the database. All required tables were created with right types and right indexes, however I still got an error telling otherwise.

After rerunning the action manually, the migrations status was confirmed as already applies and no further migrations were run: ✅ No migrations to apply!.

Here is the run logs, from the point of applying the migrations:

Run pnpm run db.apply
  pnpm run db.apply
  shell: /usr/bin/bash -e {0}
  env:
    PNPM_HOME: /home/runner/setup-pnpm/node_modules/.bin
    CLOUDFLARE_API_TOKEN: ***

> @*****/*****@ db.apply /home/runner/work/*****/*****/apps/*****/*****
> wrangler d1 migrations apply blog

▲ [WARNING] Processing wrangler.toml configuration:

    - D1 Bindings are currently in alpha to allow the API to evolve before general availability.
      Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose
      Note: Run this command with the environment variable NO_D1_WARNING=true to hide this message

      For example: `export NO_D1_WARNING=true && wrangler <YOUR COMMAND HERE>`

--------------------
🚧 D1 is currently in open alpha and is not recommended for production data and traffic
🚧 Please report any bugs to https://github.com/cloudflare/workers-sdk/issues/new/choose
🚧 To request features, visit https://community.cloudflare.com/c/developers/d1
🚧 To give feedback, visit https://discord.gg/cloudflaredev
--------------------

Migrations to be applied:
┌────────────────────────────┐
│ name                       │
├────────────────────────────┤
│ 0000_panoramic_sleeper.sql │
├────────────────────────────┤
│ 0001_melodic_sentry.sql    │
└────────────────────────────┘
? About to apply [2](https://github.com/*****/*****/actions/runs/*****/jobs/*****#step:*****) migration(s)
Your database may not be available to serve requests during the migration, continue?
🤖 Using default value in non-interactive context: yes
🕒 Creating backup...
🌀 Mapping SQL input into an array of statements
🌀 Parsing 4 statements
🌀 Executing on *****(ede788c4-771d-410a-91[3](https://github.com/*****/*****/actions/runs/*****/jobs/*****#step:*****)0-[4](https://github.com/*****/*****/actions/runs/*****/jobs/*****#step:*****)fefd3788ab[5](https://github.com/*****/*****/actions/runs/*****/jobs/*****#step:*****)):
🚣 Executed 4 commands in 21.27123999595[6](https://github.com/*****/*****/actions/runs/*****/jobs/*****#step:*****)42ms
┌────────────────────────────┬────────┐
│ name                       │ status │
├────────────────────────────┼────────┤
│ 0000_*****_*****.sql │ ✅       │
├────────────────────────────┼────────┤
│ 0001_*****_*****.sql    │ 🕒️     │
└────────────────────────────┴────────┘
🌀 Mapping SQL input into an array of statements
🌀 Parsing 4 statements
🌀 Executing on blog (ede[7](https://github.com/*****/*****/actions/runs/*****/jobs/*****#step:*****)[8](https://github.com/*****/*****/actions/runs/*****/jobs/*****#step:*****)8c4-771d-410a-[9](https://github.com/*****/*****/actions/runs/*****/jobs/*****#step:*****)130-4fefd3788ab5):
┌────────────────────────────┬────────┐
│ name                       │ status │
├────────────────────────────┼────────┤
│ 0000_*****_*****.sql │ ✅       │
├────────────────────────────┼────────┤
│ 0001_*****_*****.sql    │ ❌       │
└────────────────────────────┴────────┘

❌ Migration 0001_*****_*****.sql failed with the following errors:

✘ [ERROR] Internal error: Error sending request to D1 for database ede788c4-771d-4[10](https://github.com/*****/*****/actions/runs/*****/jobs/*****#step:*****)a-9130-4fefd3788ab5 [code: 7501]

If you think this is a bug then please create an issue at https://github.com/cloudflare/workers-sdk/issues/new/choose
 ELIFECYCLE  Command failed with exit code 1.
Error: Process completed with exit code 1.

0001**********.sql

CREATE TABLE `emails` (
    `id` integer PRIMARY KEY NOT NULL,
    `email` text(320) NOT NULL,
    `token` text(21) NOT NULL,
    `timestamp` integer NOT NULL
);

CREATE UNIQUE INDEX `idx_unique_token` ON `emails` (`token`);
CREATE UNIQUE INDEX `idx_unique_email` ON `emails` (`email`);

0002**********.sql

CREATE TABLE `access.access` (
    `id` integer PRIMARY KEY NOT NULL,
    `ipid` integer NOT NULL,
    `timestamp` integer NOT NULL,
    FOREIGN KEY (`ipid`) REFERENCES `access.ips`(`id`)
);

CREATE TABLE `access.ips` (
    `id` integer PRIMARY KEY NOT NULL,
    `ip` text(45) NOT NULL,
    `type` text NOT NULL
);

CREATE UNIQUE INDEX `idx_unique_ip` ON `access.ips` (`ip`);

Running SELECT name, type, sql FROM sqlite_schema; gives:

name | type | sql -- | -- | -- d1_kv | table | CREATE TABLE d1_kv (key TEXT PRIMARY KEY, value TEXT NOT NULL) sqlite_autoindex_d1_kv_1 | index | __drizzle_migrations | table | CREATE TABLE __drizzle_migrations( id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT UNIQUE, applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP NOT NULL ) sqlite_autoindex___drizzle_migrations_1 | index | sqlite_sequence | table | CREATE TABLE sqlite_sequence(name,seq) emails | table | CREATE TABLE `emails` ( `id` integer PRIMARY KEY NOT NULL, `email` text(320) NOT NULL, `token` text(21) NOT NULL, `timestamp` integer NOT NULL ) idx_unique_token | index | CREATE UNIQUE INDEX `idx_unique_token` ON `emails` (`token`) idx_unique_email | index | CREATE UNIQUE INDEX `idx_unique_email` ON `emails` (`email`) access.access | table | CREATE TABLE `access.access` ( `id` integer PRIMARY KEY NOT NULL, `ipid` integer NOT NULL, `timestamp` integer NOT NULL, FOREIGN KEY (`ipid`) REFERENCES `access.ips`(`id`) ) access.ips | table | CREATE TABLE `access.ips` ( `id` integer PRIMARY KEY NOT NULL, `ip` text(45) NOT NULL, `type` text NOT NULL ) idx_unique_ip | index | CREATE UNIQUE INDEX `idx_unique_ip` ON `access.ips` (`ip`)
rozenmd commented 1 year ago

Out of curiosity @uceumice , does this happen when you create a db with --experimental-backend?

cristiammercado commented 1 year ago

Hi @rozenmd,

I have the same problem! In my case, the database had already been created in the CloudFlare dashboard (i don't use --experimental-backend flag), but it was empty and without any data. I only ran the migrations command (wrangler d1 migrations apply my-db-name) from GitHub Actions using CLOUDFLARE_API_TOKEN secret in my CI:

▲ [WARNING] Processing wrangler.toml configuration:

    - D1 Bindings are currently in alpha to allow the API to evolve before general availability.
      Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose
      Note: Run this command with the environment variable NO_D1_WARNING=true to hide this message

      For example: `export NO_D1_WARNING=true && wrangler <YOUR COMMAND HERE>`

--------------------
🚧 D1 is currently in open alpha and is not recommended for production data and traffic
🚧 Please report any bugs to https://github.com/cloudflare/workers-sdk/issues/new/choose
🚧 To request features, visit https://community.cloudflare.com/c/developers/d1
🚧 To give feedback, visit https://discord.gg/cloudflaredev
--------------------

Migrations to be applied:
┌────────────────────────────────┐
│ name                           │
├────────────────────────────────┤
│ 0000_create_initial_tables.sql │
└────────────────────────────────┘
? About to apply 1 migration(s)
Your database may not be available to serve requests during the migration, continue?
🤖 Using default value in non-interactive context: yes
🕒 Creating backup...
🌀 Mapping SQL input into an array of statements
🌀 Parsing 3 statements
🌀 Executing on my-db-name (e2***ea6-8**5-4**4-b**0-f79e****cfc97):
┌────────────────────────────────┬────────┐
│ name                           │ status │
├────────────────────────────────┼────────┤
│ 0000_create_initial_tables.sql │ ❌       │
└────────────────────────────────┴────────┘

❌ Migration 0000_create_initial_tables.sql failed with the following errors:

✘ [ERROR] Internal error: Error sending request to D1 for database e2***ea6-8**5-4**4-b**0-f79e****cfc97 [code: 7[50](https://github.com/****/****/actions/runs/5284****86/jobs/956****365#step:5:51)1]

If you think this is a bug then please create an issue at https://github.com/cloudflare/workers-sdk/issues/new/choose
Error: Process completed with exit code 1.

I ran the command again from my PC, with the same token, and this time it showed me the same message as the issue's initiator comments:

After rerunning the action manually, the migrations status was confirmed as already applies and no further migrations were run: ✅ No migrations to apply!.

Thanks!

uceumice commented 1 year ago

no, I did not use the --experimental-backend.

rozenmd commented 1 year ago

@uceumice / @cristiammercado could you try again with the --experimental-backend? it fixes a lot of the issues D1 was having with queries running out of RAM and randomly failing.

cristiammercado commented 1 year ago

Hi @rozenmd,

When executing the command wrangler d1 migrations apply a-db-test with a database created with the flag --experimental-backend, an error occurs and it is that it is not possible to create backups with an experimental database, I leave the output of the command below:

▲ [WARNING] Processing wrangler.toml configuration:

    - D1 Bindings are currently in alpha to allow the API to evolve before general availability.
      Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose
      Note: Run this command with the environment variable NO_D1_WARNING=true to hide this message

      For example: `export NO_D1_WARNING=true && wrangler <YOUR COMMAND HERE>`
    - D1 Bindings are currently in alpha to allow the API to evolve before general availability.
      Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose
      Note: Run this command with the environment variable NO_D1_WARNING=true to hide this message

      For example: `export NO_D1_WARNING=true && wrangler <YOUR COMMAND HERE>`

--------------------
🚧 D1 is currently in open alpha and is not recommended for production data and traffic
🚧 Please report any bugs to https://github.com/cloudflare/workers-sdk/issues/new/choose
🚧 To request features, visit https://community.cloudflare.com/c/developers/d1
🚧 To give feedback, visit https://discord.gg/cloudflaredev
--------------------

Migrations to be applied:
┌────────────────────────────────┐
│ name                           │
├────────────────────────────────┤
│ 0000_create_initial_tables.sql │
└────────────────────────────────┘
√ About to apply 1 migration(s)
Your database may not be available to serve requests during the migration, continue? ... yes
🕒 Creating backup...

X [ERROR] A request to the Cloudflare API (/accounts/f7cd1e1e*******10ae71******9f2/d1/database/82*****9-c**1-4**0-8**8-6ca******673/backup) failed.

  The request is malformed: Experimental DBs do not support backups yet:
  82*****9-c**1-4**0-8**8-6ca******673 (v3) [code: 7400]

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/workers-sdk/issues/new/choose
rozenmd commented 1 year ago

Ah @cristiammercado that's a bug I need to fix - can you try run wrangler d1 migrations apply a-db-test --experimental-backend for now?

cristiammercado commented 1 year ago

Ey @rozenmd, executing the command with the flag wrangler d1 migrations apply a-db-test --experimental-backend the command did work! I'm testing it from my local, if I notice a difference in CI I will share it with you, but I am almost sure that it is the same behavior.

I leave the output of the command:

▲ [WARNING] Processing wrangler.toml configuration:

    - D1 Bindings are currently in alpha to allow the API to evolve before general availability.
      Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose
      Note: Run this command with the environment variable NO_D1_WARNING=true to hide this message

      For example: `export NO_D1_WARNING=true && wrangler <YOUR COMMAND HERE>`
    - D1 Bindings are currently in alpha to allow the API to evolve before general availability.
      Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose
      Note: Run this command with the environment variable NO_D1_WARNING=true to hide this message

      For example: `export NO_D1_WARNING=true && wrangler <YOUR COMMAND HERE>`

--------------------
🚧 D1 is currently in open alpha and is not recommended for production data and traffic
🚧 Please report any bugs to https://github.com/cloudflare/workers-sdk/issues/new/choose
🚧 To request features, visit https://community.cloudflare.com/c/developers/d1
🚧 To give feedback, visit https://discord.gg/cloudflaredev
--------------------

Migrations to be applied:
┌────────────────────────────────┐
│ name                           │
├────────────────────────────────┤
│ 0000_create_initial_tables.sql │
└────────────────────────────────┘
√ About to apply 1 migration(s)
Your database may not be available to serve requests during the migration, continue? ... yes
🌀 Mapping SQL input into an array of statements
🌀 Parsing 7 statements
🌀 Executing on a-db-test (82*****9-c**1-4**0-8**8-6ca******673):
🚣 Executed 7 commands in 0.8147279999684542ms
┌────────────────────────────────┬────────┐
│ name                           │ status │
├────────────────────────────────┼────────┤
│ 0000_create_initial_tables.sql │ ✅       │
└────────────────────────────────┴────────┘
rodbs commented 1 year ago

HI, I'm having a similar error when applying migrations. It works on local smoothly but it doesn't when it's applied via GitHub Actions This works when applied on the command line wrangler d1 migrations apply d1_turbolang --preview --experimental-backend

When applied via GH Actions

Run cd ./apps/remix-admin
  cd ./apps/remix-admin
  npx wrangler d1 migrations apply d1_turbolang --preview --experimental-backend
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    PNPM_HOME: /home/runner/setup-pnpm/node_modules/.bin
    CLOUDFLARE_API_TOKEN: ***
    CLOUDFLARE_ACCOUNT_ID: ***
▲ [WARNING] Processing wrangler.toml configuration:

    - D1 Bindings are currently in alpha to allow the API to evolve before general availability.
      Please report any issues to https://github.com/cloudflare/workers-sdk/issues/new/choose
      Note: Run this command with the environment variable NO_D1_WARNING=true to hide this message

      For example: `export NO_D1_WARNING=true && wrangler <YOUR COMMAND HERE>`

--------------------
🚧 D1 is currently in open alpha and is not recommended for production data and traffic
🚧 Please report any bugs to https://github.com/cloudflare/workers-sdk/issues/new/choose
🚧 To request features, visit https://community.cloudflare.com/c/developers/d1
🚧 To give feedback, visit https://discord.gg/cloudflaredev
--------------------

Error: Process completed with exit code 1.

I'm using drizzle-kit to generate the migrations