cloudflare / workers-sdk

โ›…๏ธ Home to Wrangler, the CLI for Cloudflare Workersยฎ
https://developers.cloudflare.com/workers/
Apache License 2.0
2.72k stars 715 forks source link

๐Ÿ› BUG: [ERROR] A request to the Cloudflare API during d1 execute from .sql file #2544

Closed Tombarr closed 1 year ago

Tombarr commented 1 year ago

Which Cloudflare product(s) does this pertain to?

D1

What version of Wrangler are you using?

2.7.1

What operating system are you using?

Mac M1 13.1

Describe the Bug

Wrangler D1 databases can be created but running wrangler d1 execute "kaios-apps-dev" --file ./apps.sql crashes.

๐ŸŒ€ Mapping SQL input into an array of statements
๐ŸŒ€ Parsing 5952 statements
๐ŸŒ€ Executing on kaios-apps-dev (8325d3a1-9fac-485e-a7b4-c5486ba4ada0):

This is a sample row from the apps.sql file I am trying to upload.

PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
DROP TABLE IF EXISTS apps;
CREATE TABLE apps (
    id VARCHAR(20) NOT NULL PRIMARY KEY CHECK (length(id) <= 20),
    name VARCHAR NOT NULL CHECK (length(name) > 0),
    summary VARCHAR,
    display VARCHAR,
    description VARCHAR,
    subtitle VARCHAR,
    version VARCHAR NOT NULL,
    bundle_id VARCHAR,
    google_bundle_id VARCHAR,
    developer VARCHAR,
    developer_url VARCHAR,
    manifest_url VARCHAR,
    jio_manifest_url VARCHAR,
    v3_manifest_url VARCHAR,
    thumbnail_url VARCHAR,
    background_url VARCHAR,
    default_locale VARCHAR,
    screenshots VARCHAR,
    theme VARCHAR(7) CHECK (length(theme) <= 7),
    category VARCHAR,
    type VARCHAR(10) CHECK (length(type) <= 10),
    size INTEGER CHECK (size IS NULL OR size >= 0),
    packaged_size INTEGER CHECK (packaged_size IS NULL OR packaged_size >= 0),
    paid BOOLEAN CHECK (paid IN (NULL, 0, 1)),
    priority INTEGER CHECK (priority >= 0 OR priority IS NULL),
    release_date INTEGER NOT NULL CHECK (release_date >= 0),
    ad BOOLEAN CHECK (ad IN (NULL, 0, 1)),
    hidden BOOLEAN CHECK (hidden IN (NULL, 0, 1)),
    silent BOOLEAN CHECK (silent IN (NULL, 0, 1)),
    cursor BOOLEAN CHECK (cursor IN (NULL, 0, 1)),
    fullscreen BOOLEAN check (fullscreen IN (NULL, 0, 1)),
    package_path VARCHAR,
    origin VARCHAR,
    short_name VARCHAR,
    start_url VARCHAR,
    activities VARCHAR,
    chrome VARCHAR,
    permissions VARCHAR,
    messages VARCHAR,
    dependencies VARCHAR,
    CHECK (
        manifest_url IS NOT NULL OR
        jio_manifest_url IS NOT NULL OR
        v3_manifest_url IS NOT NULL
    )
);
INSERT INTO apps VALUES('UxappJMyyWGDpPORzsyl','PodLP','Discover, listen, and subscribe to podcasts','PodLP','Discover, listen, and subscribe to podcasts from around the world for free. PodLP is designed specifically for KaiOS to be easy to use both online and offline.','Listen and subscribe to podcasts','3.1.4','com.podlp.podlp',NULL,'PodLP','https://podlp.com','https://api.kaiostech.com/apps/manifest/UxappJMyyWGDpPORzsyl','https://api.kai.jiophone.net/v2.0/apps/manifest/UxappJMyyWGDpPORzsyl','https://api.kaiostech.com/apps/manifest/UxappJMyyWGDpPORzsyl','https://storage.kaiostech.com/v3.0/files/app/L/BJcBEI8Am6_f9gA58E9jJJYAmJ62x-Oy09hys-/ICON_IMAGE.png','https://storage.kai.jiophone.net/v2.0/files/app/v/nIykXKxCrJL-uWwZZ0NuNV7p5NjRllWs11AYPK/BG_IMAGE.jpeg','en-US','["https://storage.kaiostech.com/v3.0/files/app/g/Xzl34fQDaZZYd3TKb21xDtR9OpHa-6odZxoitP/SCREENSHOT_IMAGE.png","https://storage.kaiostech.com/v3.0/files/app/s/DIcbNgqycvYuzqgvB9LxHhvmJ1gfrhVEbPssEL/SCREENSHOT_IMAGE.png","https://storage.kaiostech.com/v3.0/files/app/f/XtDjcxe8jaPuZbNz5lYMwukQKQb-AJGYhqXzzD/SCREENSHOT_IMAGE.png","https://storage.kaiostech.com/v3.0/files/app/t/6_cJj0eFhyJaPpWV5eDfGXE9ZOsuGpitvTLDmG/SCREENSHOT_IMAGE.png","https://storage.kaiostech.com/v3.0/files/app/c/fUe6e_u5cjaRwpiuzEwYbWe-OcVk2mmp2B5Wxa/SCREENSHOT_IMAGE.png"]','#005665','Education','privileged',1168054,398619,0,1,1647917981895,0,0,0,NULL,NULL,'https://storage.kaiostech.com/v3.0/files/zip/2/ThUZpW5fbpT-5_ynXcw7Otw8z4PRLn69TG78hV/3.1.4_APP_ZIP_FILE.zip',NULL,'PodLP',NULL,NULL,NULL,NULL,NULL,NULL);
COMMIT;

If I keep the transaction block, it hangs indefinitely at "๐ŸŒ€ Mapping SQL input into an array of statements," and I have to terminate the node process. If I remove the transaction block, I get this crash:

Error 9000: exception caught in D1 core. unreachable RuntimeError: unreachable
      at wasm://wasm/00f3cca6:wasm-function[4802]:0x31da3f
      at wasm://wasm/00f3cca6:wasm-function[97]:0xfe6c
      at wasm://wasm/00f3cca6:wasm-function[301]:0x3ef55
      at wasm://wasm/00f3cca6:wasm-function[3556]:0x2199e2
      at wasm://wasm/00f3cca6:wasm-function[3511]:0x214653
      at wasm://wasm/00f3cca6:wasm-function[3282]:0x1e7202
      at wasm://wasm/00f3cca6:wasm-function[3279]:0x1da6fb
      at wasm://wasm/00f3cca6:wasm-function[172]:0x20f6f
      at wasm://wasm/00f3cca6:wasm-function[169]:0x1efba
      at wasm://wasm/00f3cca6:wasm-function[4957]:0x33d39c [code: 7500]

OS: Mac Ventura 13.1 (M1) Node: v18.13.0 (also tested 16.16.0) npm: 8.19.3 (and 8.11.0 for Node 16 LTS) nvm: 0.39.0 wrangler: 2.7.1 (tested locally & globally on 2.6.1 & 2.5.0)

Tombarr commented 1 year ago

I've been able to successfully run commands, but it still seems like executing CREATE or INSERT statements are no-ops. For instance, I'm running the much simpler command:

WRANGLER_LOG=debug npx wrangler d1 execute kaios-apps-dev --file ./test.sql   

test.sql

BEGIN TRANSACTION;
CREATE TABLE apps (
    id VARCHAR(20) NOT NULL PRIMARY KEY CHECK (length(id) <= 20),
    name VARCHAR NOT NULL CHECK (length(name) > 0),
    summary VARCHAR,
    display VARCHAR,
    description VARCHAR,
    subtitle VARCHAR,
    version VARCHAR NOT NULL,
    bundle_id VARCHAR,
    google_bundle_id VARCHAR,
    developer VARCHAR,
    developer_url VARCHAR,
    manifest_url VARCHAR,
    jio_manifest_url VARCHAR,
    v3_manifest_url VARCHAR,
    thumbnail_url VARCHAR,
    background_url VARCHAR,
    default_locale VARCHAR,
    screenshots VARCHAR,
    theme VARCHAR(7) CHECK (length(theme) <= 7),
    category VARCHAR,
    type VARCHAR(10) CHECK (length(type) <= 10),
    size INTEGER CHECK (size IS NULL OR size >= 0),
    packaged_size INTEGER CHECK (packaged_size IS NULL OR packaged_size >= 0),
    paid BOOLEAN CHECK (paid IN (NULL, 0, 1)),
    priority INTEGER CHECK (priority >= 0 OR priority IS NULL),
    release_date INTEGER NOT NULL CHECK (release_date >= 0),
    ad BOOLEAN CHECK (ad IN (NULL, 0, 1)),
    hidden BOOLEAN CHECK (hidden IN (NULL, 0, 1)),
    silent BOOLEAN CHECK (silent IN (NULL, 0, 1)),
    cursor BOOLEAN CHECK (cursor IN (NULL, 0, 1)),
    fullscreen BOOLEAN check (fullscreen IN (NULL, 0, 1)),
    package_path VARCHAR,
    origin VARCHAR,
    short_name VARCHAR,
    start_url VARCHAR,
    activities VARCHAR,
    chrome VARCHAR,
    permissions VARCHAR,
    messages VARCHAR,
    dependencies VARCHAR
);
INSERT INTO apps VALUES('velPJ_36DVeV4P1e72ma','jiogames',NULL,'JioGames','Have fun by playing the exciting games','Have fun playing exciting games!','2.6.0.7',NULL,NULL,'Jio','http://www.jio.com',NULL,'https://api.kai.jiophone.net/v2.0/apps/manifest/velPJ_36DVeV4P1e72ma',NULL,'https://storage.kai.jiophone.net/v2.0/files/app/I/P3SpFVYpzWUPv4T8WiCtDqYHcDz_D-EDCXOYKK/ICON_IMAGE.png','https://storage.kai.jiophone.net/v2.0/files/app/I/Y6vhEjmMECVEZjb0oGAZw7pYeF2v7Tx_RJQrki/BG_IMAGE.png','en-US',NULL,'#63B208','Games','privileged',5072597,1572644,0,0,1644844359448,NULL,0,0,NULL,NULL,'https://storage.kai.jiophone.net/v2.0/files/app/s/gazFr5CRlWn3QBe7bnoRNtAWV7K21UjyAY8E6w/2.6.0.7_APP_ZIP_FILE.zip',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
CREATE TABLE locales (
    id VARCHAR(20) NOT NULL CHECK (length(id) <= 20),
    language VARCHAR NOT NULL,
    name VARCHAR NOT NULL,
    subtitle VARCHAR,
    description VARCHAR
);
INSERT INTO locales VALUES('velPJ_36DVeV4P1e72ma','as-IN','เฆœเฆฟเฆ…โ€™เฆ—เง‡เฆฎเงเฆš','Play highly engaging games','Rich collection of games across genres. Create top scores and get featured in leaderboards');
COMMIT;

In this example, I've removed CHECK constraints and Foreign Keys, and reduced ~5k INSERT commands to two rows. There's a warning at the top of the console output:

Failed to load .env file ".env": Error: ENOENT: no such file or directory, open '.env'
    at Object.openSync (node:fs:585:3)
    at Object.readFileSync (node:fs:453:35)

But execution continues

๐ŸŒ€ Mapping SQL input into an array of statements
๐ŸŒ€ Parsing 6 statements
Retrieving cached values for account from node_modules/.cache/wrangler
...
๐ŸŒ€ Executing on kaios-apps-dev (e6f3ef99-d12b-4315-ac90-8bade2796835):
-- START CF API REQUEST: POST https://api.cloudflare.com/client/v4/accounts/9c8d0030925ba2d451646f8b9d581d4e/d1/database/e6f3ef99-d12b-4315-ac90-8bade2796835/query
..
RESPONSE: {
  "result": [
    {
      "results": [],
      "success": true,
      "meta": {
        "duration": 0.32485300302505493,
        "last_row_id": null,
        "changes": null,
        "served_by": "primary-e6f3ef99-d12b-4315-ac90-8bade2796835.db3",
        "internal_stats": null
      }
    }
  ],
  "result_info": null,
  "success": true,
  "errors": [],
  "messages": []
}
-- END CF API RESPONSE
๐Ÿšฃ Executed 1 command in 0.32485300302505493ms

But when I check the D1 dashboard in on the Cloudflare website, it shows "0 tables," and when I query for the result, nothing shows up.

 npx wrangler d1 execute kaios-apps-dev --command "SELECT * FROM sqlite_schema" 

Returns the following

๐ŸŒ€ Mapping SQL input into an array of statements
๐ŸŒ€ Parsing 1 statements
๐ŸŒ€ Executing on kaios-apps-dev (e6f3ef99-d12b-4315-ac90-8bade2796835):
๐Ÿšฃ Executed 1 command in 0.20165200531482697ms
โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
โ”‚ type  โ”‚ name                     โ”‚ tbl_name โ”‚ rootpage โ”‚ sql                                                            โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ table โ”‚ d1_kv                    โ”‚ d1_kv    โ”‚ 2        โ”‚ CREATE TABLE d1_kv (key TEXT PRIMARY KEY, value TEXT NOT NULL) โ”‚
โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
โ”‚ index โ”‚ sqlite_autoindex_d1_kv_1 โ”‚ d1_kv    โ”‚ 3        โ”‚                                                                โ”‚
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
Tombarr commented 1 year ago

Update (1/20/23): I originally thought the issue was related to character encodings, but I've been able to upload the database using d1 execute in chunks of a few hundred rows per upload. It seems the WASM RuntimeError happening in D1 core may be related to the number of INSERT statements or the size of the file. For reference, the entire SQL file I tried uploading was ~3.7mb.

penalosa commented 1 year ago

Thanks for flagging this. In an effort to remove stale issues so we can have better signal on current friction points, weโ€™re closing out issues that have not been updated within the last 6 months. If this is still a problem, please feel free to open a new issue referencing this one. Please be sure to include an updated summary and any additional context and we will review. Thank you!