drizzle-team / drizzle-orm

Headless TypeScript ORM with a head. Runs on Node, Bun and Deno. Lives on the Edge and yes, it's a JavaScript ORM too 😅
https://orm.drizzle.team
Apache License 2.0
23.02k stars 553 forks source link

[BUG]: cannot insert many sql variables when using cloudflare d1 #2659

Open chesha1 opened 1 month ago

chesha1 commented 1 month ago

What version of drizzle-orm are you using?

0.32.0

What version of drizzle-kit are you using?

0.23.0

Describe the Bug

When inserting many sql variables, it throws an exception saying D1_ERROR: too many SQL variables at offset 435: SQLITE_ERROR.

In my case, it is 50 rows inserting into a table with 5 colomns (short text or integer) using one sentence:

await db.insert(table).values(updatedData);

If the number of rows is decreased, it works expectedly.

I am wondering if there will be any techiques behind drizzle to automatically split large sql sentence into smaller ones to meet the cloudflare d1 limits.

Expected behavior

No response

Environment & setup

No response

samuba commented 1 month ago

The Prisma people solved it exactly like you suggested by splitting large sql into smaller parts https://github.com/prisma/prisma/issues/23743

Right now I'm doing this by hand everywhere which is really cumbersome.

if (events.length > 10) {
    const chunks = chunk(events, 10);
    for (const chunk of chunks) {
        await db.insert(s.events).values(chunk);
    }
} else {
    await db.insert(s.events).values(events);
}

function chunk<T>(arr: T[], size: number): T[][] {
    const result = [];
    for (let i = 0; i < arr.length; i += size) {
        result.push(arr.slice(i, i + size));
    }
    return result;
}