electric-sql / pglite

Lightweight Postgres packaged as WASM into a TypeScript library for the browser, Node.js, Bun and Deno
https://electric-sql.com
Apache License 2.0
4.76k stars 81 forks source link

OOB when select() with over about 1k response? #88

Closed pvdz closed 1 month ago

pvdz commented 1 month ago

Hi. Is there some kind of limit in place with how much data can be returned with a select? Or a way to increase it, if there is one? I feel like 1KiB is a really low limit in this context so I'm unsure if it's my setup or not.


I'm using Drizzle to talk to pglite in a pretty vanilla way:

const client = new PGlite();
const db = drizzlePglite(client);

I've got a table like

pgTable('stuff', { id: integer('id'), data: text('data') });

I can insert data into it, I can query this data, but if my payload exceeds what appears to be close to 1KiB then it'll throw an OOB.

RuntimeError: memory access out of bounds
 ❯ null.<anonymous> wasm:/wasm/018421ca:1:4451557
 ❯ null.<anonymous> wasm:/wasm/018421ca:1:4561193
 ❯ null.<anonymous> wasm:/wasm/018421ca:1:4561497
 ❯ null.<anonymous> wasm:/wasm/018421ca:1:1953804
 ❯ null.<anonymous> wasm:/wasm/018421ca:1:4803833
 ❯ null.<anonymous> wasm:/wasm/018421ca:1:1954597
 ❯ null.<anonymous> wasm:/wasm/018421ca:1:81379
 ❯ null.<anonymous> wasm:/wasm/018421ca:1:4791378
 ❯ null.<anonymous> wasm:/wasm/018421ca:1:1678555
 ❯ null.<anonymous> wasm:/wasm/018421ca:1:3094049

I would get this with db.select().from(stuff). Through raw queries I was able to roughly pinpoint the size to 1KiB.

This way:

await sql.raw('SELECT SUBSTRING(data, 0, 1013) AS a FROM stuff')

would be fine while this:

await sql.raw('SELECT SUBSTRING(data, 0, 1014) AS a FROM stuff')

would throw the above OOB error.

When adding id to the columns to fetch, the size drops and 1013 also throws the OOB error, leading me to think there's some kind of limit to the whole data set. So adding some overhead, it feels to me like that is awkwardly close to a precise 1KiB.

I've tried using debug output, up to level 5, but it's not showing me any relevant information when this error is thrown. Just shows the select etc.

So my question is whether I'm indeed hitting a limit somehow and how I might resolve this.

AntonOfTheWoods commented 1 month ago
  • node v18.20.2, and also v20.12.2

In Chrome on a relatively beefy laptop I'm able to easily and quickly get 10s of MB in a single select (using conn.exec), so at the very least this appears to be node-specific.

pvdz commented 1 month ago

Hmm I'm not sure. Right now I can't seem to repro this anymore. Thinking memory leak of some kind. But one that required a restart to get around? I dunno... Closing as I can't repro this at this time. Maybe somebody else finds this that can.