pgcentralfoundation / pgrx

Build Postgres Extensions with Rust!
Other
3.42k stars 223 forks source link

Storing a Vec in Postgres memory #1745

Closed ccleve closed 3 days ago

ccleve commented 4 days ago

I need to cache a struct with a Vec in Relation.rd_amcache. This struct has to be tied to the memory context stored in Relation.indexctx.

I have tried leak_and_drop_on_delete(), but that generates a "pfree called with invalid pointer 0x111604f30 (header 0x0007000000000000)" error.

let mystruct = struct MyStruct {
  myfield: u32,
  myvec: Vec<u32>
};
let mut ctx = PgMemoryContexts::For((*rel).rd_indexcxt);
let x = ctx.leak_and_drop_on_delete(mystruct);

I could try to .palloc0() some memory and copy the struct into it, but that gets tricky with a Vec.

I do not need to write to the struct (or vec) once it gets cached. It's read-only.

What's my best approach here?

eeeebbbbrrrr commented 4 days ago

Based on that ERROR, it looks like (*rel).rd_indexcxt isn't a valid MemoryContext? Have you confirmed that Postgres always sets it when you get it/need it?

ccleve commented 4 days ago

The indexctx is legit.

Unfortunately, CLion won't debug break on the error and won't do a proper stack trace, so this is tough to track down.

Marching through the code manually shows that it's happening in ProcessInvalidationMessages(), which has something to do with system caching.

I'm thinking I'll just bail and do my own cache with a static mut.

workingjubilee commented 4 days ago

amusingly we literally just recently merged https://github.com/pgcentralfoundation/pgrx/pull/1737 but that doesn't quite provide enough infra for you to do what you want, I think.

ccleve commented 3 days ago

Closing this for now, as it appears to be specific to the pg system cache. Long term, the answer is some kind of tighter integration between pg memory and Rust memory.