Expanded Shared Memory Management

[mspi92]

Looking at the Examples, i can see the usage of Postgres Shared Memory, where the Data-Structures are defined in size at compile time. A very useful addition to the pgx Project, would be a shared memory management if possible.

Say, i want to create an in memory key-value store in Postgres Memory, which you can access via functions. The easiest implementation, would be, to have a HashMap Datatype. This is possible right now, but one has to predefine how many elements the hashmap can store at maximum. Of course this can lead to various problems, either overflow of memory, if too less elements were chosen, or waste of memory, if too many are chosen (as HashMap element count has to be in Power of 2).

One Solution, circumventing that (and which is possible in C at least (have already implemented something like this)) is creating a memory pool. During Initialization the Extension Claims a Chunk of x MB (up to GB) Shared Memory and can allocate/deallocate all kinds of things in it, for example a dynamically allocated HashMap (and its element)

This Extension uses a similar approach, though in a less dynamic fashion: https://github.com/knizhnik/imcs

Unfortunately i am not an Expert in Rust (more of a C/C++ Guy), and i don't now if a) it is possible to implement such a memory pool manager and more importantly b) it is possible to create objects like a HashMap or other container objects with a custom allocator / deallocator in Rust at runtime

If it is possible i would like to support with the implementation of such feature, as this takes down a big "infrastructural" problem in getting started with Postgres Extensions

[jamessewell]

Hello!

(Absent) Author of the shared memory stuff here.

What you described is what I wanted to do originally, but it's not quite smooth sailing - this code kept me up at night for many months and went through many iterations.

I'd say the current implementation was a functional first cut to prove we could at least play in this area safely.

The limiting factors that I faced were:

• You can't (currently) use more than one memory manager at a time in Rust (you can only set the global allocator). There is work happening here, I need to check back in on where it is - I'm sure I saw Vec<Alloc>

• We need to be able to not only have a custom allocator, we need all allocations to able to fail (we can't grow past our shared memory chunk). This makes the std lib Vec hard to use.

• Until const generics exist a lot of this stuff is just painful in practice

I want to get back onto this at some stage, and when I do I'd love to be able to bounce ideas around a group of people who have practical use cases. I really can't promise when this will be though, I'm lost in Kubernetes land currently. I'd say much later this year if const generics land.

Maybe some of the other contributors have ideas in this area? @Hoverbear @zombodb ?

[mspi92]

Thanks for your answer, so by the reads of it, right now this isn't possible. Maybe i can share some Key-Learnings regarding the Handling of Shared and "Normal" Memory in C++ and Postgres (which might also help in Rust)

1. You don't want to mix Memory. Shared Memory is Shared Memory and Local Memory is Local Memory. If you have Polymorphic Allocators you'll lose track if you/the compiler already has "copied" the object into the right memory space. (especially from local -> shared)
2. Reference Counting and all kinds of "objects" don't work in Local Memory. As the MemoryContext can be released the Destructors of Objects are never called.