search

mirage / io-page

IO memory page library for Mirage backends
ISC License
10 stars 20 forks source link

caml_alloc_pages does not instruct the GC of how much memory could be free by a GC #38

Open rixed opened 7 years ago

rixed commented 7 years ago

For some reasons (I guess: because of the page alignment constraint) io-page uses its own allocator for bigarrays, which seems to replace OCaml caml_ba_alloc. But caml_ba_alloc was itself calling caml_alloc_custom, which would instruct the GC the amount of memory that could be reclaimed when finalizing that bloc. As imperfect as this is, it is still preferable than not letting the GC know that there are potentially a lot of memory to be reclaimed by GCing the bigarrays.

In my test program that does a lot of IO within 128MiB of RAM and dies from out-of-memoriness after a few minutes, this caml_alloc_pages holds around 20MiB alone, making it the second memory holder after caml_aligned_malloc (the mallocer for the heap) which holds 70MiB.

I think calling caml_alloc_custom or something to the same effect would improves the situation. I'm not even sure those blocs are freed at all (and I'm setting up another experiment to check this).

rixed commented 7 years ago

I'm not even sure those blocs are freed at all (and I'm setting up another experiment to check this).

I checked this, and indeed many allocs made by this function are freed, so this concern was unjustified. Still much too many pages are alloced by io-page when my test program dies.

hannesm commented 2 years ago

AFAICT this package uses caml_ba_alloc_dims which calls out to caml_ba_alloc, which in turn calls caml_alloc_custom_mem with the size indication (as explained in https://github.com/ocaml/ocaml/pull/1738). Is this still an issue (@talex5 who mentioned this recently)?

talex5 commented 2 years ago

caml_ba_alloc does pass the size, but only since https://github.com/ocaml/ocaml/pull/11022 was merged a couple of months ago. Before that, it treated the size as zero for pre-allocated memory.

hannesm commented 2 years ago

Ah, thanks @talex5. Would you know what change would be needed in io-page to pass this information to the GC in earlier OCaml versions (or should we just wait until those with the information get released / popular)?

talex5 commented 2 years ago

You'd have to call caml_alloc_custom_mem directly, but since this issue is open since 2016 it can probably wait a little longer for the next OCaml release.