search

wc-duck / datalibrary

Open Source Data Library for data serialization.
Other
42 stars 8 forks source link

datalibrary allocates crazy amounts of memory when loading many type libraries #110

Closed lazy909 closed 5 years ago

lazy909 commented 5 years ago

The combination of

dl_context_load_type_library dl_ctx->member_descs = dl_grow_array( &dl_ctx->alloc, dl_ctx->member_descs, &dl_ctx->member_capacity, dl_ctx->member_count + header.member_count); and dl_grow_array size_t new_cap = ( ( old_cap < min_inc ) ? old_cap + min_inc : old_cap ) * 2;

means the library always doubles the amount of memory allocated for each library loaded.

For my libraires it allocated

  1. 30 capacity for a total of 15 members (0 + 15) * 2
  2. 60 capacity for a total of 25 members (30) * 2
  3. 120 capacity for a total of 33 members (60) * 2 etc...

After loading 18 type libraries with a total of ~120 members, dl_grow_array tried to realloc 250mb of memory

I suggest size_t new_cap = old_cap < min_inc ? min_inc * 2 : old_cap;

wc-duck commented 5 years ago

Ouch! That is definitely not correct. I'll hopefully have some time to fix that tonight!

On Tue, Mar 26, 2019, 11:52 lazy909 notifications@github.com wrote:

The combination of

dl_context_load_type_library dl_ctx->member_descs = dl_grow_array( &dl_ctx->alloc, dl_ctx->member_descs, &dl_ctx->member_capacity, dl_ctx->member_count + header.member_count ); and dl_grow_array size_t new_cap = ( ( old_cap < min_inc ) ? old_cap + min_inc : old_cap ) * 2;

means the library always doubles the amount of memory allocated for each library loaded.

For my libraires it allocated

  1. 30 capacity for a total of 15 members (0 + 15) * 2
  2. 60 capacity for a total of 25 members (30) * 2
  3. 120 capacity for a total of 33 members (60) * 2 etc...

After loading 18 type libraries with a total of ~120 members, dl_grow_array tried to realloc 250mb of memory

I suggest size_t new_cap = old_cap < min_inc ? min_inc * 2 : old_cap;

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/wc-duck/datalibrary/issues/110, or mute the thread https://github.com/notifications/unsubscribe-auth/AAXGOiqbEZYSX1F0n2vEJAtANKNmRLXfks5vafvZgaJpZM4cLI34 .

wc-duck commented 5 years ago

Issue should be solved now. I removed the use of dl_grow_array and just realloc:ed it thus always keeping it at min-size allocation.

wc-duck commented 5 years ago

Oh! And thanks for the bug-report! Its appreciated!

lazy909 commented 5 years ago

np. It's a nice library :) /Andrés Hansson