PDI allocates new memory that is never freed everytime a variable is exposed

[jbigot]

Report by Emilio J. Padrón on Slack:

@Romain and I are using PDI in a long numeric simulation and we have noticed an important memory leak. After a minimal mem profiling I think the issue comes from PDI allocating new memory every time a variable is exposed, memory that is not freed ever. Is it normal (I mean, the expected behavior)?

Just with a small proof of concept I can reproduce this memory leak: (our issue is with a Fortran code, but I obtain the same behavior in C)

#include <pdi.h>

int main(int argc, char* argv[]) { PDI_init(PC_parse_path("example6.yml"));

int my_int = 0; float my_float = 0; int my_loop = 100000;

while (my_loop > 0) { PDI_multi_expose("event_between", "an_int", &my_int, PDI_OUT, "a_float", &my_float, PDI_OUT, NULL); my_int += 1; my_float += 0.1; my_loop -= 1; }

PDI_finalize();

return 0; }

> with this simple yaml file:
``` yaml
data:
  an_int: int
  a_float: float

plugins:
  trace: ~

This is a snapshot taken with Massif (Valgrind):

And this is the report from Valgrind Memcheck, warning about the memory being allocated every PDI expose that is never freed:

==366206==
==366206== HEAP SUMMARY:
==366206==     in use at exit: 48,006,358 bytes in 300,048 blocks
==366206==   total heap usage: 7,300,528 allocs, 7,000,480 frees, 829,731,027 bytes allocated
==366206==
==366206== 160 bytes in 1 blocks are possibly lost in loss record 23 of 31
==366206==    at 0x4839F2F: operator new(unsigned long) (vg_replace_malloc.c:417)
==366206==    by 0x493C891: PDI::Scalar_datatype::clone_type() const (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x48EF431: PDI::Data_descriptor_impl::share(void*, bool, bool) (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492C397: PDI_share (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492C95B: PDI_multi_expose (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x109209: main (example6.c:12)
==366206==
==366206== 4,346 (104 direct, 4,242 indirect) bytes in 1 blocks are definitely lost in loss record 29 of 31
==366206==    at 0x48397B5: malloc (vg_replace_malloc.c:380)
==366206==    by 0x496D5A5: PC_parse_file (in /opt/pdi/lib/libparaconf.so.0.4.6)
==366206==    by 0x496D75A: PC_parse_path (in /opt/pdi/lib/libparaconf.so.0.4.6)
==366206==    by 0x109186: main (example6.c:4)
==366206==
==366206== 15,999,840 bytes in 99,999 blocks are definitely lost in loss record 30 of 31
==366206==    at 0x4839F2F: operator new(unsigned long) (vg_replace_malloc.c:417)
==366206==    by 0x493C891: PDI::Scalar_datatype::clone_type() const (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x48EF431: PDI::Data_descriptor_impl::share(void*, bool, bool) (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492C397: PDI_share (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492C95B: PDI_multi_expose (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x109209: main (example6.c:12)
==366206==
==366206== 32,000,000 bytes in 200,000 blocks are definitely lost in loss record 31 of 31
==366206==    at 0x4839F2F: operator new(unsigned long) (vg_replace_malloc.c:417)
==366206==    by 0x493C891: PDI::Scalar_datatype::clone_type() const (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x48EF431: PDI::Data_descriptor_impl::share(void*, bool, bool) (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492C397: PDI_share (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x492CB82: PDI_multi_expose (in /opt/pdi/lib/libpdi.so.1.4.0-alpha.2021-08-04.09fd084)
==366206==    by 0x109209: main (example6.c:12)
==366206==
==366206== LEAK SUMMARY:
==366206==    definitely lost: 47,999,944 bytes in 300,000 blocks
==366206==    indirectly lost: 4,242 bytes in 39 blocks
==366206==      possibly lost: 160 bytes in 1 blocks
==366206==    still reachable: 1,996 bytes in 7 blocks
==366206==         suppressed: 16 bytes in 1 blocks
==366206== Reachable blocks (those to which a pointer was found) are not shown.
==366206== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==366206==
==366206== For lists of detected and suppressed errors, rerun with: -s
==366206== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)

I am using PDI from the master branch in the repo, specifically commit 09fd0842009fa41bba9dbdbc4b027a6592698efa from August 4th

[jbigot]

In GitLab by @ksiero on Oct 18, 2021, 10:57

mentioned in commit 5e7db40d99bdbed86fba0c075a3b103169bfcdce

[jbigot]

In GitLab by @ksiero on Oct 18, 2021, 11:06

mentioned in commit 1b1baad640a53b109874d9f94fd6e2d6ed280aa1

[jbigot]

In GitLab by @ksiero on Oct 18, 2021, 13:17

mentioned in commit f6773eae768659a74ba3f98edaee3742b8bad003

[jbigot]

In GitLab by @ksiero on Oct 19, 2021, 14:59

mentioned in commit 13015e8d0a0f3af6326b854f1201b6d9c85f6445