MEN-Mikro-Elektronik / 13MD05-90

MDIS5 System Package for Linux (including drivers)
Other
4 stars 4 forks source link

Memory leak in dbg builds #201

Closed dpfeuffer closed 4 years ago

dpfeuffer commented 4 years ago

HW: CPCI System with F26L (2c) with 2x F405 (only Z127 GPIO) Reproduce: run 'z127_io -g &' for gpio_19 and gpio_10 (see system.dsc in attached .zip) frequently leads to slowly decreasing free memory in system! or run 'sudo stress_z127 gpio_10 gpio_19' in attached zip

See MDIS_stress_z127.zip

Work to do:

  1. Reproduce the problem. Use non-debug and debug builds.
  2. Define a test setup to reproduce the problem. It would be good to use an already defined setup and create an test case that can be integrated into the regular MDIS release test. The test should cover as many MDIS libraries as possible. One possibility would be to compare used memory before/after all test case runs.
  3. Fix the problem
  4. Make some stress tests
dpfeuffer commented 4 years ago

Note for MEN: This issue belongs to MAIN_PR006793

mlassota commented 4 years ago

I used kmemleak to obtain leak info: kmemleak.log It seems that #200 should be solved first as this might be related. Also the Internet shows that entry_SYSCALL_64_after_hwframe+0x44/0xa9 might be causing problems.

dpfeuffer commented 4 years ago

Does kmemleak report only memory leaks for debug builds? I think #200 affects debug and non-debug builds. kmemleaks detects only memory leaks in kernel space, but we have to detect memory leaks also in user space. You can test again after fixing #200, but please think also about a general way to detect memory leaks in user and kernel space for a new test case.

KonradJPE commented 4 years ago

Does kmemleak report only memory leaks for debug builds?

It does not matter if debug/nodebug version is in use. This bug is present in both cases. Acctually we cannot reproduce the case with nodbg drivers without memory leak.

I think #200 affects debug and non-debug builds.

Yes, you are right. And workaround for #200 works also for this #201 issue.

KonradJPE commented 4 years ago

1 . Reproduce the problem. Use non-debug and debug builds. 3 . Fix the problem 4 . Make some stress tests

Done, after commit 415e127 we cannot reproduce memory leak in kernel / user space while running z127 simple tool in a loop.

It would be good to use an already defined setup and create an test case that can be integrated into the regular MDIS release test. The test should cover as many MDIS libraries as possible. One possibility would be to compare used memory before/after all test case runs.

We can perform additional test on OS with CONFIG_DEBUG_KMEMLEAK option enabled in kernel - this shall detect leak in kernel space. Additionally we can use valgrind to check if there is leak in test tool. As you suggest we can compare used memory before/after all test case runs, but sometimes it might be tricky to interpret results due to other running processes.

It seems that this test case can be integrated with #200.

@dpfeuffer: Currently our release functional tests run on 'nodbg' mode. Should we run some of them in 'dbg' mode?

dpfeuffer commented 4 years ago

It does not matter if debug/nodebug version is in use.

So I think it isn't necessary to perform a test with 'dbg' binaries.

KonradJPE commented 4 years ago

I have created automated stress test for z127_gpio. We cannot observe memory leaks, so I'm closing this issue.