search

elixir-circuits / circuits_gpio

Use GPIOs from Elixir
129 stars 23 forks source link

Memory Leak #81

Closed alanj853 closed 2 years ago

alanj853 commented 4 years ago

This is a continuation of https://github.com/fhunleth/elixir_ale/issues/76.

Also, I have a separate branch for the circuits_gpio switch in my example project: https://github.com/alanj853/gpio_bug/tree/circuits_gpio

I retested with circuits_gpio as requested, and there is still a memory leak, but not as significant. Here are the results:

+----------------------+-----------+---------------+----------+--------+----------+--------+---------+--------+ | Total | Processes | ProcessesUsed | System | Atom | AtomUsed | Binary | Code | ETS | +----------------------+-----------+---------------+----------+--------+----------+--------+---------+--------+ | 14714896 | 3494508 | 3494244 | 11220388 | 387413 | 377749 | 27384 | 6172109 | 346624 | | 14783344 | 3536028 | 3535956 | 11247316 | 387413 | 377749 | 54312 | 6172109 | 346648 | | 14783576 | 3540168 | 3539556 | 11243408 | 387413 | 377749 | 50376 | 6172109 | 346648 | | 14750824 | 3529448 | 3528836 | 11221376 | 387413 | 377749 | 28344 | 6172109 | 346648 | | 14813168 | 3544420 | 3544420 | 11268748 | 387413 | 377749 | 75736 | 6172109 | 346624 | | 14833848 | 3565360 | 3565360 | 11268488 | 387413 | 377749 | 75560 | 6172109 | 346624 | | 14833848 | 3565360 | 3565360 | 11268488 | 387413 | 377749 | 75560 | 6172109 | 346624 | | 14756728 | 3534856 | 3534856 | 11221872 | 387413 | 377749 | 28888 | 6172109 | 346624 | | 14828648 | 3560104 | 3560104 | 11268544 | 387413 | 377749 | 75560 | 6172109 | 346624 | | 14837848 | 3569332 | 3569332 | 11268516 | 387413 | 377749 | 75560 | 6172109 | 346624 | | GPIO Reading stopped | | | | | | | | | | 15276120 | 4003388 | 4003388 | 11272732 | 387413 | 377936 | 71552 | 6178789 | 347088 | | 15240752 | 3988148 | 3988148 | 11252604 | 387413 | 377936 | 51424 | 6178789 | 347088 | | 15259944 | 4007340 | 4007340 | 11252604 | 387413 | 377936 | 51424 | 6178789 | 347088 | | 15243200 | 3990596 | 3990596 | 11252604 | 387413 | 377936 | 51424 | 6178789 | 347088 | | 15244728 | 3992444 | 3992444 | 11252284 | 387413 | 377936 | 50992 | 6178789 | 347088 | | 15229120 | 3976836 | 3976836 | 11252284 | 387413 | 377936 | 50992 | 6178789 | 347088 | | 15247264 | 3994548 | 3994548 | 11252716 | 387413 | 377936 | 51424 | 6178789 | 347088 | | 15228048 | 3975332 | 3975332 | 11252716 | 387413 | 377936 | 51424 | 6178789 | 347088 | | 15222720 | 3970436 | 3970436 | 11252284 | 387413 | 377936 | 50992 | 6178789 | 347088 | | 15242880 | 3990596 | 3990596 | 11252284 | 387413 | 377936 | 50992 | 6178789 | 347088 | | 15228776 | 3976492 | 3976492 | 11252284 | 387413 | 377936 | 50992 | 6178789 | 347088 | | 15155072 | 3825260 | 3825260 | 11329812 | 387413 | 378080 | 117376 | 6186693 | 348712 | | 15174144 | 3843900 | 3843900 | 11330244 | 387413 | 378080 | 117808 | 6186693 | 348712 | | 15255776 | 3925532 | 3925532 | 11330244 | 387413 | 378080 | 117808 | 6186693 | 348712 | +----------------------+-----------+---------------+----------+--------+----------+--------+---------+--------+

More Info

Elixir and OTP Version: IEx 1.10.3 (compiled with Erlang/OTP 21) circuits_gpio version: 0.4.5 raspberry PI version: 3 Kernel: Linux raspberrypi 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux

fhunleth commented 4 years ago

Could you call :erlang.garbage_collect? I'm wondering if this is normal memory usage that would eventually correct itself when the gc runs.

If that doesn't change anything or has unconvincing results, then the only thing that I can think of doing is to progressively delete code and see where things start looking better. I'd probably start by deleting C code so that the NIF calls are still being made. Your example is so simple that it seems like the leak has to be something really basic.

alanj853 commented 3 years ago

I reran this test both with and without the :erlang.garbage_collect() over 24 hours and this is what I found.

Without :erlang.garbage_collect()

After running for about 30 mins, the memory "stabilises", and hovers over and back between the same 90 kB. It never recovers the initial jump though, but that could possibly just be accepted as "this is the extra usage that circuits_gpio will impose" without explicit garbage collection. So I am not fully convinced anymore that this is a leak.

Below is a graph with the output: without_gc

With :erlang.garbage_collect()

For the "with :erlang.garbage_collect" test, I call the :erlang.garbage_collect() just after I close the GPIO reading processes https://github.com/alanj853/gpio_bug/commit/2c956daac5794689a34f2d74c9dc67a26104ac86.

The results are interesting. The memory fluctuates a bit at the start, then drops a lot after about 2 hours, rises again about 1 hours later, rises again 2.5 hours later, and then stabilises and hovers between the same ~4.5 kB for the rest of the test. The vast majority of the memory usage is below the initial amount though, and the fluctuations are a lot smaller when :erlang.garbage_collect() is called.

Below is a graph with the output: with_gc

Conclusion

So whether there is a leak or not is as clear cut as I originally thought, making use of :erlang.garbage_collect() after closing the GPIO reading processes seems to definitely give cleaner usage of circuits_gpio, with lower memory fluctuations and lower overall memory usage.

fhunleth commented 3 years ago

I just took a look at https://github.com/alanj853/gpio_bug/blob/circuits_gpio/lib/reader.ex. I think the majority of your memory usage is going to be in your test infrastructure. You generate garbage open and writing to files and collecting stats. I suspect that the circuit_gpio usage will be quite small after this. You can minimize circuits_gpio's overhead even more if you open it once, save the reference and then use the reference. There's no need to repeatedly open and close the GPIO especially when you have a GenServer that can cache the reference.

In summary, there's no leak in your example and you're measuring your test infrastructure's memory usage. The :erlang.garbage_collect/0 call helps by freeing the memory that you allocated more quickly.

I'll wait for your response, but I think this issue should be closed. If you want to pursue this further, update your example so that you have two processes: the first one only makes calls to :circuits_gpio in a tight loop and the second one gets the pid of the first one and periodically dumps memory stats to disk on it.

alanj853 commented 2 years ago

Hi, sorry for delay on getting back to you on this.

I ran seven 24hr based on your comments: Test 0: As is now, from head of circuits_gpio branch Test 1. As is now with GPIO reading + GC commented out test 1 branch Test 2. As is now with GPIO reading commented out but with GC test 2 branch Test 3. As is now with single opening of GPIO reading w/ GC test 3 branch Test 4. As is now with single opening of GPIO reading w/o GC test 4 branch Test 5. As is now with single opening of GPIO reading w GC, with single opening of file test 5 branch Test 6. As is now with single opening of GPIO reading w/o GC with single opening of file test 6 branch

Attached are the results from each test, and graphs comparing all of them.

So you're right @fhunleth , there is no leak. I didn't run the exact test you specified, but tests 1 and 2 show the "test infrastructure" without the GPIO reading just to get an idea of the usage of it.

Few interesting points:

It's worth noting that the RPI I ran the tests on was completely wiped and was left to do nothing else only run the test. i.e. no browsing or other activity was occurring on the PI, it was even disconnected from the network while the tests ran.

But all in all, this can be closed, as there is no leak. I will leave it up to you @fhunleth if you want to close the original in https://github.com/fhunleth/elixir_ale/issues/76 or not, although I assume it can be.

Test Details Elixir and OTP Version: IEx 1.10.3 (compiled with Erlang/OTP 21) circuits_gpio version: 0.4.5 raspberry PI version: 3 Kernel: Linux raspberrypi 5.10.92-v7+ #1514 SMP Mon Jan 17 17:36:39 GMT 2022 armv7l GNU/Linux performance_test.xlsx

fhunleth commented 2 years ago

Thanks for the update and reporting all of the findings! I will go ahead and close the issues.