Potential memory leak

[msoxzw]

Memory leak occurs, at least for Python 3.10.3 on Windows.

import pylsqpack

for _ in range(65536):
    decoder = pylsqpack.Decoder(4096, 16)
    del decoder

import pylsqpack

for _ in range(65536):
    encoder = pylsqpack.Encoder()
    del encoder

[msoxzw]

Additionally, "https://github.com/aiortc/aioquic/blob/main/src/aioquic/_buffer.c" and "https://github.com/aiortc/aioquic/blob/main/src/aioquic/_crypto.c" suffer memory leak, as well.

The following log manifests the top 15 memory blocks allocated by Python after my simple "DNS over HTTP/3" proxy baesed on aioquic 0.9.19 ran for 10 hours or so.

315314 memory blocks: 14282.4 KiB
  File "aioquic\h3\connection.py", line 534
    decoder, headers = self._decoder.feed_header(stream_id, frame_data)
4329 memory blocks: 6730.2 KiB
  File "aioquic\quic\crypto.py", line 114
    self.aead = AEAD(aead_cipher_name, key, iv)
4329 memory blocks: 6628.8 KiB
  File "aioquic\quic\crypto.py", line 116
    self.hp = HeaderProtection(hp_cipher_name, hp)
641 memory blocks: 5298.3 KiB
  File "aioquic\h3\connection.py", line 312
    self._encoder = pylsqpack.Encoder()
76189 memory blocks: 2976.1 KiB
  File "aioquic\quic\packet_builder.py", line 101
    self._buffer = Buffer(PACKET_MAX_SIZE)
641 memory blocks: 2804.4 KiB
  File "aioquic\h3\connection.py", line 307
    self._decoder = pylsqpack.Decoder(
33588 memory blocks: 1312.0 KiB
  File "aioquic\quic\connection.py", line 2232
    buf = Buffer(data=plain)
32093 memory blocks: 1253.6 KiB
  File "aioquic\quic\connection.py", line 723
    buf = Buffer(data=data)
15040 memory blocks: 1194.0 KiB
  File "aioquic\h3\connection.py", line 546
    encoder, frame_data = self._encoder.encode(stream_id, headers)
25526 memory blocks: 997.1 KiB
  File "aioquic\h3\connection.py", line 149
    buf = Buffer(capacity=frame_length + 2 * UINT_VAR_MAX_SIZE)
23581 memory blocks: 921.1 KiB
  File "aioquic\asyncio\server.py", line 59
    buf = Buffer(data=data)
14871 memory blocks: 580.9 KiB
  File "aioquic\h3\connection.py", line 835
    buf = Buffer(data=stream.buffer)
11279 memory blocks: 503.8 KiB
  File "aioquic\h3\connection.py", line 532
    decoder, headers = self._decoder.resume_header(stream_id)
12463 memory blocks: 486.8 KiB
  File "aioquic\h3\connection.py", line 928
    buf = Buffer(data=stream.buffer)
7879 memory blocks: 307.8 KiB
  File "aioquic\quic\packet.py", line 392
    param_buf = Buffer(capacity=65536)

[jlaine]

Let's just focus on pylsqpack, issues in aioquic are out of scope and should be reported in the correct repository.

What gives you the impression there is a memory leak here, have you tried running valgrind?

The decoder's destructor calls lsqpack_dec_cleanup: https://github.com/aiortc/pylsqpack/blob/a4b9e8c445625a075fab09aec263d05e7e668150/src/pylsqpack/binding.c#L105

The encoder's destructor calls lsqpack_enc_cleanup https://github.com/aiortc/pylsqpack/blob/a4b9e8c445625a075fab09aec263d05e7e668150/src/pylsqpack/binding.c#L356

[msoxzw]

I feel sorry that I have not yet run valgrind.

After running aforementioned code snippet, pylsqpack.Decoder will occupy approximately 300 MiB memory, and pylsqpack.Encoder will occupy roughly 550 MiB memory, which should be released.

In addition, from above memory consumption log generated by a long-running process, not only pylsqpack.Decoder, pylsqpack.Encoder, but also pylsqpack.Decoder.feed_header, pylsqpack.Decoder.resume_header, do not release nontrivial memory that is no longer needed.

[jlaine]

Hi @msoxzw could you try #15 and let me know if it fixes your issue?

[msoxzw]

It has not yet been completely fixed.

First, pylsqpack.Decoder and pylsqpack.Encoder still do not free some memory which is much less than the prior's

Second, pylsqpack.Decoder.feed_header, pylsqpack.Decoder.resume_header, and pylsqpack.Encoder.encode do not release memory, as well.

Third, it is worthy to investigate the memory allocation and de-allocation of every methods in pylsqpack.Decoder and pylsqpack.Encoder.