possible memory leak on DHT and on session::remove_torrent()

[GoogleCodeExporter]

Hi Arvid,

I'm running almost 2k torrents on a libtorrent::session on my proper client and 
its memory footprint points to a possible memory leak on libtorrent 0.16.0, 
given it does not reduce any MB after calling session::remove_torrent() 
function for the whole torrents set.

Starting a memory check using valgrind using only 1 torrent running on my 
client, it resulted to a leak detected on the DHT announces. (see attached file)

Testing again with the session::stop_dht() option, this memleak disappered as 
expected. 

However, as I couldn't test valgrind using a huge torrents set (the memcheck 
almost freezes my client), I still don't know why libtorrent is handling such 
memory from the removed torrents.

Any explanation or suggestion related to this issue will be very appreciated.

Thanks again,
Diogo

PS: libtorrent will save the planet :P

Original issue reported on code.google.com by mul...@gmail.com on 21 Jun 2012 at 4:22

Attachments:

• valgrind.log

[GoogleCodeExporter]

I'm attaching the test results running a hundred of torrents.

There's a suspicious point when creating torrent_info's struct.
I'll check the torrent files used which produce these leaks. 

Original comment by mul...@gmail.com on 21 Jun 2012 at 1:47

Attachments:

• valgrind100.log

[GoogleCodeExporter]

Attached there is my massif heap profile which reveals the main functions which 
are handling this amount of memory.

It points the calling of m_policy.add_peer() at on_resume_data_checked() event 
handler (torrent.cpp:1778)

Looking to the code, there is a quite strange loop, which I can't understand 
how it could be working.

torrent.cpp:1769
----------------
            using namespace libtorrent::detail; // for read_*_endpoint()
            peer_id id(0);

            if (lazy_entry const* peers_entry = m_resume_entry.dict_find_string("peers"))
            {
                int num_peers = peers_entry->string_length() / (sizeof(address_v4::bytes_type) + 2);
                char const* ptr = peers_entry->string_ptr();
                for (int i = 0; i < num_peers; ++i)
                {
                    m_policy.add_peer(read_v4_endpoint<tcp::endpoint>(ptr)
                        , id, peer_info::resume_data, 0);
                    /* Don't need to advance ptr += (sizeof(address_v4::bytes_type) + 2); ??? */
                }
            }
... /* it also repeats for peers6 */
----------------

Even with this, I don't think how this can be responsible for the whole leak 
I'm facing.

So, here is my question: When/where these peers added on m_policy are being 
released on the code? On session::remove_torrent() ?

Cheers,
Diogo

Original comment by mul...@gmail.com on 21 Jun 2012 at 8:14

Attachments:

• massif.log

[GoogleCodeExporter]

The convention among the read_*() functions are that they progress the pointer 
(or iterator) themselves. read_v4_endpoint() is supposed to do this. There's an 
upper limit on the number of peers stored in m_policy, and this list is also 
pruned every now and then. When the policy object is destructed, they're surely 
deleted (i.e. remove_torrent()). Also, when a torrent is paused the list is 
shrunk to hold a lot fewer peers.

Original comment by arvid.no...@gmail.com on 22 Jun 2012 at 2:15

[GoogleCodeExporter]

oh. one thing just occurred to me. The peer structs are all pool allocated, in 
session_impl::m_ipv4_peer_pool and session_impl::m_ipv6_peer_pool. These pools 
never shrink. There's a function called release_memory() on them, but it's not 
clear it would do much, since it would only free whole blocks where all slots 
are free.

This is probably what you're observing. You might want to dial down the max 
peer list to something more reasonable, say 1000 peers?

Original comment by arvid.no...@gmail.com on 22 Jun 2012 at 2:28

[GoogleCodeExporter]

Perfect, I imagined such behaviour.. That's all I need to know :)
I'll reduce this limit and check my memory usage.

By the way, have you looked to the DHT leak pointed on my valgrind profile ?

Thanks again

Original comment by mul...@gmail.com on 22 Jun 2012 at 2:50

[GoogleCodeExporter]

I haven't had a chance to look at that yet. Hopefully this weekend.

Original comment by arvid.no...@gmail.com on 22 Jun 2012 at 2:52