High memory usage

[gadelkareem]

I limited the requests to 10 concurrent requests to the DHT server and dropped the torrent from the client after collecting the metainfo:

var limiter = make(chan struct{}, 10)
func findTorrent(w http.ResponseWriter, r *http.Request) *torrent.Torrent {
    limiter <- struct{}{}
    t := torrentForRequest(r)
    defer func() {
        <-limiter
        t.Drop()
    }()
    if nowait, err := strconv.ParseBool(r.URL.Query().Get("nowait")); err == nil && nowait {
        select {
        case <-t.GotInfo():
        default:
            http.Error(w, "info not ready", http.StatusAccepted)
            return nil
        }
    } else {
        // w.WriteHeader(http.StatusProcessing)
        select {
        case <-t.GotInfo():
        case <-r.Context().Done():
            return nil
        }
    }

    // w.WriteHeader(http.StatusOK)

    return t
}

Yet the memory usage gets up to 2GB in its peak. Here is a sample from stackimpact:

Is it normal that the communication requires that much ram? Could it be a problem with the bencode serialisation?

[anacrolix]

Where are you deriving the heap allocation value? Often people conflate the virtual and resident memory usage, I want to rule that out.

[gadelkareem]

it is the default stackimpact allocation https://stackimpact.com/blog/go-profiler-internals/

[anacrolix]

Okay that makes sense. It's likely due to query rate limiting due to the conntrack implementation. I intend to redesign this sometime. In the meanwhile, try calling SetNoMaxEntries and setting Timeout to a function that returns zero duration on the ConnectionTracking field in the dht ServerConfig for your instance.

[gadelkareem]

I made some tests using github.com/shiyanhui/dht and the memory/CPU usage is much lower.

[anacrolix]

@gadelkareem try again now with the latest version of anacrolix/dht.

[gadelkareem]

Thanks @anacrolix will let you know when I get the time.

[anacrolix]

If this isn't fixed by previous changes, https://github.com/anacrolix/dht/commit/8f69499a9cd0dfd8bf846abe05bda3f17197561d will most certainly do it.

[anacrolix]

See v1.5.0.