Process only N PartialDownloads at the same time

[jan-carreras]

The VPS has low specs and I want to be able to limit the number of torrents downloaded at the same time, since they are memory intensive.

Probably using one existing Middleware or creating a new one will do the trick.

[jan-carreras]

I don't think this task makes sense:

• The idea is to limit the concurrent downloads. But the problem is deeper. Since this application is memory bound and we don't have control of how many jobs are published in the queue, limiting the number of jobs for this particular queue would do the trick, but it's difficult since the middleware should keep state and all the jobs should be of the same size.

Each job represents a partial download of a Torrent. Some torrents with GB of data will represent various jobs. Some smaller torrents might be just one job. Constrains:

• Two parts of the same torrent cannot be downloaded by the same process. Would not make sense, either, since the split would not make sense
• Parts must be downloadable from various other hosts/processes. Like in a distributed system.
• It's very hard to keep track of the torrent download status, since we don't know how many parts do we have for each download
• Each host can have different memory sizes, thus it would be nice if we could get DownloadParts that fit into the free memory we have, but no more.
• If we use a queue system and not adjust well the concurrency parameters, the OOMKiller is going to pay a visit every few minutes. Something we want to avoid.

Thus, I'll close this task an open another one that will store the Download Parts in a persistent datastore, and each hosts will be able to query and get the downloadPart that fits in memory.