search

miguelfreitas / twister-core

twister core / daemon
MIT License
1.42k stars 251 forks source link

Enable to see the 'firehose' #20

Open eschnou opened 10 years ago

eschnou commented 10 years ago

With a small community and to help bootstrap things, having access to the full firehose of messages in the UI would be awesome. Since I haven't looked at the technical details yet, I wonder if it is at all possible (i.e. does my client see all messages or only the one routed to me?).

If someone think is possible and has some pointers to get me started, I'm happy to dive in the code and try a first contribution.

toyg commented 10 years ago

I think Miguel is the only one who can answer. From what I understand from the bug report about "too many users crashing twisterd", there are architectural limits that are hard to deal with -- basically, if I understood correctly, every time you follow a user, you're actually joining a sort of torrent with all his posts. Joining hundreds and hundreds of torrents at the same time is hard work, and it seems to have problems when you follow more than 250 users or so.

gubatron commented 10 years ago

has there an acceptable limit of incoming messages per second been measured (to avoid crashes or freezing)? my guess is that you'll have to add some sort of throttling and queuing (might be there already, still getting my macosx to build the project, can't wait to be of help)

miguelfreitas commented 10 years ago

Giacomo,

I have been think about this. You understood correctly the part about joining torrents, but it is not difficult to fix that limit, not at all.

The way I modeled torrent swarms was removing all direct access to files (within a given torrent) and replacing them with a leveldb instance. I don't know much about leveldb, but i'd guess it probably keeps a few file descriptors open, like 3 per torrent or something (i don't know how many).

It is not difficult to replace individual torrent's leveldb instances for a single global leveldb. That is, just one database for all swarms. We just need to concatenate the hash or username when computing the key in leveldb.

libtorrent already seems to do a good job about round-robing the torrents, so file descriptors used for connections shouldn't be an architectural limitation.

regards,

Miguel

On Sun, Jan 5, 2014 at 1:10 AM, Giacomo Lacava notifications@github.comwrote:

I think Miguel is the only one who can answer. From what I understand from the bug report about "too many users crashing twisterd", there are architectural limits that are hard to deal with -- basically, if I understood correctly, every time you follow a user, you're actually joining a sort of torrent with all his posts. Joining hundreds and hundreds of torrents at the same time is hard work, and it seems to have problems when you follow more than 250 users or so.

— Reply to this email directly or view it on GitHubhttps://github.com/miguelfreitas/twister-core/issues/20#issuecomment-31595114 .

miguelfreitas commented 10 years ago

On Sun, Jan 5, 2014 at 2:50 PM, Angel Leon notifications@github.com wrote:

has there an acceptable limit of incoming messages per second been measured (to avoid crashes or freezing)?

Not that i can think of. But what you mean by incoming messages per second, the ones from users you follow?

I'd guess the ability of libtorrent to download pieces/second = posts/second is pretty high.

iShift commented 10 years ago

Old issue i think we can close that