search

bessey / bt-elixir

Learning Elixir via building a BitTorrent client
3 stars 1 forks source link

Move block / block assembly into Mnesia or ETS? #4

Open mpope9 opened 4 years ago

mpope9 commented 4 years ago

I wouldn't say this is crucial, but it seems if block sizes reach 512 kB or 1 MB you might want to move them from Task statedata to being in Mnesia (or maybe just an ETS table because only a single process would interact with them). If you're really going for it and you have a bunch of torrents going at once you might hit your VM memory limit and get OOM'd by the kernel. ETS is offheap and fast!

Or if you want to support recovery from a partially downloaded block, you could use Mnesia with disk copies, or use this: https://github.com/klarna/mnesia_eleveldb Bindings for LevelDB K/V store. Doesn't have a 2GB on-disk limit and is highly optimized C++ code.

bessey commented 4 years ago

Previously we were writing blocks straight to disk, stupid because blocks are at most 2^14. I've since rearchitected massively so that each PeerDownloader builds up an entire Piece and dispatches multiple requests for blocks of that piece until it has a complete buffer, then writes the whole lot to disk.

With that in mind, it could well be time to move the buffer implementation to ETS. Mnesia seems little better than writing files to disk for my purposes though

Matt

On Sat, 25 Jan 2020 at 23:35, Matthew Pope notifications@github.com wrote:

I wouldn't say this is crucial, but it seems if block sizes reach 512 kB or 1 MB you might want to move them from Task statedata to being in Mnesia (or maybe just an ETS table because only a single process would interact with them). If you're really going for it and you have a bunch of torrents going at once you might hit your VM memory limit and get OOM'd by the kernel. ETS is offheap and fast!

Or if you want to support recovery from a partially downloaded block, you could use Mnesia with disk copies, or use this: https://github.com/klarna/mnesia_eleveldb Bindings for LevelDB K/V store. Doesn't have a 2GB on-disk limit and is highly optimized C++ code.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bessey/bt-elixir/issues/4?email_source=notifications&email_token=AAFM42EXRKI4FHX6BHL25DTQ7TEEVA5CNFSM4KLTVBO2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IIW7GIA, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFM42CDD53JE6SOSCLOWVDQ7TEEVANCNFSM4KLTVBOQ .