read_piece_alert and streaming (continued from arvidn/libtorrent)

I skimmed the above, but I suspect the issue lies in not providing or wanting to add a synchronous wake up/alert mechanism when a piece is available, whether it was already available, or just became available. You are correct in that anacrolix/torrent does make that possible and guarantee that the read can occur.

All the alerts are asynchronous in libtorrent @anacrolix (not synchronous) The proposal has a read piece call that will notify (with an alert called read_piece_alert) when the specified piece is downloaded, and provides the piece data in a buffer. That's all.

I'm not sure it's appropriate to discuss anacrolix/torrent stuff specific here, but assuming it's relevant to your feature request here:

What is the purpose of having the implementation call a user specified readahead function when you already provide for the user to set the number of readahead bytes?

As the Reader read position moves through a stream as reads occur, it might be desirable to dynamically calculate readahead, rather than have to call back in and adjust it and occur overhead in correcting the client's readahead-based calculations after it had already made them with a previous value. For example if you are reading slowly you might reduce readahead, or you might balance it dynamically across the available cache space.

You talk about cache space, is this something your torrent library maintains and is configurable by the client? Or is this just a generic client side caching mechanism you're talking about that is not relevant to your library?

Why is your Reader implementation blocking when the programming world is moving to asynchronous interfaces that return results in an efficient non-blocking manner?

Because people are confused about function colouring and concurrency and think that async/await syntax and callbacks somehow make "asynchronous" code possible when it's an entirely different axis. This is opinionated, but see how Haskell and Go do concurrency vs. other languages. I'm not convinced all languages have the tradeoffs figured out but they're getting there. In short, my implementation in Go does use asynchronous I/O, and it's programmed against using a blocking API. Those are the separate axes.

I'm not sure what function colouring is (attaching an "async" keyword?), though I thought Go concurrency and the async/await mechanism pioneered by C# are fundamentally using the same thing under the hood: coroutines (but one is stackfull and the other is stackless.)

Your view is as you admit opinionated, and that these are different axes of consideration .. Do you have a post or article in mind that would help explain this view to me so that I may learn more about it?

anacrolix/torrent will wake up blocked Readers when their data is available (which could be immediately), and as many blocks/pieces that are available are stitched together as necessary to satisfy the Reader interface constraints.

I don't doubt your implementation is efficient, but why is it programmed against using a blocking API, and not an async/nonblocking API?

I don't know if libtorrent supports evicting pieces like anacrolix/torrent, but if it does, a synchronous API, or an asynchronous one that ensures that data is delivered before it is evicted would be required to match the behaviour.

You talk about evicting pieces, I guess your library maintains a cache of some sort that is user visible/configurable? Why?

Why do pieces necessarily need to be in a cache or main memory to be delivered to callers as part of a stream oriented Reader API? If they happen to be already downloaded and on disk or in an OS buffer (and not in the library maintained cache) can't they still be read and then delivered / stitched together with relevant pieces to be delivered to the caller?

anacrolix / torrent

read_piece_alert and streaming (continued from arvidn/libtorrent) #832