[BUG] Missing pieces when downloading/Streaming

[ahizap]

TorrServer does not properly download files, the downloaded files has partial or missing pieces when rechecked with qbittorrent. The missing pieces causes the downloads to become corrupt and in media files it causes playback errors where the data piece is missing

Steps to reproduce the behavior:

1. Download files using TorrServer
2. Recheck the downloaded files with qbittorrent

3. The file will have missing pieces

   • Ubuntu 22.04
   • TorrServer Version MatriX.130

[tsynik]

Can't reproduce on Mac with TorrServer Matrix.130. 3.3 GB torrent downloaded in Mozilla from Open link in Web GUI and it's content checked with qBittorrent v4.6.4. No missing pieces on recheck with the same torrent / magnet.

p.s. Anyway, TorrServer is optimized for video streaming and in it's reader code check of peace completion is disabled for better streaming experience. You can try use TorrServer builded from new-engine branch where no such improvements and more actual torrent download library too. And use proper soft for full files download.

[ahizap]

Maybe the problem is specific to linux?

[oshitikov]

Can confirm that the problem is present and the most probable reason is disabled piece completion check. Corrupted pieces are problematic for some players (based on ExoPlayer, for example), but VLC or Kodi are ok whith them (you can notice some artifacts during playback, but it doesn't crash). I've tried the build from new-engine branch and noticed no problems for "sensitive" players so far. @ahizap I have docker image for new-engine build and you can try it if you want: https://hub.docker.com/layers/linefight/torrserver-new-engine/132.5/images/sha256-24f14fa15c83e58d70f7a7320a012f891877da72ebb6b43a91bf3e0e174f16e6?context=repo. It's built from my fork with updated dependencies and the latest changes from master branch.

@tsynik Are there any plans to migrate to new-engine with piece completion check enabled? I'm running torrserver on my raspberry pi 4b and it's pretty performant and capable of streaming 4k content with no problems.

[oshitikov]

Maybe the problem is specific to linux?

I believe it's not, because it's how torrents work I suppose. There is always a possibility to download corrupted piece of content from some peer with corrupted HDD, for example. The problem happened a lot of times for me on different platforms while using qbittorrent.

[tsynik]

@tsynik Are there any plans to migrate to new-engine with peace completion check enabled? I'm running torrserver on my raspberry pi 4b and it's pretty performant and capable of streaming 4k content with no problems.

There is no plans to switch to actual codebase till it will have same preformance. I can reenable this check https://github.com/tsynik/torrent/commit/d16ba45e0bbdec48be31491931ad87e51582fa1c in current lib and compare, but I don't have corrupted pieces with my samples at all.

[YouROK]

Can you check with udp protocol disabled? You need to disable μTP (Micro Transport Protocol) in the settings

[oshitikov]

@tsynik thanks! I'll test it then and let you know in a week or so about the results.

@YouROK

Can you check with udp protocol disabled? You need to disable μTP (Micro Transport Protocol) in the settings

I could and I believe it'll also fix the problem. I suppose that μTP without hash check is no better than UDP in terms of data reliability, but I would prefer to have data consistency check on my side and spend more CPU cycles, than potentially reduce number of discovered seeds/peers because of μTP protocol disabled. I'm not an expert in BitTorrent networks, but I suspect that some peers (even trackers?) can refuse TCP connections.

[oshitikov]

@tsynik I'm sure the implementation without pieces check will be noticeably faster, but it will have the probability to get a corrupted piece even with TCP mode only, because seeds could have corrupted pieces on their side and TCP won't be able to fix that in any way. Probability increases with uTP because of UDP nature where data corruption while transmitted over the network is possible. So the only way to get durable stream of data is to use pieces check on application level. Maybe it's possible to implement some config option for that (enable/disable pieces check)? I'm ready to contribute, but I'm not sure when I have a time for that :(

[tsynik]

I don't understand. There is always piece check before it's marked as complete. Incomplete piece will be requested again. There is no corrupted pieces after download via torrserver an rechecking data in Transmission. Reader got data from storage, not peers or connections, and it knows which ones are complete and already checked.

[oshitikov]

I thought that with https://github.com/tsynik/torrent/commit/d16ba45e0bbdec48be31491931ad87e51582fa1c change reader didn't wait for check to complete

[oshitikov]

There is always piece check before it's marked as complete. Incomplete piece will be requested again.

I checked the torrent library code and confirmed that there is indeed a piece check logic, even if the piece check is disabled in the reader's code (I apologize for any mistakes in my assumptions; I am new to this codebase as well as to golang). However, you mentioned:

There is no corrupted pieces after download via torrserver an rechecking data in Transmission

Could it be that all pieces are verified and re-downloaded already by the time the download is finished (which is not usually the case for most situations with the active reader and download is in progress)?

Reader got data from storage, not peers or connections, and it knows which ones are complete and already checked.

I understand that. In my previous messages, I wanted to highlight that when data is in storage, it might be corrupted either because it was received via UDP (data could be corrupted while passing through the network) or because it was received from a peer that has corrupted data on their side (in this case, even TCP won't help). However, if the piece check is disabled in the reader's code, the corrupted data can be available to readers, potentially causing artifacts or even crashes during playback. This could happen if a corrupted piece is read before it has been verified and re-downloaded by the torrent library.

[tsynik]

I understand that. In my previous messages, I wanted to highlight that when data is in storage, it might be corrupted either because it was received via UDP (data could be corrupted while passing through the network) or because it was received from a peer that has corrupted data on their side (in this case, even TCP won't help). However, if the piece check is disabled in the reader's code, the corrupted data can be available to readers, potentially causing artifacts or even crashes during playback. This could happen if a corrupted piece is read before it has been verified and re-downloaded by the torrent library.

Nope. every downloaded piece verified before it's marked as complete. There is also /stat web page where you can see how many chunks were wasted during download. This function check every piece and it will be redownloaded if it's hash not good - https://github.com/tsynik/torrent/blob/7014fcb01f59f42e6dc538231b87fa5e65c994c2/torrent.go#L1627

[oshitikov]

@tsynik My following reasoning applies to the case when this line of code is commented out.

Am I missing something?..

1. Suppose we have a reader R and it wants to read some offset that targets the chunk C of piece P. Let's assume that piece P is in the process of being downloaded and chunk C is not the final chunk for piece P, so the verification process won't start after chunk C is received and written out.
2. R waits in the waitAvailable function until the available function returns non-zero. This will happen when the haveChunk function returns true.
3. Message with chunk C for piece P arrives to the receiveChunk function
4. Chunk C is written to the storage. Now the haveChunk function returns true.
5. Finally, R returns from the waitAvailable function and reads chunk C. However, it is not verified and can be corrupted since the verification process only starts when all chunks for piece P are received.

There is even a comment for SetResponsive function which says that

Don't wait for pieces to complete and be verified. Read calls return as soon as they can when the underlying chunks become available.

So actually there is no need to comment out this line of code to get this behaviour. It's sufficient to call SetResponsive for a reader right after it's created.

[tsynik]

So actually there is no need to comment out this line of code to get this behavior. It's sufficient to call SetResponsive for a reader right after it's created.

But if you'll search for this field use in whole project you'll find it only in reader code and it doesn't apply without peace completion.

[oshitikov]

Yes, and this flag controls whether it should make not completed piece available for readers or not. So it does it's job. If responsive set to true, then there will no completion check and data will be available for readers if there are some chunks ready, otherwise reader will wait until whole piece is completed and verified.

[tsynik]

Yes, and this flag controls whether it should make not completed piece available for readers or not. So it does it's job. If responsive set to true, then there will no completion check and data will be available for readers if there are some chunks ready, otherwise reader will wait until whole piece is completed and verified.

Where this check occurs? available for ... breaks if responsive is false or piece is not complete. It will not break only if both responsive is true AND piece is complete

[oshitikov]

1. responsive = true -> go for next condition (haveChunk)
2. responsive = false, pieceComplete = true -> go for next condition
3. responsive = false, pieceComplete = false -> breaks, data won't be available for readers until pieceComplete is true. This is what we need to allow verified only data for readers. pieceComplete returns true only when piece is verified.

[tsynik]

• responsive = true -> go for next condition (haveChunk)

• responsive = false, pieceComplete = true -> go for next condition

• responsive = false, pieceComplete = false -> breaks, data won't be available for readers until pieceComplete is true. This is what we need to allow verified only data for readers. pieceComplete returns true only when piece is verified.

break in for loop means stop and not execute next condition. so responsive = true, pieceComplete = false -> don't change available ret responsive = false, pieceComplete = true -> don't change available ret responsive = false, pieceComplete = false -> don't change available ret responsive = true, pieceComplete = true - > don't break and return new available ret if haveChunk is also true

[oshitikov]

For the all cases except responsive = false, pieceComplete = false, it will check the rest conditions. For the responsive = false, pieceComplete = false it will break, loop will stop and function will return. And this essentially means that reader can't read chunks of incomplete piece.

P.S. I've checked several golang resources about break statement. And seems like it's just exit the loop, as in most of languages.

https://www.tutorialspoint.com/go/go_break_statement.htm https://gobyexample.com/for

[tsynik]

For the all cases except responsive = false, pieceComplete = false, it will check the rest conditions. For the responsive = false, pieceComplete = false it will break, loop will stop and function will return. And this essentially means that reader can't read chunks of incomplete piece.

Okay. We use SetResponsive() only in preload readers and this code was reenabled in MatriX.133 release. Probably will fix this issue (but I can't reproduce corrupted pieces after download whole torrent file from torrserver even with previous ones).

[oshitikov]

Maybe I'll try to reproduce it later. It's needed to create a torrent for some video, start seeding it, then corrupt the file using some hex editor, and then try to download this torrent using Matrix.132 torrserver. Then run hash check using some torrent client.

[oshitikov]

@tsynik thank you for your time!

[tsynik]

probably fixed in MatriX.133 + release. Reopen if still reproducible