Closed borrelnoten closed 3 years ago
alanmcgovern
commented
3 years ago Can you share a torrent which reproduces the issue? Both of these ones seem seem to be fine with pre-release build 0112. This is the one I just pushed to nuget/github.
By default StreamProvider downloads the last 2 pieces as part of it's pre-buffering phase. Out of curiosity are you passing prebuffer: false when calling StreamProvider.CreateStreamAsync? Does the problem go away if you pass true for this?
borrelnoten
commented
3 years ago I use the CreateStreamAsync with prebuffer set to false. I start with byte offset 0 and wait till about 1% is downloaded. The download speed is around 500Kb to 2Mb at that point. Next I feed the downloaded data to FFMPEG and it returns me the byte offset it needs to start playing the file. I move the torrent download cursor (seek ) to that byte offset. Normaly the torrent downloader picks up the new download offset and I signal the data provider for ffmpeg when the needed bytes are available for processing (real time download and trancoding and streaming to chormecast in a nutshell).
Examples where the speed dropped to close to 0 and did not recover:
magnet:?xt=urn:btih:2795162BA4332C7D18534522CB05BF41D8AADE95&dn=The.Good.Doctor.S04E09.Irresponsible.Salad.Bar.Practices.1080p.AMZN.WEBRip.DDP5.1.x264-TOMMY&tr=udp://tracker.coppersurfer.tk:6969/announce&tr=udp://tracker.leechers-paradise.org:6969/announce&tr=http://tracker.foreverpirates.co:80/announce&tr=udp://tracker.cyberia.is:6969/announce&tr=udp://tracker.birkenwald.de:6969/announce&tr=udp://explodie.org:6969/announce&tr=udp://tracker.opentrackr.org:1337/announce&tr=udp://tracker.tiny-vps.
magnet:?xt=urn:btih:f80b30ab62eaa044af67b7a8cd66bf9fa6c53a8a&dn=The.Good.Doctor.S04E09.Irresponsible.Salad.Bar.Practices.1080p.AMZN.WEBRip.DDP5.1.x264-TOMMY%5Beztv.re%5D.mkv%5Beztv%5D&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A80&tr=udp%3A%2F%2Fglotorrents.pw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969
magnet:?xt=urn:btih:9ae674fbc453d2b058086262d4d5637b19c5a9c6&dn=The.Good.Doctor.S04E09.HDTV.x264-PHOENiX%5Beztv.re%5D.mkv%5Beztv%5D&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A80&tr=udp%3A%2F%2Fglotorrents.pw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969
magnet:?xt=urn:btih:9953ad1bf1a9dffd42d66f221f750151a3924f03&dn=The.Good.Doctor.S04E09.1080p.WEB.H264-STRONTiUM%5Beztv.re%5D.mkv%5Beztv%5D&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A80&tr=udp%3A%2F%2Fglotorrents.pw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969
magnet:?xt=urn:btih:5acabab0a319bd46ce4f11c1f7541b8b8cd20965&dn=The.Good.Doctor.S04E09.Irresponsible.Salad.Bar.Practices.1080p.HDTV.x264%5Beztv.re%5D.mkv%5Beztv%5D&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A80&tr=udp%3A%2F%2Fglotorrents.pw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969
This is how I seek and set the correct file focus in case of multiple files inside the torrent:
public override void MoveDownloadCursor(Int64 offset)
{
var cf = GetCurrentlyDownloadingFile();
cur_torFileIndexNum = MediaBrowserTorrentFile.IndexOf(cf.DownloaderFile_Key);
// don't switch cursor if not needed (already file is downloaded compleetly)
if (torrentStreamProvider.Files[cur_torFileIndexNum].BytesDownloaded() != torrentStreamProvider.Files[cur_torFileIndexNum].Length)
{
// switch to other file only when is not current file
if (old_torFileIndexNum != cur_torFileIndexNum)
{
old_torFileIndexNum = cur_torFileIndexNum;
SetTorrentFileFocus(cur_torFileIndexNum).Wait();
}
if (torrentStream != null)
{
torrentStream.Seek(offset, SeekOrigin.Begin);
}
}
else
{
cur_torFileIndexNum = old_torFileIndexNum;
}
}
private async Task SetTorrentFileFocus(int findex)
{
if (torrentStream != null)
{
torrentStream.Dispose();
}
if (torrentStreamProvider != null)
{
torrentStream = await torrentStreamProvider.CreateStreamAsync(torrentStreamProvider.Files[findex], false, new CancellationToken());
}
}and this is how I check if the needed piece is available / ready to use:
private void Manager_PieceHashed(object sender, PieceHashedEventArgs e)
{
//Console.WriteLine(e.PieceIndex + " : " + e.HashPassed);
try
{
if (engine != null && torrentStreamProvider != null
&& GetDownloaderData().RoutineStatus == EntityData.MediaBrowserDownloader.DownloaderRoutineStatus.Started)
{
if (e.HashPassed)
{
//engine.DiskManager.FlushAsync(e.TorrentManager.Torrent).Wait();
// update the media browser file range with the newly downloaded piece
int inum = 0;
foreach (var f in e.TorrentManager.Files)
{
// find the torrent file this piece belongs to
if (e.PieceIndex >= f.StartPieceIndex && e.PieceIndex <= f.EndPieceIndex)
{
// calc start and end byte pos of the piece downloaded for this file
int pieceInFileNr = e.PieceIndex - f.StartPieceIndex;
long Rangepstartbyte = e.TorrentManager.Torrent.PieceLength * pieceInFileNr;
long Rangependbyte = Rangepstartbyte + e.TorrentManager.Torrent.PieceLength-1;
// compensate for the non-zero byte start point of piece 0 of the file
Rangependbyte -= f.StartPieceOffset;
if (pieceInFileNr > 0)
{
Rangepstartbyte -= f.StartPieceOffset;
}
// prevent file end byte pos overflow
if (Rangependbyte > f.Length)
{
Rangependbyte = f.Length;
}
//Console.WriteLine(inum + "("+ e.PieceIndex + ")" + " : " + Rangepstartbyte + " -> " + Rangependbyte);
// update the range for the media browser file
var mbfileg = MediaBrowserTorrentFile[inum];
GetRangeByFileID(mbfileg).AddNewRange(Rangepstartbyte, Rangependbyte);
// check if this file is done.
if (f.BytesDownloaded() >= f.Length)
{
// update current file to completed
GetDownloadFileByKey(mbfileg).FileStatus = MediaBrowserEntityData.EntityData.MediaBrowserDownloader.DownloaderFileStatus.Completed;
// complete range to be sure we can deliver entire file
GetRangeByFileID(mbfileg).AddNewRange(0, f.Length);
// check if this was the current file we were dowloading, if so, more focus to a file that is not yet downloaded
if (inum == cur_torFileIndexNum)
{
// is current file, so look for other file to download
for (int i = 0; i < e.TorrentManager.Files.Count; i++)
{
// find a file that is not yet downloaded and put the focus on it:
if (e.TorrentManager.Files[i].BytesDownloaded() < e.TorrentManager.Files[i].Length)
{
MarkFileForDownload(MediaBrowserTorrentFile[i]);
break;
}
}
}
}
}
inum++;
}
}
}
}
catch { }
}and this is how I init the stream provider:
torrentStreamProvider = new StreamProvider(engine, GetDownloaderData().FullDownloaderPath, torrent);
await torrentStreamProvider.StartAsync();
// wait till we have metadata
while (torrentStreamProvider.Manager.HasMetadata == false
&& GetDownloaderData().RoutineStatus != MediaBrowserEntityData.EntityData.MediaBrowserDownloader.DownloaderRoutineStatus.Stopping)
{
Thread.Sleep(2000);
}Oh this is the strangest thing.
I took one of those at random,
magnet:?xt=urn:btih:f80b30ab62eaa044af67b7a8cd66bf9fa6c53a8a&dn=The.Good.Doctor.S04E09.Irresponsible.Salad.Bar.Practices.1080p.AMZN.WEBRip.DDP5.1.x264-TOMMY%5Beztv.re%5D.mkv%5Beztv%5D&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A80&tr=udp%3A%2F%2Fglotorrents.pw%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969This one has the a piece length of 2318336 bytes. Pieces are divided into blocks of size 16kb. That means this particular torrent has 141.5 blocks per piece. This has never intentionally been supported by monotorrent, though perhaps this was inadvertently supported due to how pieces were constructed before. I'm going to have to see how to make this work as it is not something i was aware was acceptable!
It's probably a fairly trivial change, but.... oops?
borrelnoten
commented
3 years ago I see. Also note that it happened to all the above listed torrents
alanmcgovern
commented
3 years ago Great - if you can try master builds then this commit might do the trick:
https://github.com/alanmcgovern/monotorrent/commit/fcc4edd85227257a0f2ddfff3f6b3bb96fee4af7
Otherwise i'll try one or two more of those and ensure things are working OK. So far two of them are streaming as expected, and i can seek. I'm using a slightly custom version of https://github.com/mfkl/lvst for testing, one which supports seeking.
borrelnoten
commented
3 years ago you are fast! I have compiled the root master version (https://github.com/alanmcgovern/monotorrent) but it seems to crash when I try and use the compiled Release version of monotorrent.dll.
Debug dump:
02/19/2021 11:15:36.787 PM: Error (CurrentDomain_UnhandledException) System.UnhandledExceptionEventArgs at MediaBrowserDownloader.TorrentDownloader2020.Manager_PieceHashed(Object sender, PieceHashedEventArgs e)
at MonoTorrent.AsyncInvoker`1.<>c.<.cctor>b__4_0(Object o)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() CurrentDomain_UnhandledException Ln:0 Col:0 -
Manager_PieceHashed Ln:0 Col:0 -
<.cctor>b__4_0 Ln:0 Col:0 -
WaitCallback_Context Ln:0 Col:0 -
RunInternal Ln:0 Col:0 -
Run Ln:0 Col:0 -
System.Threading.IThreadPoolWorkItem.ExecuteWorkItem Ln:0 Col:0 -
Dispatch Ln:0 Col:0 -
PerformWaitCallback Ln:0 Col:0 -
at MediaBrowserController.UtilsController.CurrentDomain_UnhandledException(Object sender, UnhandledExceptionEventArgs e)
at MediaBrowserDownloader.TorrentDownloader2020.Manager_PieceHashed(Object sender, PieceHashedEventArgs e)
at MonoTorrent.AsyncInvoker`1.<>c.<.cctor>b__4_0(Object o)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
System.MissingMethodException: Method not found: 'Int32 MonoTorrent.ITorrentFile.get_StartPieceOffset()'.
at MediaBrowserDownloader.TorrentDownloader2020.Manager_PieceHashed(Object sender, PieceHashedEventArgs e)
at MonoTorrent.AsyncInvoker`1.<>c.<.cctor>b__4_0(Object o)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
02/19/2021 11:15:36.872 PM: Debug (DownloaderSetupAndStart) Downloader started in c:\dtest\d763c355-59b6-41f7-a4bf-ae27a195c689
02/19/2021 11:15:43.251 PM: Warning (GetURLContent) Fatal response code received: NotFound. Request aborted: https://zoink.ch/torrent/WandaVision.S01E07.1080p.WEBDL.H264.Atmos-EVO%5Beztv.re%5D.mkv%5Beztv%5D.torrent
02/19/2021 11:15:43.252 PM: Debug (GetTorrentFromMagnetUri) Download torrent from: https://zoink.ch/mirror/{DN}.torrent hash = 1E9404B236A0DA5306831F8C39BEC12830C00797 dn = WandaVision.S01E07.1080p.WEBDL.H264.Atmos-EVO%5Beztv.re%5D.mkv%5Beztv%5D url=https://zoink.ch/mirror/WandaVision.S01E07.1080p.WEBDL.H264.Atmos-EVO%5Beztv.re%5D.mkv%5Beztv%5D.torrent
02/19/2021 11:15:43.252 PM: Debug (MakeRequest) Loading url:https://zoink.ch/mirror/WandaVision.S01E07.1080p.WEBDL.H264.Atmos-EVO%5Beztv.re%5D.mkv%5Beztv%5D.torrent
02/19/2021 11:15:43.262 PM: Error (CurrentDomain_UnhandledException) System.UnhandledExceptionEventArgs at MediaBrowserDownloader.TorrentDownloader2020.Manager_PieceHashed(Object sender, PieceHashedEventArgs e)
at MonoTorrent.AsyncInvoker`1.<>c.<.cctor>b__4_0(Object o)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() CurrentDomain_UnhandledException Ln:0 Col:0 -
Manager_PieceHashed Ln:0 Col:0 -
<.cctor>b__4_0 Ln:0 Col:0 -
WaitCallback_Context Ln:0 Col:0 -
RunInternal Ln:0 Col:0 -
Run Ln:0 Col:0 -
System.Threading.IThreadPoolWorkItem.ExecuteWorkItem Ln:0 Col:0 -
Dispatch Ln:0 Col:0 -
PerformWaitCallback Ln:0 Col:0 -
at MediaBrowserController.UtilsController.CurrentDomain_UnhandledException(Object sender, UnhandledExceptionEventArgs e)
at MediaBrowserDownloader.TorrentDownloader2020.Manager_PieceHashed(Object sender, PieceHashedEventArgs e)
at MonoTorrent.AsyncInvoker`1.<>c.<.cctor>b__4_0(Object o)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
System.MissingMethodException: Method not found: 'Int32 MonoTorrent.ITorrentFile.get_StartPieceOffset()'.
at MediaBrowserDownloader.TorrentDownloader2020.Manager_PieceHashed(Object sender, PieceHashedEventArgs e)
at MonoTorrent.AsyncInvoker`1.<>c.<.cctor>b__4_0(Object o)
at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()That looks like you've got a bad binary in place.
You've compiled against an older monotorrent but are loading a new one at runtime.
MonoTorrent.ITorrentFile.get_StartPieceOffset()
This property was removed and "OffsetInTorrent" was added instead. The new property is equivalent to this old code:
ITorrentFile.StartPieceIndex * Manager.PieceLength + ITorrentFile.StartPieceOffice
borrelnoten
commented
3 years ago I have downloaded the master. Compiled it and copy pasted the monotorrent dll overwiting the nuget version And i see now indeed that was the issue. Will retry today.
borrelnoten
commented
3 years ago I've tested the nuget package 2.0 instable rev 0122 version. I can't say for 100% sure but this version is simply not downloading at all compared with the previous version. The download speeds is 0 bytes or somewhere close. I've tried 20 different torrents, updated trackers and tried several VPN connections. Still no proper download. When using the same torrents with the previous version the download speed was fine right away.
Not sure what this version has changed but I'm sticking with the previous one even though seeking is not working in that version, it does download.
alanmcgovern
commented
3 years ago Yeah, version 0122 has the bug which makes it not work for the torrents you provided.
This is fixed in master, which is version 0123. I'll push this to nuget and here now.
borrelnoten
commented
3 years ago always delighted to see new version1 :) I've tested it using 10 different torrent (incl magnet URLs) but the Magnet Urls just do not get any speed. Stuck at 50Kb on avrg. or lower most of the time after 10 minutes waiting.
borrelnoten
commented
3 years ago More info for testing: I've now compiled thr master and that version works better. I've used the following torrent / Magnet URLs and getting speed up to max 500Kb p/s:
borrelnoten
commented
3 years ago Got much better download speeds now with the master build (around 1 Mb p/s). Seeking seems to work good now!!
The Good Doctor S04E09 Irresponsible Salad Bar Practices 1080p HDTV x264 aFi rarbg
But after I start playing while downloading, it seems the engine has trouble keeping up speed (drops to 0, and climbs back up to 1 mb after 1 minute)
alanmcgovern
commented
3 years ago @borrelnoten Out of curiosity I popped the torrent into uTorrent and it looks like only 1 peer is online who can provide more than 1MB/sec of throughput, and uTorrent was only able to initiate transfers from 2 peers - the second peer gave about 600kB/sec.
I can max out my 250Mb/sec connection fairly easily (downloading about 30 megabytes a second) on torrents with more active peers, so I'm pretty certain neither utorrent or monotorrent are responsible for the slow transfer rate for that torrent in particular.
One of the design decisions in the new streaming mode is that once all the high priority pieces are downloaded, monotorrent will choose to request fewer pieces from peers. On higher latency connections this will result in low priority blocks being requested at a slower rate than normal downloading mode, but as soon as you 'read' data (seek to a new location in the torrent) and pieces become high priority, the engine will pipeline dozens of concurrent requests instead of just a handful.
The benefit gained from this approach is that seeking becomes significantly better in some cases as there will not be a deep queue of piece requests from an area of the torrent which is not needed.
I'll keep looking through things and will run some more tests. If you can have torrents where uTorrent and MonoTorrent are radically different performance wise then those will be easiest to use to diagnose any potential issues.
borrelnoten
commented
3 years ago I agree. Best guess is that the piece focus changes causes peer loss which makes sence.
alanmcgovern
commented
3 years ago @borrelnoten Out of curiosity - could you dump the value of TorrentManager.HashFails when you have downloading issues. Is it zero, or a very very low number, for the duration of time you're downloading? If not, this could explain why your download rate is dropping down to nearly-zero.
Race conditions, or logic bugs, can show up as hash failures. This is something I've begun monitoring closely while working on the memory cache and other DiskManager related changes.
borrelnoten
commented
3 years ago I've update to latest stable release and did what you've asked and monitored the hashfails. Indeed they are zero when speed is dropping as soon as I start to play the file (and the seek is used to get the pieces needed to start stranscoding the file using ffmpeg.
I've copied my debug log below. It starts with the torrent being downloaded using the magnet hash value. Next I start the engine and download the pieces needed for ffmpeg to get the video duration. After that, I try to start to play the video. To do that, ffmpeg trancodes the video in real time using my local server ip to serve the downloaded pieces of the downloaded torrent file on harddisc.
Each seek I do, I print out the hashfails value. Note that I do not actualy seek when the offset is the same as the previous seek offset to prevent repeated seeking to the same offset.
Each seek also is noted in the logs below. I hope you can see why this is not working. The torrent was downloading at 1.2 Mb p/s but as soon as I started to play the file and use ffmpeg to transcode the file (and seek), the download speed dropped to zero and the hashfails are visible at that time.
03/06/2021 11:53:57.385 PM: Debug (DownloaderSetupAndStart) Downloader started in c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105
03/06/2021 11:53:57.385 PM: Debug (GetTorrent) Torrent URL: magnet:?xt=urn:btih:599BDF5B5015D8EB4149184A6F2638977F696BBA&dn=%5Bzooqle.com%5D%20The%20Blacklist%20S08E02%201080p%20WEB%20H264&tr=http://bt1.archive.org:6969/announce&tr=http://bt2.archive.org:6969/announce&tr=http://tracker.etree.org:6969/announce&tr=http://nethd.org/announce.php&tr=http://tracker.minglong.org:8080/announce
03/06/2021 11:53:57.385 PM: Debug (GetTorrentFromMagnetUri) Download torrent from: https://zoink.ch/torrent/{DN}.torrent hash = 599BDF5B5015D8EB4149184A6F2638977F696BBA dn = %5Bzooqle.com%5D%20The%20Blacklist%20S08E02%201080p%20WEB%20H264 url=https://zoink.ch/torrent/%5Bzooqle.com%5D%20The%20Blacklist%20S08E02%201080p%20WEB%20H264.torrent
03/06/2021 11:53:57.385 PM: Debug (MakeRequest) Loading url:https://zoink.ch/torrent/%5Bzooqle.com%5D The Blacklist S08E02 1080p WEB H264.torrent
03/06/2021 11:53:57.469 PM: Warning (.ctor) Request FAILED. Response code: NotFound https://zoink.ch/torrent/%5Bzooqle.com%5D The Blacklist S08E02 1080p WEB H264.torrent
03/06/2021 11:53:58.404 PM: Warning (GetURLContent) Fatal response code received: NotFound. Request aborted: https://zoink.ch/torrent/%5Bzooqle.com%5D%20The%20Blacklist%20S08E02%201080p%20WEB%20H264.torrent
03/06/2021 11:53:58.404 PM: Debug (GetTorrentFromMagnetUri) Download torrent from: https://yts.ag/torrent/download/{MH} hash = 599BDF5B5015D8EB4149184A6F2638977F696BBA dn = %5Bzooqle.com%5D%20The%20Blacklist%20S08E02%201080p%20WEB%20H264 url=https://yts.ag/torrent/download/599BDF5B5015D8EB4149184A6F2638977F696BBA
03/06/2021 11:53:58.404 PM: Debug (MakeRequest) Loading url:https://yts.ag/torrent/download/599BDF5B5015D8EB4149184A6F2638977F696BBA
03/06/2021 11:53:58.757 PM: Warning (.ctor) Request FAILED. Response code: NotFound https://yts.mx/torrent/download/599BDF5B5015D8EB4149184A6F2638977F696BBA
03/06/2021 11:53:59.407 PM: Warning (GetURLContent) Fatal response code received: NotFound. Request aborted: https://yts.ag/torrent/download/599BDF5B5015D8EB4149184A6F2638977F696BBA
03/06/2021 11:53:59.407 PM: Debug (GetTorrentFromMagnetUri) Download torrent from: http://itorrents.org/torrent/{MH}.torrent?title={DNENCODED} hash = 599BDF5B5015D8EB4149184A6F2638977F696BBA dn = %5Bzooqle.com%5D%20The%20Blacklist%20S08E02%201080p%20WEB%20H264 url=http://itorrents.org/torrent/599BDF5B5015D8EB4149184A6F2638977F696BBA.torrent?title=%255Bzooqle.com%255D%2520The%2520Blacklist%2520S08E02%25201080p%2520WEB%2520H264
03/06/2021 11:53:59.407 PM: Debug (MakeRequest) Loading url:http://itorrents.org/torrent/599BDF5B5015D8EB4149184A6F2638977F696BBA.torrent?title=%255Bzooqle.com%255D%2520The%2520Blacklist%2520S08E02%25201080p%2520WEB%2520H264
03/06/2021 11:54:57.647 PM: Debug (_downloadedChecker_Elapsed) 1% or more downloaded for c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv
03/06/2021 11:54:57.647 PM: Debug (BuildRawMediaItemUrl) Build raw media item url: http://192.168.2.3:9000/api/Downloader/GetRawMediaItemStream?DownloaderFileKey=8b2aa9ad-5614-4088-9d46-bf1d53317065&DownloaderKey=9f9e78fa-b8cc-4f99-aee3-32cea5bd9105
03/06/2021 11:54:57.647 PM: Debug (Dmanager_HandleDownloadedFileForPlayback) File added to media player list c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv
03/06/2021 11:54:57.647 PM: Debug (GetMediaFileDuration) Getting duration for http://192.168.2.3:9000/api/Downloader/GetRawMediaItemStream?DownloaderFileKey=8b2aa9ad-5614-4088-9d46-bf1d53317065&DownloaderKey=9f9e78fa-b8cc-4f99-aee3-32cea5bd9105
03/06/2021 11:54:57.731 PM: Debug (GetRawMediaItemStream) Build raw filestream: The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv
03/06/2021 11:54:57.731 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:139b9172-ae19-4ac7-9c5b-84f683886dee) Requested download from: 0 To: 3180775438
03/06/2021 11:54:57.832 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:139b9172-ae19-4ac7-9c5b-84f683886dee) Polling for more bytes from: 0 till 3180775438
03/06/2021 11:54:57.832 PM: Debug (GetRangeDownloadedFromFile) Found range for c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv 0 - 2097022 = 2097023
03/06/2021 11:54:57.832 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:139b9172-ae19-4ac7-9c5b-84f683886dee) Found bytes from: 0 till 2097022
03/06/2021 11:54:57.901 PM: Debug (GetRawMediaItemStream) Build raw filestream: The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv
03/06/2021 11:54:57.901 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:d21e0e3e-4ca3-425d-b098-d5b21dd46cae) Requested download from: 3180774184 To: 3180775438
03/06/2021 11:54:58.013 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:139b9172-ae19-4ac7-9c5b-84f683886dee) Polling for more bytes from: 2097023 till 3180775438
03/06/2021 11:54:58.013 PM: Debug (GetRangeDownloadedFromFile) Found range for c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv 2097023 - 4194174 = 2097152
03/06/2021 11:54:58.013 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:139b9172-ae19-4ac7-9c5b-84f683886dee) Found bytes from: 2097023 till 4194174
03/06/2021 11:54:58.217 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:d21e0e3e-4ca3-425d-b098-d5b21dd46cae) Polling for more bytes from: 3180774184 till 3180775438
03/06/2021 11:54:58.217 PM: Debug (GetRangeDownloadedFromFile) Requesting range for c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv 3180774184
03/06/2021 11:54:58.217 PM: Debug (MoveDownloadCursor) Torrent hash state (c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv hash:MonoTorrent.InfoHash): 0
03/06/2021 11:56:58.580 PM: Warning (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:d21e0e3e-4ca3-425d-b098-d5b21dd46cae) Bytes request timed out.
03/06/2021 11:56:58.580 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:d21e0e3e-4ca3-425d-b098-d5b21dd46cae) Done sending: 3180774184 till 3180775438
03/06/2021 11:56:58.580 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:d21e0e3e-4ca3-425d-b098-d5b21dd46cae) Closing downloader target stream.
03/06/2021 11:57:08.655 PM: Error (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:139b9172-ae19-4ac7-9c5b-84f683886dee) Downloader exeption occured. Stopping this stream. , System.Net.HttpListenerException (0x80004005): The specified network name is no longer available
at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at Microsoft.Owin.Host.HttpListener.RequestProcessing.ExceptionFilterStream.Write(Byte[] buffer, Int32 offset, Int32 count)
03/06/2021 11:57:08.655 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:139b9172-ae19-4ac7-9c5b-84f683886dee) Closing downloader target stream.
03/06/2021 11:57:08.655 PM: Debug (GetRawMediaItemStream) Build raw filestream: The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv
03/06/2021 11:57:08.655 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:d053bf4d-b2d5-4388-9edb-48a628851239) Requested download from: 5687 To: 3180775438
03/06/2021 11:57:08.655 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:d053bf4d-b2d5-4388-9edb-48a628851239) Polling for more bytes from: 5687 till 3180775438
03/06/2021 11:57:08.655 PM: Debug (GetRangeDownloadedFromFile) Found range for c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv 0 - 2097022 = 2097023
03/06/2021 11:57:08.655 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:d053bf4d-b2d5-4388-9edb-48a628851239) Found bytes from: 5687 till 2097022
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: Guessed Channel Layout for Input Stream #0.1 : 5.1
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: Input #0, matroska,webm, from 'async:http://192.168.2.3:9000/api/Downloader/GetRawMediaItemStream?DownloaderFileKey=8b2aa9ad-5614-4088-9d46-bf1d53317065&DownloaderKey=9f9e78fa-b8cc-4f99-aee3-32cea5bd9105':
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: Metadata:
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: encoder : libebml v1.4.0 + libmatroska v1.6.2
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: creation_time : 2020-11-21T08:02:19.000000Z
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: Duration: 00:42:55.71, start: 0.000000, bitrate: 9879 kb/s
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) Duration found: 00:42:55.71
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: Stream #0:0(eng): Video: h264 (High), yuv420p(tv, bt709, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc (default)
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) Track found: Video (eng)
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: Stream #0:1(eng): Audio: eac3, 48000 Hz, 5.1, fltp (default)
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) Track found: Audio (eng)
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: Stream #0:2(eng): Subtitle: subrip
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) Track found: Subtitle (eng)
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: Metadata:
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: title : English
03/06/2021 11:57:08.740 PM: Debug (<GetMediaFileDuration>b__0) ffmpeg output: At least one output file must be specified
03/06/2021 11:57:08.771 PM: Error (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:d053bf4d-b2d5-4388-9edb-48a628851239) Downloader exeption occured. Stopping this stream. , System.Net.HttpListenerException (0x80004005): The specified network name is no longer available
at System.Net.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at Microsoft.Owin.Host.HttpListener.RequestProcessing.ExceptionFilterStream.Write(Byte[] buffer, Int32 offset, Int32 count)
03/06/2021 11:57:08.771 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:d053bf4d-b2d5-4388-9edb-48a628851239) Closing downloader target stream.
03/06/2021 11:57:11.963 PM: Debug (GetMediaFileDuration) Duration request ffmpeg finished: http://192.168.2.3:9000/api/Downloader/GetRawMediaItemStream?DownloaderFileKey=8b2aa9ad-5614-4088-9d46-bf1d53317065&DownloaderKey=9f9e78fa-b8cc-4f99-aee3-32cea5bd9105
03/06/2021 11:57:11.963 PM: Debug (GetMediaFileDuration) Working with duration: 00:42:55.7100000
03/06/2021 11:57:20.714 PM: Debug (GetMediaFileTrancodedURL) Created transcoder settings for file: 8b2aa9ad-5614-4088-9d46-bf1d53317065
03/06/2021 11:57:20.714 PM: Debug (BuildTranscoderMediaUrl) Build transcoded media item url: http://192.168.2.3:9000/api/MediaPlayer/HandlePlayerRequests?DownloaderFileKey=8b2aa9ad-5614-4088-9d46-bf1d53317065&DownloaderKey=9f9e78fa-b8cc-4f99-aee3-32cea5bd9105&PlayerName=HTML5 player mp4
03/06/2021 11:57:20.852 PM: Debug (.ctor) ffmpeg (8b2aa9ad-5614-4088-9d46-bf1d53317065 9e167bc2-16ea-44a3-852e-274a4ad97563) C:\Users\tm\Desktop\mb\Helpers\ffmpeg\ffmpeg.exe -hide_banner -loglevel warning -ss 0 -re -i async:http://192.168.2.3:9000/api/Downloader/GetRawMediaItemStream?DownloaderFileKey=8b2aa9ad-5614-4088-9d46-bf1d53317065&DownloaderKey=9f9e78fa-b8cc-4f99-aee3-32cea5bd9105 -map 0:0 -map 0:1 -c:v libx264 -pix_fmt yuv420p -preset ultrafast -movflags +faststart+empty_moov -c:a aac -ac 2 -ar 44100 -b:a 128k -map_metadata -1 -max_muxing_queue_size 4096 -sn -f mp4 -
03/06/2021 11:57:20.852 PM: Debug (MoveNext) 8b2aa9ad-5614-4088-9d46-bf1d53317065) Starting transcoder stream.
03/06/2021 11:57:20.852 PM: Debug (StartTranscoder) FFmpeg (8b2aa9ad-5614-4088-9d46-bf1d53317065 9e167bc2-16ea-44a3-852e-274a4ad97563) starting ffmpeg session.
03/06/2021 11:57:20.952 PM: Debug (GetRawMediaItemStream) Build raw filestream: The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv
03/06/2021 11:57:20.952 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:dbeaff21-4cba-4ae6-bff5-d7381adc6c97) Requested download from: 0 To: 3180775438
03/06/2021 11:57:20.952 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:dbeaff21-4cba-4ae6-bff5-d7381adc6c97) Polling for more bytes from: 0 till 3180775438
03/06/2021 11:57:20.952 PM: Debug (GetRangeDownloadedFromFile) Found range for c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv 0 - 2097022 = 2097023
03/06/2021 11:57:20.952 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:dbeaff21-4cba-4ae6-bff5-d7381adc6c97) Found bytes from: 0 till 2097022
03/06/2021 11:57:20.968 PM: Debug (GetRawMediaItemStream) Build raw filestream: The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv
03/06/2021 11:57:20.968 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:30f97aa1-dd69-4b95-b3ed-31d0b25b7e9b) Requested download from: 3180774184 To: 3180775438
03/06/2021 11:57:20.968 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:30f97aa1-dd69-4b95-b3ed-31d0b25b7e9b) Polling for more bytes from: 3180774184 till 3180775438
03/06/2021 11:57:20.968 PM: Debug (GetRangeDownloadedFromFile) Requesting range for c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv 3180774184
03/06/2021 11:57:20.968 PM: Debug (MoveDownloadCursor) Torrent hash state (c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv hash:MonoTorrent.InfoHash): 0
03/06/2021 11:57:20.968 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:dbeaff21-4cba-4ae6-bff5-d7381adc6c97) Polling for more bytes from: 2097023 till 3180775438
03/06/2021 11:57:20.968 PM: Debug (GetRangeDownloadedFromFile) Found range for c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv 2097023 - 4194174 = 2097152
03/06/2021 11:57:20.968 PM: Debug (StreamMediaFile) 8b2aa9ad-5614-4088-9d46-bf1d53317065:dbeaff21-4cba-4ae6-bff5-d7381adc6c97) Found bytes from: 2097023 till 4194174
03/06/2021 11:57:22.983 PM: Debug (GetRangeDownloadedFromFile) Requesting range for c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv 3180774184
03/06/2021 11:57:22.983 PM: Debug (MoveDownloadCursor) Torrent hash state (c:\dtest\9f9e78fa-b8cc-4f99-aee3-32cea5bd9105\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM[rarbg]\The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv hash:MonoTorrent.InfoHash): 0
03/06/2021 11:59:20.857 PM: Error (<StreamTranscodedMediaFile>b__0) 8b2aa9ad-5614-4088-9d46-bf1d53317065) Transcoder stream timeout. Stopping transcoder.
03/06/2021 11:59:20.857 PM: Debug (KillTranscoder) FFmpeg (8b2aa9ad-5614-4088-9d46-bf1d53317065 9e167bc2-16ea-44a3-852e-274a4ad97563) killing ffmpeg session.
Are you using the latest stable, or latest unstable?
Is your application opensource? Could I use it for debugging purposes? I can't match your issues with the latest unstable even after seeking to the end. I'll try again tomorrow anyway.
alanmcgovern
commented
3 years ago @borrelnoten I added a little sample to master. If you compile the sample client from master and pass a single argument to it (a magnet link URI) it will use the streaming API to download and seek between pieces in the torrent.
I've attached a screenshot of it here. For this particular torrent this seems about right. It takes a little bit of time to discover and connect to a few live peers, but once that occurs seeking to any piece generally results in the data showing up within a few seconds.
With the magnetlink you used: magnet:?xt=urn:btih:599BDF5B5015D8EB4149184A6F2638977F696BBA
If I use a more popular torrent with a healthy swarm, and a smaller piece size, the performance is better as it is easier to fetch the data:
a ubuntu iso: magnet:?xt=urn:btih:5FFF0E1C8AC414860310BCC1CB76AC28E960EFBE
I'm definitely curious about your use case, but from the stats above I wonder if there's something we can tweak with how you set up and use the engine which might help.
alanmcgovern
commented
3 years ago One thing about the screenshot above is that the ubuntu torrent has 256kB pieces, the other one had 2MB pieces. This is one of the main reasons why performance is better for the ubuntu torrent it only has to download 256kB to get a piece - for the other torrent it needs 8x more data so it could take longer if the number of peers which have unchoked the engine is still low.
borrelnoten
commented
3 years ago I'm testing aswell. Haven't used your routine but very nice of you to dive deep into this matter!
So far I have found 2 errors in my routine that should not affect monotorrent. In hashed event I calculate the range of bytes received for the file I want to download. I was not using ulong and run into overflow causing negative range sized. That is now fixed. Also, the calculation of the pieces to ranges downloaded is not perfect yet. But all that should not affect monotorrent.
I have simulated the same seek events / locations as ffmpeg would but I als can't see a drop in speed yet. What I did find in the torrent is that the file size of the mkv I want is not corrent when I multiply the piece size with amount if pieced for the file. See below for details. Also, I think the drop could be linked to large piece size. Will continue with more testing.
Testing with this torrent (use VPN!!): http://itorrents.org/torrent/599BDF5B5015D8EB4149184A6F2638977F696BBA.torrent?title=%255Bzooqle.com%255D%2520The%2520Blacklist%2520S08E02%25201080p%2520WEB%2520H264
Monotorrent version: v1.0.0-alpha unstable rev0132
piece size: 2097152
File I download in the torrent: The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv
Start piece: 0
End piece: 1516
File length according to the torrent: 3180775439
Doing the math (end piece number time the piece length) I get this very big file size: 3179282432
file offset (I guess in bytes): 129
seek sequence executed: 0, 2097023, 3180774184
Looking at your example code I found this: Can't try the command below. Missing cancalation token parameter: var stream = await manager.StreamProvider.CreateStreamAsync (largestFile, false);
Instead I do this: torrentStream = await torrentStreamProvider.CreateStreamAsync(torrentStreamProvider.Files[findex], false, new CancellationToken());
the torrentStreamProvider is created like this: torrentStreamProvider = new StreamProvider(engine, GetDownloaderData().FullDownloaderPath, torrent); await torrentStreamProvider.StartAsync();
alanmcgovern
commented
3 years ago I wonder if it would be simpler if you used the stream Monotorrent creates in the StreamProvider.
If ffmpeg wants to read 2048bytes at offset 1,000,000 then you could just seek monotorrent's stream to offset 1,000,000 and then read 2048 bytes.
This way you don't need to do any math to convert offsets/ranges to piece indexes, and you don't have to manually check if the piece is downloaded. If the stream actually reads the data then you know everything is good.
Its possible that if there's a bug in your logic and you calculate different piece indexes to the engine, then you'll block waiting for a piece that the engine isn't even trying to download. It seems unlikely, but maybe it's possible?
The latest master version has a change to make it simpler to calculate these kinds of offsets by exposing extension methods which can be called on a TorrentManager instance.
https://github.com/alanmcgovern/monotorrent/commit/d18276a33ffa27e09444f6ddb2ed0e4c92f86bbd
If you copy/paste these then it might help rule out differences in calculations.
borrelnoten
commented
3 years ago Unfortunalty my system needs to know if a range of bytes in a file is available for use. I make use of other protocols to get data aswell. This is why I use a generic way to get a file range readyness.
I looked into the BytesPerPiece funtion but it seems it only calculates the start byte of a piece in the entire torrent. I try to calculate the piece byte position relative to the file I am downloading. A piece can belong to more than 1 file at the same time so the start position of a file is not pieceindex piece length. Also, the end position is not always the pieceindex piece length but can be less. That last part is easy to get as long as we get the correct relative file start position.
The way I do that is by multiplying the piece with the piece size and next substract the file start byte position. That should get the current file piece position:
int pieceInFileNr = e.PieceIndex;
int pieceLength = e.TorrentManager.Torrent.PieceLength;
long Rangepstartbyte = (long) ((ulong)pieceLength * (ulong)pieceInFileNr);
if (e.PieceIndex > 0 )
{
Rangepstartbyte -= f.OffsetInTorrent; // offet of file in torrent in bytes
}long Rangependbyte = Rangepstartbyte + pieceLength - 1;
if (Rangependbyte > f.Length)
{
Rangependbyte = f.Length;
}Unfortunatly, when I multiply the file end piece with the piece size, I do not get the file size. This I do not understand. My routine is not waiting for a piece to become available / stall the programm by waiting. Also it is not steering monotorrent to a piece that does not exists (because ffmpeg is steering the download cursor in byte position and I use that to seek) like so:
public override void MoveDownloadCursor(Int64 offset)
{
var cf = GetCurrentlyDownloadingFile();
cur_torFileIndexNum = MediaBrowserTorrentFile.IndexOf(cf.DownloaderFile_Key);
// don't switch cursor if not needed whe already file is downloaded compleetly
if (torrentStreamProvider.Files[cur_torFileIndexNum].BytesDownloaded() != torrentStreamProvider.Files[cur_torFileIndexNum].Length)
{
//MediaBrowserUtils.Logging.WriteMessage("Torrent hash state ("+ torrentStreamProvider.Files[cur_torFileIndexNum].FullPath + " hash:" + torrent.InfoHash + "): " + torrentStreamProvider.Manager.HashFails, EntityData.MediaBrowserUtils.LogLevel.Debug);
// switch to other file only when is not current file
if (old_torFileIndexNum != cur_torFileIndexNum)
{
old_torFileIndexNum = cur_torFileIndexNum;
SetTorrentFileFocus(cur_torFileIndexNum).Wait();
old_offset = -1;
}
if (torrentStream != null && old_offset != offset)
{
old_offset = offset;
Console.WriteLine("Seeking to: " + offset);
torrentStream.Seek(offset, SeekOrigin.Begin);
}
}
else
{
cur_torFileIndexNum = old_torFileIndexNum;
}
}I'm a little curious about the logic used to set the focus. It looks like offset is an index into the active file. As such I don't really know why the logic there couldn't be simplified into just:
torrentStream.Seek(offset, SeekOrigin.Begin);
And remove the rest of the method body.
If you start feeding a different file into ffmpeg then you should set monotorrent to download that file, but otherwise MoveDownloadCursor looks like it should never be used to change the active file.
(I'd need more context to really understand what's going on though. It just looks a bit odd)
borrelnoten
commented
3 years ago yes :) so the logic is: the use can select any playable (media) file in a torrent. When there are more than one, the seek function must know if the torrentstream is focused into the correct file.
In the test you see multiple projects. I use the torrent2020 for the monotorrent downloader. In the past I used libtorrent wrapper for torrent downloading. Worked fine but did not always released the downloaded files after done and also the dll's were kept active even when disposed.
I still can't see how the piece size times the pices index number of the last piece of the file does not equals to the file size (should exact or bigger but never smaller since it's the last piece of the file).
piece size: 2097152
File I download in the torrent: The.Blacklist.S08E02.1080p.WEB.H264-STRONTiUM.mkv
Start piece: 0
End piece: 1516
File length according to the torrent: 3180775439
Doing the math (end piece number time the piece length) I get this smaller file size: 3179282432
file offset (I guess in bytes): 129
So what am I missing here? IS the piece index zero based meaning I would need to add 1 and then multiply that with the pice size?
That would mean the size would then be:
3181379584 which includes piece 1516 + 1 So the last piece 1516 + 1 would overlap into the next file since the mkv file size is 3180775439
[update]
I've now tested a bit more and improved the calculations to get the correct bytes ranged relative to file on disc. Seems to work Also, now that the bugs are out of the routine, I will retest with other magnet URLS and torrent to see if the drop is still there or not.
private void Manager_PieceHashed(object sender, PieceHashedEventArgs e)
{
//Console.WriteLine(e.PieceIndex + " : " + e.HashPassed);
try
{
if (engine != null && torrentStreamProvider != null
&& GetDownloaderData().RoutineStatus == EntityData.MediaBrowserDownloader.DownloaderRoutineStatus.Started)
{
if (e.HashPassed)
{
//engine.DiskManager.FlushAsync(e.TorrentManager.Torrent).Wait();
ulong pieceInFileNr = (ulong)e.PieceIndex;
ulong pieceLength = (ulong)e.TorrentManager.Torrent.PieceLength;
// update the media browser file range with the newly downloaded piece
int inum = 0;
foreach (var f in e.TorrentManager.Files)
{
// does this piece belog to the file we are looking at now?
if (e.PieceIndex >= f.StartPieceIndex && e.PieceIndex <= f.EndPieceIndex)
{
//long bytePosOfPiece = e.TorrentManager.Torrent. BytesPerPiece
// calc start and end byte pos of the piece downloaded for this file
long Rangepstartbyte = (long) (pieceLength * pieceInFileNr);
Rangepstartbyte -= f.OffsetInTorrent; // substract file start offset (equals previous files lengths added up till this file)
if (Rangepstartbyte<0)
{
Rangepstartbyte = 0;
}
long Rangependbyte = (long) ((ulong)Rangepstartbyte + pieceLength - 1);
Rangependbyte -= f.OffsetInTorrent; // substract file start offset (equals previous files lengths added up till this file)
if (e.PieceIndex == f.EndPieceIndex)
{
Rangependbyte = f.Length;
}
Console.WriteLine(inum + " : " + f.FullPath + "(Piece hashed: "+ e.PieceIndex + ")" + " bytes from: " + Rangepstartbyte + " to: " + Rangependbyte);
// update the range for the media browser file
var mbfileg = MediaBrowserTorrentFile[inum];
GetRangeByFileID(mbfileg).AddNewRange(Rangepstartbyte, Rangependbyte);
// check if this file is done.
if (f.BytesDownloaded() >= f.Length)
{
// update current file to completed
GetDownloadFileByKey(mbfileg).FileStatus = MediaBrowserEntityData.EntityData.MediaBrowserDownloader.DownloaderFileStatus.Completed;
// complete range to be sure we can deliver entire file
GetRangeByFileID(mbfileg).AddNewRange(0, f.Length);
// check if this was the current file we were dowloading, if so, more focus to a file that is not yet downloaded
if (inum == cur_torFileIndexNum)
{
// is current file, so look for other file to download
for (int i = 0; i < e.TorrentManager.Files.Count; i++)
{
// find a file that is not yet downloaded and put the focus on it:
if (e.TorrentManager.Files[i].BytesDownloaded() < e.TorrentManager.Files[i].Length)
{
MarkFileForDownload(MediaBrowserTorrentFile[i]);
break;
}
}
}
}
}
inum++;
}
}
}
}
catch { }
}file offset (I guess in bytes): 129 Yes - the file at index '2' starts at byte 129 in the context of the whole torrent. There are two files before it.
Doing the math (end piece number time the piece length) I get this smaller file size: 3179282432
Some, or all, of the final piece is data for the file you're looking at. The smaller value you calculated is accurate in that it's 'a few thousand bytes' from the actual end of the mkv file. I'm not sure what you'd use this value for - it's just an arbitrary point near the end of the file.
That would mean the size would then be:
No - the size of the file is manager.Files[2].Length. The value you computed is 3180775439 - 3179282432 ( can be calculated as manager.Files[2].Length - (long) manager.PieceLength * manager.Files[2].EndPieceIndex) bytes from the end of the file, which is 1,493,007 bytes. The first 1,493,007 bytes in piece 1516 (the final piece in the torrent) are bytes for the mkv. The next 213 bytes are for the nfo, and the final 100,729 bytes are for the jpg.
The final piece in the torrent is only 1,594,078 bytes long. The piecelength is 2,097,152 so the final piece is smaller than the standard piece length, which is pretty normal. Torrents are rarely exact multiples of the piece length in size.
Does that help?
borrelnoten
commented
3 years ago Tnx for the quick response! I've also updated the routine and seems now to give correct bytes available ranges. For some reason, FFMPEG like to download the file end (think it is related to the amount of "blocks" a viedeo file has which is not always in the file header).
I've now tested and I do see a drop for magnet urls with very poor seeding. For a bit healthy magnets, a small drop when seeking but it also build back to normal speed in matter of seconds. For me I can close the issue. I've learned that the drop is related to the focus onto the high prio pieces and switching to other pieces will pull away focus on current pieces causing a short download drop. Also, the piece size has impact on the drop when seeking.
alanmcgovern
commented
3 years ago Sounds good! Glad you got it figured out and that things are working well at the moment.
If you have further issues let me know. Hopefully the newer releases will be even better!
Testest latest pre release. Tested with 6 torrents. Download start is perfect but as soon as i seek to end of file the download speed drop to 100 or lower bytes per second and is not speeding.up.after 5 minutes waiting. This worked better in previous version but also last piece did not download until all other piecea were downloaded.