OutOfMemoryException when searching long playlists

[03stevensmi]

SubTubular.exe c https://www.youtube.com/@TheVisualMediaArchive98 -f bath -t 7000 -h 0

on Microsoft Windows NT 10.0.19041.0 .NET 7.0.19 SubTubular 3.0.1+7eda30ff

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags) at System.GC.g__AllocateNewUninitializedArray|660[T](Int32 length, Boolean pinned) at System.Buffers.TlsOverPerCoreLockedStacksArrayPool1.Rent(Int32 minimumLength) at System.Text.Json.JsonDocument.MetadataDb.CreateRented(Int32 payloadLength, Boolean convertToAlloc) at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedArrayPoolBytes, PooledByteBufferWriter extraPooledByteBufferWriter) at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 json, JsonDocumentOptions options) at System.Text.Json.JsonDocument.Parse(String json, JsonDocumentOptions options) at YoutubeExplode.Utils.Json.Parse(String source) in //YoutubeExplode/Utils/Json.cs:line 44 at YoutubeExplode.Bridge.PlaylistNextResponse.Parse(String raw) in //YoutubeExplode/Bridge/PlaylistNextResponse.cs:line 66 at YoutubeExplode.Playlists.PlaylistController.GetPlaylistNextResponseAsync(PlaylistId playlistId, Nullable`1 videoId, Int32 index, String visitorData, CancellationToken cancellationToken) in //YoutubeExplode/Playlists/PlaylistController.cs:line 100 at YoutubeExplode.Playlists.PlaylistClient.GetVideoBatchesAsync(PlaylistId playlistId, CancellationToken cancellationToken)+MoveNext() in //YoutubeExplode/Playlists/PlaylistClient.cs:line 88 at YoutubeExplode.Playlists.PlaylistClient.GetVideoBatchesAsync(PlaylistId playlistId, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult() at YoutubeExplode.Utils.Extensions.AsyncCollectionExtensions.SelectManyAsync[TSource,T]+MoveNext() in //YoutubeExplode/Utils/Extensions/AsyncCollectionExtensions.cs:line 34 at YoutubeExplode.Utils.Extensions.AsyncCollectionExtensions.SelectManyAsync[TSource,T](IAsyncEnumerable1 source, Func2 transform)+MoveNext() in //YoutubeExplode/Utils/Extensions/AsyncCollectionExtensions.cs:line 34 at YoutubeExplode.Utils.Extensions.AsyncCollectionExtensions.SelectManyAsync[TSource,T](IAsyncEnumerable1 source, Func2 transform)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult() at YoutubeExplode.Utils.Extensions.AsyncCollectionExtensions.TakeAsync[T](IAsyncEnumerable`1 source, Int32 count)+MoveNext() in //YoutubeExplode/Utils/Extensions/AsyncCollectionExtensions.cs:line 22 at YoutubeExplode.Utils.Extensions.AsyncCollectionExtensions.TakeAsync[T](IAsyncEnumerable1 source, Int32 count)+MoveNext() in /_/YoutubeExplode/Utils/Extensions/AsyncCollectionExtensions.cs:line 22 at YoutubeExplode.Utils.Extensions.AsyncCollectionExtensions.TakeAsync[T](IAsyncEnumerable1 source, Int32 count)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult() at YoutubeExplode.Utils.Extensions.AsyncCollectionExtensions.ToListAsync[T](IAsyncEnumerable1 source) in /_/YoutubeExplode/Utils/Extensions/AsyncCollectionExtensions.cs:line 49 at YoutubeExplode.Utils.Extensions.AsyncCollectionExtensions.ToListAsync[T](IAsyncEnumerable1 source) in //YoutubeExplode/Utils/Extensions/AsyncCollectionExtensions.cs:line 49 at YoutubeExplode.Common.BatchItemExtensions.CollectAsync[T](IAsyncEnumerable`1 source, Int32 count) in //YoutubeExplode/Common/IBatchItem.cs:line 39 at SubTubular.Youtube.GetPlaylistAsync(SearchPlaylistCommand command, CancellationToken cancellation) in D:\projects\SubTubular\SubTubular\Youtube.cs:line 109 at SubTubular.Youtube.SearchPlaylistAsync(SearchPlaylistCommand command, CancellationToken cancellation)+MoveNext() in D:\projects\SubTubular\SubTubular\Youtube.cs:line 35 at SubTubular.Youtube.SearchPlaylistAsync(SearchPlaylistCommand command, CancellationToken cancellation)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult() at SubTubular.Program.SearchAsync(SearchCommand command, String originalCommand, Func2 getResultsAsync) in D:\projects\SubTubular\SubTubular\Program.cs:line 150 at SubTubular.Program.SearchAsync(SearchCommand command, String originalCommand, Func2 getResultsAsync) in D:\projects\SubTubular\SubTubular\Program.cs:line 150 at SubTubular.Program.SearchAsync(SearchCommand command, String originalCommand, Func2 getResultsAsync) in D:\projects\SubTubular\SubTubular\Program.cs:line 172 at CommandLine.ParserResultExtensions.WithParsedAsync[T](ParserResult1 result, Func`2 action) at SubTubular.Program.Main(String[] args) in D:\projects\SubTubular\SubTubular\Program.cs:line 35

[03stevensmi]

I can confirm it work with -t 1000 but not -t 7000. Is it possible to increase this limitation for bigger YT Channels or playlists? it also runs extremely slow with -t 4500 set. if its not a great idea to increase the limit and make it faster, then could something like a "continue where left off" option be possible. or something like "--start-from [NUMBER]"?

[h0lg]

Hey @03stevensmi, I've thought about your proposal for a bit and think it has merit. The "continue where left off" option is not a route I want to go down, as the "--start-from [NUMBER]" one offers similar capabilities but more flexibility.

My current mind is to introduce a --skip [number of videos] parameter and rename the existing --top one to --take.

Tackling the performance issues you're running into with long playlists requires some fundamental changes to the way the search indexes are stored, refreshed and searched. I'll try to improve performance for this scenario in an upcoming version by

• returning early from playlist refresh: Waiting for the playlist refresh only until we start continuously hitting cached video IDs and then starting the search on the (now assumed fresh) playlist while continuing to refresh it in the background, notifying you about unexpected changes to made by the background refresh.
• index sharding: Splitting the search index into multiple files that index different ranges of videos in the playlist. That should yield results earlier by make loading and searching individual shards quicker and save memory by loading less data into it at a time.

Hang tight while I'm trying to work out the kinks of the next version.

[03stevensmi]

Thanks man. I'll leave it with you. If anyone I trust to get this sorted, it'll definitely be you! Good Luck! ;)