Concurrent ChunksDownloader

tungda-ccbji commented 2 years ago

On the method QueryAsync, you do have a check for Chunks > 0, then call ChunksDownloader to get data from all the chunks, however, you missed this on method QueryRawResponseAsync.

Another thing I think that can be improved is the ChunksDownloader, currently, you are downloading multiple chunks 1 by 1. When I tried to make a query that returned around 14000 records, it took about 5 seconds to complete. If you let these chunks run by themselves on multiple tasks, then merge the result once all the tasks are completed, it could reduce the time to under 1 second.

grexican commented 1 year ago

If you do that you have to merge them in the correct order

Right now I'm facing an issue with the chunk data being in the incorrect order. Trying to determine if the chunk data is wrong or if the way it's being processed by this library is wrong.

tungda-ccbji commented 1 year ago

Yes, you need to merge the result in the correct order, which can be completed easily by marking each task with a number, then ordering by the number before merge the chunks. Something like this:

        var result = new ConcurrentDictionary<int, List<List<string>>>();
        var tasks = new List<Task>();
        foreach (var chunk in chunksDownloadInfo.Chunks)
        {
            var downloadRequest = BuildChunkDownloadRequest(chunk, chunksDownloadInfo.ChunkHeaders, chunksDownloadInfo.Qrmk);
            var task = Task.Run(() => result.TryAdd(chunksDownloadInfo.Chunks.IndexOf(chunk), GetChunkContentAsync(downloadRequest, ct).Result));
            tasks.Add(task);
        }

        Task.WaitAll(tasks.ToArray());
        var orderedResult = result.OrderBy(x => x.Key).ToList();
        foreach (var chunk in orderedResult) 
        {
            rowSet.AddRange(chunk.Value);
        }

grexican commented 1 year ago

Yep, agreed.

Also, updating my client library from .4.0 to .4.3 seems to have solved my issue with data being in the incorrect order.