Improved Downloads

[ast3150]

Summary

• Improve OCI download performance by performing multiple downloads in parallel
• Make OCI downloads interruptible and resumable
• Reduce resource usage of OCI downloads

Reason

• Downloads were relatively slow using the existing approach, compared to what the network can offer
• Downloads could not be restarted previously, meaning the user has to download everything from scratch if the download fails
• Downloads used a high memory footprint due to dealing with in-memory Data

Solution

• Changes OCI download logic to use AsyncHTTPClient and FileDownloadDelegate (based on SwiftNIO) instead of URLSession.bytes
• Changes download and decompression logic to use byte streams
• Changes OCI download logic to download disk layers into chunk files
• Parallelizes OCI chunk downloads to max out network performance

Context

Closes #31

Work In Progress

• [x] Downloaded chunks should be removed after completed download
• [x] Progress indication for image decompression
• [x] Retry decompressing image if chunks are still present and download is not marked as completed

[Marcocanc]

Hi @ast3150, first of all I really appreciate the contribution! A few things have changed since the latest tart release though, which we might be able to leverage to implement a cleaner solution to accelerating download speeds. (Blog post). I will look into the detailed changes and see if we can reuse this solution or find a simpler one. In any case, I would like to separate the download and unzipping logic into a separate class.

[Marcocanc]

Tart's implementation even supports resuming and we will have to implement support for this anyway, as the Sonoma images are only using V2 disk layers.

https://github.com/cirruslabs/tart/blob/4bf9bdd531e955f1f49794a62701714a09952a4d/Sources/tart/OCI/Layerizer/DiskV2.swift#L35

[Marcocanc]

Update: I started working on splitting the download/unzipping logic out and building a tart-inspired resumable concurrent download solution. Will close this ticket for now.