Audit - optimize using segment io digests for fixity

[dloy]

Problem

Current digest generation for performing fixity requires downloading the entire cloud file to local storage and then creating a digest on that file as a single IO stream. This was initially used to avoid the problem of partial/failed downloads on large content.

The downloading of large files locally has two problems:

• time required to write content to local disk
• blowing out local storage because multi-threading and system failures blow out local temp space (here 1T).

Solution

This solution was suggested by the following article: Streaming large objects from S3 with ranged GET requests

Digest creation is handled by iteratively modifying the digest using blocks of data. The typical approach:

• download remote object to local file
• get input stream for object requiring a digest
• fill local buffer
• add buffer content to construction of digest
• using single input stream refill buffer
• continue until all input stream content complete
• complete digest creation
• delete file

Segment approach:

• read a block of remote content into the local buffer
• add buffer content to construction of digest
• continue until all remote content has been read and added to digest

With the use of partial S3 reads off cloud storage, the local file can be eliminated. The trade-off eliminates temp space but does require additional memory.

Tests

Current testing indicates:

• At least twice the speed of using a file
• Buffer sizes in the 32M range appear to optimize the use of large content 1G+

Timing on 4.66G: AWS - 143.7 sec Wasabi - 135 sec SDSC - 155.2

Install CloudChecksum into Audit

• [ ] confirm CloudChecksum works
• [ ] confirm CloudChecksum status match works
• [ ] Incorporate CloudChecksum as part of Audit process
• [ ] Stage testing

[dloy]

The original technique ran extremely fast on a large EC2 box. When running on a small, used by audit, it ran very close to hosing the system and throughput was roughly 10% of what I was seeing on my large server (sandbox2).

The issue was memory.

Two changes allowed it to run at close to the speed on the larger box:

• be able to dynamically set the max size for the buffer
• dynamically set the buffer to the cloud reported size of the content unless it is larger than the max buffer size

Successfully running on:

• uc3-mrtaudit01x2-stg
• uc3-mrtaudit02x2-stg