Automattic / hostmgr

A tool for managing macOS VM hosts
Mozilla Public License 2.0
9 stars 3 forks source link

Increase S3 client timeout to 2 hours #36

Closed crazytonyli closed 2 years ago

crazytonyli commented 2 years ago

Currently, only 90% of the image was downloaded in 30 minutes, I'm not entirely sure why the downloading speed is so slow. In this PR, the timeout is increased to 2 hours—we can potentially reduce this if later find the cause of the slow downloading speed.

One suspect is the logging, there are lots of trace logs from the soto libraries which might get in the way of downloading from S3 bucket. This PR changes the default logging level to info, but with an option to set it using environment variable, like LOG_LEVEL=trace hostmgr sync vm_images. I tried to add a --verbose global option, but I don't think swift-argument-parser supports specifying a global option at the top level command.

Another change is removing Hostmgr.run function, which does two things: print version or help message. They are replaced with the version flag came with CommandConfiguration and the help flag came with [the default run implementation](https://apple.github.io/swift-argument-parser/documentation/argumentparser/parsablecommand/run()-7p2fr). So, no CLI is changed, but we have less code.

AliSoftware commented 2 years ago

Random ideas why the transfer might have been slow:

This is complete speculation from my end no idea if any of those could really be the culprit in practice but figured those are worth checking as a starting point

crazytonyli commented 2 years ago

Cutting down logging seems to improved the image download speed:

2022-08-31T08:01:16+0000 info com.automattic.hostmgr : Downloading the VM – this will take a few minutes
2022-08-31T08:01:16+0000 info com.automattic.hostmgr : Using Accelerated S3 Download
2022-08-31T08:25:25+0000 info com.automattic.hostmgr : Download Complete