saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
13.98k stars 5.47k forks source link

[BUG] Performance of get_hash() method. #66651

Open slackfan opened 1 week ago

slackfan commented 1 week ago

Description

The methods modules/file.py/get_hash() and util/hashutil.py/get_hash() use a default chunk size of 64KB. The method is e.g. used by the file.managed state using that default chunk size without any ability to modify the chunk size.

The impact of that is that the hash compution of a 12 GB file we have to deal with, may require up to 30 minutes and delay the instance provisioning. This may happen if the file is already cached and present in target destination. Means, we wait 30 minutes for - not so much.

Setup

Salt minion 3006.8 on Windows Server 2019.

Links

via states/file.py/managed() -> modules/file.py/get_managed()

Expected behavior

My request would be to improve the performance by either reviewing the default chunk size, make it configurable, or review the logic of the file.managed state.

Screenshots 2024-06-19_14h37_47

welcome[bot] commented 1 week ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!