Limit size of archives put to HSI - Githubissues

nd-nuclear-theory / mcscript

Scripting package for control of families of related runs on an HPC cluster

MIT License

1 stars 0 forks source link

Limit size of archives put to HSI #2

Open kc9jud opened 4 years ago

kc9jud commented 4 years ago

According to the NERSC HPSS documentation (https://docs.nersc.gov/filesystems/archive/#avoid-very-large-files), files over 2TB are inefficient when put to HPSS. They recommend breaking files up into 500GB chunks if they get over that limit.

The hsi handler mcscript.task.archive_handler_hsi() should:

[x] Inspect the size of an archive file to put.
[x] Put the file directly using hsi if smaller than threshold, or
[x] use split to break up the file into smaller segments and put those with hsi.

In addition (so that consumers of archives don't need to be aware of this splitting behavior):

[ ] mcscript should also provide a wrapper function to fetch and reassemble archives,

kc9jud commented 4 years ago

c849af4887cd94fe9141cbdd11eac50ba3a8799a implements the logic in archive_handler_hsi()