Suggesting to introduce an optional link_only boolean parameter in file_sources_conf.yml (default: false). When enabled, datasets are linked rather than copied during uploads. Although the upload/fetch mechanism already supports link_only uploads, this functionality is not currently configurable for file sources. By enabling this feature, users can avoid duplicating large datasets when the remote file source resides on a filesystem already mounted to their compute nodes. Potentially this can also be done using the file source type attribute.
Ping @jmchilton for suggestions and comments. Thanks in advance.
guerler
commented
2 weeks ago
Suggesting to introduce an optional
link_onlyboolean parameter infile_sources_conf.yml(default: false). When enabled, datasets are linked rather than copied during uploads. Although the upload/fetch mechanism already supportslink_onlyuploads, this functionality is not currently configurable for file sources. By enabling this feature, users can avoid duplicating large datasets when the remote file source resides on a filesystem already mounted to their compute nodes. Potentially this can also be done using the file source type attribute.Ping @jmchilton for suggestions and comments. Thanks in advance.