cwl-tes currently downloads all input files. However, in a setting where tasks are computed in TES, cwltool/cwl-tes will only need access to the files in cases tasks need to be created/scheduled dynamically depending on file content. In most cases, this probably won't be necessary and just makes the implementation slow and inflexible.
I would propose to implement a flag to explicitly download all input files, so that cwl-tes's default behavior is to download only those files it really needs to inspect.
There is some discussion here on where/how this could be implemented.
cwl-tescurrently downloads all input files. However, in a setting where tasks are computed in TES,cwltool/cwl-teswill only need access to the files in cases tasks need to be created/scheduled dynamically depending on file content. In most cases, this probably won't be necessary and just makes the implementation slow and inflexible.I would propose to implement a flag to explicitly download all input files, so that
cwl-tes's default behavior is to download only those files it really needs to inspect.There is some discussion here on where/how this could be implemented.