CWLProv: add option exclude raw copies of the data

[mr-c]

https://matrix.to/#/!RQMxrGNGkeDmWHOaEs:gitter.im/$AJGFCdt6jVAn3aR5lQ0PK3_0SGgvFrubf5SMClsOgGA (a.k.a https://gitter.im/common-workflow-language/common-workflow-language?at=61d6a7bfbfe2f54b2e04661d )

• [ ] --prov-exclude-inputs Skips copying the input files into the CWLProv ResearchObject
• [ ] --prov-exclude-intermediates Skips copying the intermediate files into the CWLProv ResearchObject
• [ ] --prov-exclude-outputs Skips copying the output files into the CWLProv ResearchObject

[jjkoehorst]

Thanks for creating the ticket for me personally only the metadata (rdf files, workflows files) are needed. As the input and output files are preserved on a cloud store.

[mr-c]

Areas to investigate, (add flag to skip the copying, but still calculate and store the checksums)

https://github.com/common-workflow-language/cwltool/blob/a1e3449560b964d90818b2f1bfeb9b411415a786/cwltool/provenance.py#L790 https://github.com/common-workflow-language/cwltool/blob/a1e3449560b964d90818b2f1bfeb9b411415a786/cwltool/provenance.py#L929

https://github.com/common-workflow-language/cwltool/blob/a1e3449560b964d90818b2f1bfeb9b411415a786/cwltool/provenance.py#L741 called from https://github.com/common-workflow-language/cwltool/blob/a1e3449560b964d90818b2f1bfeb9b411415a786/cwltool/main.py#L1413

https://github.com/common-workflow-language/cwltool/blob/a1e3449560b964d90818b2f1bfeb9b411415a786/cwltool/provenance.py#L894

[jjkoehorst]

To update this, when providing Directory or Files as input it will copy the entire content to /tmp. Solution for now is to use Strings instead of Directory when possible.

[mr-c]

Solution for now is to use Strings instead of Directory when possible.

FYI, while that may work for now, that will break mulit-node execution of the workflow