New features

[corentincarton]

Is your feature request related to a problem? Please describe.

Some things that should be changed for next release:

• Get rid of definition file and talk directly to the ecflow server (via pyflow? or tracksuite?)
• Handle multiple requests in static data config
• Rename parse_static_data_item into parse_data_item
• Add postscript to tools
• Find a way to specify module version through ecflow variables (maybe an optional "variable" option could be added to all the tools)
• Cleanup build folder automatically when using "files" in data
• Update install procedure in readme and docs to use pyflow-wellies from PyPI directly
• print versions of wellies, pyflow and tracksuite in quickstart
• point to readthedocs for the documentation https://pyflow-wellies.readthedocs.io
• add copy_options for the data type "copy" (as we have for rsync)
• add description execution context through Troika in the documentation
• add mars_command to static_data yaml config
• add data_dir attribute for StaticData class to retrieve the path where the data is stored
• As for the data store, we could create the same for archiving, which would contain script snippets to push data to remote locations (mars, ECFS, rsync, scp, ftp, etc.)
• remove data dir before installing new data

[colonesej]

Add version as ecflow variable at suite level for any module provided. Use wellies tools key, which is already unique, as variable name. example

tools:
  modules:
    python:
      name: python3
      version: 3.10
   python_new:
      name: python3
      version: 3.12

pf.Suite(
   (...)
   variables={
      "PYTHON_VERSION":"3.10",
      "PYTHON_NEW_VERSION": "3.12",
  },
  (...)
)

[corentincarton]

This is something we discussed with @oiffrig a few weeks ago as it was needed in one of the operational suites. There's a way to do this without modifying the wellies code:

tools:
  modules:
    python:
      name: python3
      version: ${PYTHON_VERSION:-3.10}
   python_new:
      name: python3
      version: ${PYTHON_VERSION_NEW:-3.12}

The good thing with this approach is that it doesn't pollute your list of variables by default and you can just add the variable to the suite when you need it. But I agree it's not straightforward, so maybe it could be featured in the documentation or the quickstart?

[colonesej]

How about changing pyflow to identify bash variables in scripts with defaults and treat them as exportables even if not defined on a node/tree?

[corentincarton]

Ah yes, you're right, it will need to be identified in the script generation otherwise the variable will not be exported. Not sure it's safe to do that automatically in pyflow... I would prefer your first suggestion. What bothers me is that we do all sorts of gymnastics when we could use directly ecflow variables with default values %PYTHON_VERSION:3.12% :D

[colonesej]

Another thing that came to me (it might not belong to wellies but maybe tracksuits, not sure):

• Deploy one yaml file with the final parsed configuration used to generate the suite to deploy_dir
• add, as a comment, on top of the config a message including timestamp and git-based info of source project, example # deployed with tracksuite from efas@release/5.1+12314312321asdve324234

Second one could be replaced by tracksuite commit message but do we include there information about the git state of the source project by default?

[corentincarton]

Yes we could definitely do that! It doesn't belong in tracksuite, but more in wellies. We already have add a wellies-specific message which includes the version of wellies used to deploy the suite, but we could add the local git hash/branch/tag as well if we find one.