Specify ECM definitions with user arguments

[aspeake]

Fixes https://github.com/trynthink/scout/issues/365

Introduces new arguments to allow users to choose a subset of ECM definitions. The summary of updates includes:

• --ecm_directory arg: optional argument to choose a location for ECMs that differs from the default ./ecm_definitions directory
  • If present, this argument will update the FilePaths.ECM_DEF variable
• --ecm_files arg: optional argument that takes a list of ECM files that must match exactly with files located in --ecm_directory
• --ecm_files_regex arg: optional argument that takes a list of regexs that will be matched to files in --ecm_directory
• Moved the FilePaths class into config.py. These are not technically constants as there is potential for ECM_DEF to be updated during argument parsing, and it makes sense to be lumped in with configuration. Deleted constants.py.
• --ecm_packages arg: list of packages that match those in package_ecms.json, default is []
• Because FilePaths is now dynamic, added capabilities that store the baseline variables, allow updates to variables, and allows one to revert back to baseline variables (currently used in testing)
• Add check to ensure all contributing ECMs in a package are present before prepping the package

Example usage in a yml file is provided below. This example uses a custom ecm_definitions directory, and from those definitions selects two files explicitly as well as all files that match the ecm_files_regex parttern.

ecm_prep:
  ecm_directory: "./my_custom_ecms"
  ecm_files: ["My Custom ECM 1", "My Custom ECM 2"]
  ecm_files_regex: [".*Air Sealing*"] 

[trynthink]

How is run.py finding out what ECMs to include? Is it drawing from generated/ecm_prep.json or is it based on the ECMs specified in the YAML?

[aspeake]

How is run.py finding out what ECMs to include? Is it drawing from generated/ecm_prep.json or is it based on the ECMs specified in the YAML?

run.py is unchanged so it should continue drawing from ecm_prep.json. As touched on in another comment, there may be potential for prepared measures in that file that are not specified in the yaml if ecm_prep.py is run twice. We may want to discuss if/how to address that.

[aspeake]

How is run.py finding out what ECMs to include? Is it drawing from generated/ecm_prep.json or is it based on the ECMs specified in the YAML?

run.py is unchanged so it should continue drawing from ecm_prep.json. As touched on in another comment, there may be potential for prepared measures in that file that are not specified in the yaml if ecm_prep.py is run twice. We may want to discuss if/how to address that.

This will be addressed by setting only the ECM files in the current yml (opts.ecm_files) as active in run_setup.json

[aspeake]

How is run.py finding out what ECMs to include? Is it drawing from generated/ecm_prep.json or is it based on the ECMs specified in the YAML?

run.py is unchanged so it should continue drawing from ecm_prep.json. As touched on in another comment, there may be potential for prepared measures in that file that are not specified in the yaml if ecm_prep.py is run twice. We may want to discuss if/how to address that.

This will be addressed by setting only the ECM files in the current yml (opts.ecm_files) as active in run_setup.json

Fixed in https://github.com/trynthink/scout/pull/372/commits/c8e6bdc234cc9276fa87c27394701a1163a1d8a7

[aspeake]

@trynthink this commit defaults ecm packages to [] and automatically pulls in ECMs if they contribute to a package.

Note that this warning: https://github.com/trynthink/scout/blob/select_ecms/scout/ecm_prep.py#L13139-L13141 is still valid as there could be contributing ECMs listed that do not exist in whatever ECM definitions directory was specified.

This warning is also valid, as the ECMs that are part of a package still do not run individually: https://github.com/trynthink/scout/blob/select_ecms/scout/ecm_prep.py#L13921-L13924