Profiler

[aspeake]

Adds an optional profiler to the run_workflow.py script to quantify CPU time and peak memory. This PR also uses this new feature as part of the CI build integration testing workflow, while making some edits to .github/workflows/tests.yml.

New features to run_workflow.py:

• Ability to run workflow steps with the cProfile library. Use the flag --with_profiler
• Ability to select a single step (ecm_prep or run) to run. Use the argument --step

New behavior on CI:

• If there is a PR with base branch=master, then integration tests will be run w/o profiler (same as before). Results will be uploaded as artifacts and, if there are changes, also committed to the repo.
• If there is a push to master, then integration tests will be run with profilers. ecm_prep.py and run.py will be run as two separate commands so that distinct profile metrics can be output for each. Two files, profile_ecm_prep.csv and profile_run.csv, will be saved as artifacts to report the function times and peak memories. No files will be committed from master
• Artifacts produced by CI on master:
• Artifacts produced by CI with PR:

Bugfixes on CI:

• Only run integration tests if the commit is associated with an active, non-draft PR, or there is a push to master. Previously, the "Check-PR" step would return true, even after a PR was merged, so we were getting integration tests run on version releases and branches off of master.
• Add a pull_request event trigger to run when PRs are opened or changed from draft, previously, CI would always require a push

[aspeake]

To-Do:

• [x] Breakout running psrecord on ecm_prep.py and run.py separately
• [x] Output a memory_log.txt for ecm_prep and run
• [x] Append peak memory to respective profile_*.csv files
• [ ] If not possible, write memory log to separate file
• [x] Revert committing the profile stats csvs, store them as artifacts and only run profilers on master (ie do not commit anything via master)

[trynthink]

Should we prevent the profiler from running on commits that have changes only in directories or files that do not have an impact on runtime (e.g., .github/*, docs/*, .gitignore)?

[aspeake]

Should we prevent the profiler from running on commits that have changes only in directories or files that do not have an impact on runtime (e.g., .github/*, docs/*, .gitignore)?

There are a couple of ways to do this, the best way I think would be to add to the if statement here: https://github.com/trynthink/scout/blob/profiler/.github/workflows/tests.yml#L72 to check for file changes. However, I think it would be smart to require a passing integration_tests run before merging, which would require us to at least start to run that workflow. I can add an if statement before every job in integration_tests.yml (workflow_call events don't have a built in filter for this). Its not the most flexible or clean approach but would probably work.