Scaling profiling datasets

[davidorme]

Description

This PR restructures the profiling code to do a number of things:

• Move the data payload fixtures into a profiling/conftest.py
• Separate the pmodel and subdaily profile workflows into separate files
• Add custom command line options to pytest to allow the scaling of the payload data to be controlled from the command line. The subdaily and pmodel tests share the same data and scaling factor option.
• Change the default scaling to something that ought not to break the GitHub runners (max of 16GB RAM).

These changes make the profiling more flexible and helps set up for looking at the scaling of functions with input sizes. It allows commands like the following:

# Look at the peak memory usage,  run all the profiling tests, using an 8 fold scaling of the inputs
/usr/bin/time -l pytest -m "profiling" --pmodel-profile-scaleup 8
# Look at the peak memory usage,  running the splash profile, with an 100 fold scaling of the inputs
/usr/bin/time -l pytest -m "profiling" tests/profiling/test_profiling_splash.py --splash-profile-scaleup 100

The peak memory size of running the full test set is basically the combined size of each test and is very roughly:

50Mb splash-profile-scaleup + 1Gb pmodel-profile-scaleup

The defaults --splash-profile-scaleup=125 and --pmodel-profile-scaleup=6 should give about a 6GB size for each dataset. If we could use a single dataset, we'd be able to use bigger loads....

Fixes #215

Type of change

• [x] New feature (non-breaking change which adds functionality)
• [ ] Optimization (back-end change that speeds up the code)
• [x] Bug fix (non-breaking change which fixes an issue)

Key checklist

• [x] Make sure you've run the pre-commit checks: $ pre-commit run -a
• [x] All tests pass: $ poetry run pytest

Further checks

• [x] Code is commented, particularly in hard-to-understand areas
• [ ] Tests added that prove fix is effective or that feature works

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.23%. Comparing base (5b50cb4) to head (77c7fae).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## develop #216 +/- ## ======================================== Coverage 95.23% 95.23% ======================================== Files 28 28 Lines 1701 1701 ======================================== Hits 1620 1620 Misses 81 81 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[davidorme]

@tztsai - Thanks for the comments. I've looked into them and think they're all 'resolved' for this PR. If you're happy, can you approve the PR?

[tztsai]

LGTM