Generate metrics from external regressors using F stats

[handwerkerd]

Closes #1009. This is an alternative approach to #1008 and #1021.

If a user provides external regressors, this will calculate a fit to those regressors to include as metrics in the component table and for use in decision trees.

Changes proposed in this pull request:

• Create a new module, metrics.external, for calculating metrics from external regressors
  • An F statistic is used with a Legendre polynomial detrending baseline and p, F, and R2 values are saved as metrics in the component table
  • These stats are saved for the Full F model using all regressors as well as partial F models. This makes it possible to both examine when the full model fits, but to also log if it's the motion vs ROI-based vs cardiac/respiration-based regressors that fit to a specific component time series.
  • There is a special class of regressors called task_keep If regressors under this label are included, these will be excluded from the full F model and a separate full F model will be calculated using just these regressors. This was a suggestion from @dowdlelt so that it would be possible to identify and conservatively retain components that fit to the overall task design.
• Add a new parameter to the tedana CLI, --external, to pass in a TSV with external regressors.
• Demo decision trees that incorporate external regressor correlation metrics. There are called demo_ because they demonstrate how to use the new functionality. They might be similar to trees we'll want to recommend, but validation on actual data (and tweaking) will be easier when this is merged and more people are using it.
  • In the decision tree there is a new field external_regressor_config which is a dictionary with the following keys:
  • info A description of what the config does that's saved
  • report A description to add to report.txt
  • calc_stats Currently the only option is "F" but this leaves open possibilities for additional options.
  • detrend: Will automatically calculate the number of polynomial detrending regressors if true, but can also be a number for the users to specify. If this is false, then will be set to 0 (mean removal) and log a warning.
  • f_states_partial_models [optional] A list of titles for the partial models (i.e. ["Motion", "CSF"])
  • If f_states_partial_models has model names, each model name is its own key and contains either a list of column labels or a regular express wildcard (i.e. "Motion": ["^mot_.*$"] means the Motion partial model will include any external regressor column label that begins with mot_ and "Motion": ["mot_x", "mot_y", "mot_z"] specifies 3 specific column label names
  • task_keep [optional] Contents are a regex wildcard or specific names to define task regressors that will not be included in the full model
  • demo_minimal_external_regressors_motion_task_models.json uses all the above options, uses the partial models to add a classification_tag, but not change results and retains components that correlate to the task (R2>0.5), have kappa>elbow irregardless of what rho is.
  • demo_minimal_external_regressors_single_model.json uses the minimum number of options to run with external regressors.
• Includes tests that should cover all new code.
  • In adding testing, I started to use the matching parameter to confirm the correct errors are triggered and caplog.text to confirm expected warnings or info statements are logged.
  • Also starting to add tests that check for specific numerical outputs.

To do:

• [ ] Update documentation to explain all this new functionality and outputs (waiting on this for enough of us to agree that the approach is generally good and we're not making more substantive changes to the structure/parameters/etc

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 88.05970% with 32 lines in your changes are missing coverage. Please review.

Project coverage is 89.61%. Comparing base (12aea7f) to head (4223bf9). Report is 1 commits behind head on main.

Files	Patch %	Lines
tedana/metrics/external.py	91.36%	7 Missing and 5 partials :warning:
tedana/selection/component_selector.py	83.01%	4 Missing and 5 partials :warning:
tedana/selection/selection_utils.py	85.71%	2 Missing and 1 partial :warning:
tedana/workflows/tedana.py	70.00%	2 Missing and 1 partial :warning:
tedana/metrics/collect.py	86.66%	1 Missing and 1 partial :warning:
tedana/stats.py	80.00%	1 Missing and 1 partial :warning:
tedana/metrics/_utils.py	94.44%	0 Missing and 1 partial :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1064 +/- ## ========================================== - Coverage 89.91% 89.61% -0.31% ========================================== Files 26 27 +1 Lines 3621 3853 +232 Branches 629 686 +57 ========================================== + Hits 3256 3453 +197 - Misses 214 233 +19 - Partials 151 167 +16 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[handwerkerd]

I'd appreciate some reviewers for this: @tsalo @eurunuela @goodalse2019 @n-reddy @dowdlelt . From my end, the only thing left to do is update the ReadTheDocs part of the documentation. I'm holding off on that until a couple of you agree that the overall structure makes sense.

[handwerkerd]

@tsalo I just added type hints to external.py I'm still a bit of a novice with type hints and it looks like formatting is non-trivially changing across python versions. I tried to follow guidance that's valid for python 3.8 since that's our oldest permitted version. I'll keep working in other files and on your remaining review comments, but let me know if I'm appropriately adding type hints so far.

Update as I was writing this comment: Lots of tests just failed so the answer to that question is"no". It looks numpy v1.24.3 that's currently in my python 3.8 environment includes numpy.typing and not numpy._typing while numpy v1.26.3 in my python 3.11 environment includes numpy._typing and not numpy.typing. I suspect there's a way to set up backwards compatibility for this, but I wanted to share what I'm seeing before I switch to childcare stuff. I figured out the issue and all tests are passing now. It seems like using import numpy.typing as npt works, but directly referencing np.typing does not work across all versions

[handwerkerd]

@tsalo Can you explain more what you mean by "It would be great if strings and lines in docstrings were broken on punctuation. This will result in cleaner diffs in the future."

[tsalo]

Absolutely. Take a paragraph in the text, like the one below:

This is sentence one. This is
sentence two. This is sentence
three- but it's still going.

Since sentences are a unit within the paragraph, we're more likely to change them than random words. Plus, since almost everything we write is in markdown or rst, the newlines don't impact the rendered text- just the code.

-This is sentence one. This is
+This is sentence one. This is a 
-sentence two. This is sentence
+new sentence two with other
-three- but it's still going.
+changes. This is sentence three-
+but it's still going.

The diff on that, where the only consideration is line length, is much more extensive (and harder to interpret for a reviewer) than the diff if we broke up the text on punctuation:

 This is sentence one.
-This is sentence two.
+This is a new sentence 
+two with other changes.
 This is sentence three-
 but it's still going.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 92.30769% with 21 lines in your changes missing coverage. Please review.

Project coverage is 89.90%. Comparing base (7ca9c27) to head (8d09599).

:exclamation: Current head 8d09599 differs from pull request most recent head 2d45383

Please upload reports for the commit 2d45383 to get more accurate results.

Files	Patch %	Lines
tedana/selection/component_selector.py	83.33%	4 Missing and 5 partials :warning:
tedana/selection/selection_utils.py	85.71%	2 Missing and 1 partial :warning:
tedana/workflows/tedana.py	70.00%	2 Missing and 1 partial :warning:
tedana/metrics/collect.py	86.66%	1 Missing and 1 partial :warning:
tedana/stats.py	80.00%	1 Missing and 1 partial :warning:
tedana/metrics/_utils.py	94.44%	0 Missing and 1 partial :warning:
tedana/metrics/external.py	99.24%	0 Missing and 1 partial :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1064 +/- ## ========================================== - Coverage 89.91% 89.90% -0.02% ========================================== Files 26 27 +1 Lines 3620 3854 +234 Branches 629 685 +56 ========================================== + Hits 3255 3465 +210 - Misses 214 226 +12 - Partials 151 163 +12 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[handwerkerd]

I think I've now addressed all feedback regarding code and I've started to add documentation to building_decision_trees.rst I still need to make flow charts for the new trees and add them to included_decision_trees.rst but otherwise this PR is (again) ready for review & for people to try it out.