Road to 1.0

[AKuederle]

A meta tracking issue on things that we need before 1.0

Revalidation

• [ ] Find a way to publish new full pipeline validation results
• [ ] Decide if we want to publish old raw results

Algorithms

• [ ] Make decision on Re-Orientation algorithm (is it feasible to include in 1.0)
• [ ] Investigate differences between old and new results likely caused by the GSD algorithms -> decide if this warrant updating the algorithm
• [ ] The model training of L/R detection algorithm is not reproducible. Implementing reproducible training, will likely slightly change the result of the algorithm, as the model will update) #186

Other

• [x] Add proper citation for repo #146

[AKuederle]

Notes on "per-block" revalidation:

From Encarna's paper: Mean and 95% confidence intervals of all digital mobility outcomes were evaluated at a cohort level (i.e., quantified using all walking bouts across all participants belonging to that specific cohort)

[pltsc18]

Statistical plan reimplementation - TODO list

Here's on overview of the adopted TVS statistical plan, as reported in Micò-Amigo et al., 2023 and Kirk et al., 2024. I already ticked what is already implemented in MobGap to my experience (although probably needs to be properly integrated with validation code as we mentioned).

Block-by-block validation (Micò-Amigo et al., 2023)

Notes on the experimental protocol: 108 subjects from the TVS free-living dataset. Results are grouped by cohorts.

Gait sequence detection block

• [ ] 1. Segmentation: Segmentation into 0.1 windows and computation of True Positive (TP), False Positive (FP), False Negatives (FN) and True Negatives (TN) from overlapped detected and INDIP reference gait sequences.
• [x] 2. Gait sequence detection performance expressed as mean ± 95% confidence intervals (CI) for each cohort:
  • [x] Accuracy
  • [x] Sensitivity
  • [x] Positive Predictive Value (PPV)
  • [x] Specificity
• [x] 3. Gait sequence duration estimation performance for each cohort:
  • [x] INDIP reference mean and CI (s)
  • [x] Single device mean and CI (s)
  • [x] Bias and LoA (s)
  • [x] Absolute error mean and CI (s)
  • [x] ICC mean and CI
• [ ] 4. Performance index

  Initial contacts detection block (takes INDIP reference gait sequences as inputs):

• [x] 1. Matching: Selection of 0.5 tolerance windows centered in each INDIP reference initial contact and computation of TP, FN, FP.
• [x] 2. Initial contacts detection performance expressed as mean ± CI for each cohort:
  • [x] Sensitivity
  • [x] PPV
• [x] 3. Initial contacts temporal estimation for each cohort:
  • [x] Absolute error mean and CI (s)
  • [x] ICC mean and CI
• [ ] 4. Performance index

  Cadence block (takes INDIP reference gait sequences as inputs):

• [x] 1. Cadence estimation performance for each cohort:
  • [x] INDIP reference mean and CI (steps/min)
  • [x] Single device mean and CI (steps/min)
  • [x] Bias and LoA (steps/min)
  • [x] Absolute error mean and CI (steps/min)
  • [x] Relative error mean and CI (steps/min)
  • [x] ICC mean and CI
• [ ] 2. Performance index

  Stride length block (takes INDIP reference gait sequences and initial contacts as inputs):

• [x] 1. Stride length estimation performance for each cohort:
  • [x] INDIP reference mean and CI (m)
  • [x] Single device mean and CI (m)
  • [x] Bias and LoA (m)
  • [x] Absolute error mean and CI (m)
  • [x] Relative error mean and CI (m)
  • [x] ICC mean and CI
• [ ] 2. Performance index

End-to-end walking speed validation (Kirk et al., 2024)

Notes on the experimental protocol: 97 subjects from the TVS laboratory dataset (reference: stereophotogrammetry) and 82 subjects from the TVS free-living dataset (reference: INDIP). Results are grouped by cohorts.

True positive evaluation

This step was done for both the laboratory and the real-world datasets.

• [x] 1. Matching: a detected gait sequence is considered a match if overlaps at least 80% with an INDIP reference gait sequence.
• [x] 2. Walking speed estimation metrics calculated by comparing the average walking speed for each matched gait sequence against the one measured by the reference. For each cohort:
  • [x] Reference mean and CI (m/s)
  • [x] Single device mean and CI (m/s)
  • [x] Bias and LoA (m/s)
  • [x] Absolute error mean and CI (m/s)
  • [x] Relative error mean and CI (m/s)
  • [x] Absolute relative error mean and CI (m/s)
  • [x] ICC mean and CI

    Combined evaluation

    This step was done for both the laboratory and the real-world datasets.

• [x] 1. Aggregation: for a given type of test (e.g., laboratory TUG; 2.5. h free-living recording, etc...), all the detected gait sequences are aggregated.
• [x] 2. Walking speed estimation metrics calculated by comparing the median walking speed from all identified WBs within each laboratory task (resulting in one value per gait task per participant) or within the 2.5 h real-world assessment per participant (resulting in one value per participant) against the combined values of the reference system. For each cohort:
  • [x] Reference mean and CI (m/s)
  • [x] Single device mean and CI (m/s)
  • [x] Bias and LoA (m/s)
  • [x] Absolute error mean and CI (m/s)
  • [x] Relative error mean and CI (m/s)
  • [x] Absolute relative error mean and CI (m/s)
  • [x] ICC mean and CI