Multimodal Modeling Methods to Support

[Oufattole]

For methods to support for multimodal modeling we should support:

• [ ] No Fusion approach: Contrastive workflows between modalities where a single embedding is generated for each modality (and/or temporal window) and some contrastive loss is applied to align these modalities. This is inspired by this work. We currently support this for temporal windows, see the EBCL and OCP models.
• [ ] Early fusion approach: Our operating assumption is that clinical data is inherently multi-modal but temporally aligned (e.g. lab measurements, medication codes, notes all occur at specific timestamps). For sparse modalities like clinical notes that occur less frequently than dense measurements, we handle this efficiently by:
  1. Processing sparse modalities separately to avoid excessive padding
  2. Encoding sparse events (e.g. notes, ecgs, echos) in a separate batch
  3. Mapping the resulting embeddings back to their temporal positions in the patient sequence
  4. Combining these with the dense measurement embeddings at matching timestamps

This approach maintains temporal alignment while avoiding memory issues from padding sparse modalities to match dense measurement frequencies. See the JNRT padding example #6 for why naive padding is problematic.

[Oufattole]

I think we can start with creating a new input_encoder class: https://github.com/Oufattole/meds-torch/blob/main/src/meds_torch/input_encoder/triplet_encoder.py, that adds a new modality specific key to batch in the forward pass.

and then making a custom multimodal supervised method: https://github.com/Oufattole/meds-torch/blob/main/src/meds_torch/models/supervised_model.py