Context-Adaptive Statistical Inference: Recent Progress, Open Problems, and Opportunities for Foundation Models

This is an open, collaborative review paper on context-adaptive statistical methods. We look at recent progress, identify open problems, and find practical opportunities for applying these methods. We are particularly excited by the opportunities for foundation models to provide context for statistical inference.

This manuscript is created automatically from the content in content using Manubot. Please contribute! Make a PR or file an issue, and see below for more information about Manubot. Live update versions of the manuscript are available at:

• HTML manuscript at https://LengerichLab.github.io/context-review/
• PDF manuscript at https://LengerichLab.github.io/context-review/manuscript.pdf

Why are we writing this?

As statistical modeling evolves, we are witnessing two complementary approaches to integrating context. Traditional statistical models are being expanded to allow explicit parameter adjustments based on context, making their adaptations transparent and interpretable. Meanwhile, large foundation models are being built that how to implicitly adapt to context, enabling impressive performance in a wide range of tasks including in-context learning. This review seeks to unite these two perspectives, combining the explicit adaptability of statistical models with the powerful, implicit adjustments of foundation models. By bringing these approaches together, we aim to provide a comprehensive overview of current progress, challenges, and opportunities in context-adaptive inference.

Key perspectives driving this review:

• Complex models contain multitudes of simpler, context-specific models – Every complex model can be understood as a combination of many smaller models, each tailored to specific contexts.
• Explicit vs. implicit adaptation – Statistical models explicitly adjust parameters based on context, while foundation models implicitly adapt to context. Combining these approaches offers new opportunities for robust and adaptive inference.
• Context reshapes inference – Context-awareness enhances both statistical and foundation models, improving personalization and accuracy in predictions.
• In-context learning as a model of implicit adaptation – Foundation models excel at tasks like in-context learning, showing how implicit adaptation can inform broader context-adaptive modeling efforts.
• Foundation models as context providers – These models offer flexible, scalable ways to incorporate context, enhancing traditional methods with richer context integration.
• Challenging traditional assumptions – Context-adaptive methods move beyond the assumption of homogeneity in data, enabling models to handle heterogeneous datasets more effectively.
• Personalization through adaptation – Uniting explicit and implicit context-adaptive models provides a path to more nuanced, personalized predictions that reflect the complexities of real-world data.

Table of Contents

1. Abstract
2. Introduction and Definitions
3. Theoretical Foundations and Advances in Varying-Coefficient Models
4. Context-Adaptive Interpretations of Context-Invariant Functions
5. Opportunities for Foundation Models
6. Applications, Case Studies, and Evaluations
7. Technological and Software Tools
8. Future Trends
9. Open Problems
10. Conclusions

How can you contribute?

We welcome contributions from the community. Please see our contribution guidelines for more information.