Prior to this PR, compute_interval_params was written to determine the absolute set of intervals to be processed. This meant taking into account depends on past limitations and signals. In addition it would then batch these intervals based on the model's batch size.
This PR changes this function's behavior to instead return a superset of what may actually need to run just based on the intervals that are missing. When actually running the final set of intervals are calculated based on depends on past, signals, batch size, ect. Therefore what is actually run will be a subset of what is provided.
The overall functionality of SQLMesh is unchanged in this PR just this internal detail has been changed.
Two key other changes in the PR:
Prior to this PR we had Intervals defined as tuple of datetimes in some places and tuples of ints in other places. Now Intervals is consistently ints and the functions consistently exchange ints. I couldn't see a good reason why this different existed.
Airflow was "insulated" from these changes by making of copy of existing functionality into the Airflow codebase and then also moving over the datetime tuple to there too.
Prior to this PR,
compute_interval_paramswas written to determine the absolute set of intervals to be processed. This meant taking into account depends on past limitations and signals. In addition it would then batch these intervals based on the model's batch size.This PR changes this function's behavior to instead return a superset of what may actually need to run just based on the intervals that are missing. When actually running the final set of intervals are calculated based on depends on past, signals, batch size, ect. Therefore what is actually run will be a subset of what is provided.
The overall functionality of SQLMesh is unchanged in this PR just this internal detail has been changed.
Two key other changes in the PR:
Intervalsdefined as tuple of datetimes in some places and tuples of ints in other places. NowIntervalsis consistently ints and the functions consistently exchange ints. I couldn't see a good reason why this different existed.