Currently, microbatch models get a "current_time" (datetime.datetime.now(pytz.UTC)) when they are executed. Notably, each microbatch model gets a different "current_time". This works, but has some funkiness.
Consider the following:
There is a source, source_1, which is constantly being updated by an external process
Microbatch model model_a pulls from source_1
Microbatch model model_b pulls from source_1
Regardless if we're in a small or large project, given any delay in the execution of model_a and model_b, if there is any new data in source_1, the result of model_a will be different from model_b. An example would be:
source_1 has 3 rows with event times: 2024-10-07 12:17:00, 2024-10-07 12:16:00, 2024-10-07 12:15:00
Currently, microbatch models get a "current_time" (
datetime.datetime.now(pytz.UTC)) when they are executed. Notably, each microbatch model gets a different "current_time". This works, but has some funkiness.Consider the following:
source_1, which is constantly being updated by an external processmodel_apulls fromsource_1model_bpulls fromsource_1Regardless if we're in a small or large project, given any delay in the execution of
model_aandmodel_b, if there is any new data insource_1, the result ofmodel_awill be different frommodel_b. An example would be:source_1has 3 rows with event times:2024-10-07 12:17:00,2024-10-07 12:16:00,2024-10-07 12:15:002024-10-07 12:18:00model_ais executed, picking up the 3 rowssource_1:2024-10-07 12:18:302024-10-07 12:19:00model_bis executed, picking up 4 rows.Is the discrepancy okay?