It turns out that the most_recent_beliefs_only and most_recent_events_only subqueries in timely-beliefs are missing the optional filters that are also applied to the main query.
Unfortunately, I also found an incompatibility. Two of those filters only roughly filter within the query, and only properly filter on the query results: namely beliefs_after and beliefs_before. They do so because they require taking into account the sensor's knowledge horizon function, which are python functions and not suitable for injection into the SQL query. The SQL filters essentially introduce some slack around the restricted range of belief times, with the size of this slack equal to the bounds of the specific knowledge horizon function. For example, the knowledge horizon for events of a day-ahead market with a gate closure at noon is normally bounded between 12 and 36 hours (to account for possibly double daylight savings time transitions this is even widened to between 10 and 38 hours). In any case, this messes up selecting the minimum horizon in the subquery, especially for economic sensors with non-trivial knowledge horizons (physical sensors are more straightforward and typically lead to no slack being introduced).
[x] Add the optional filter from the main query to both subqueries.
[x] Avoid using the most_recent_beliefs_only subquery in the special case of querying with the most_recent_beliefs_only and beliefs_after (and/or beliefs_before) filters on sensors with unsuitable knowledge horizon functions, and use belief_utils.select_most_recent_belief on the query results instead. Resolved in #117.
It turns out that the
most_recent_beliefs_onlyandmost_recent_events_onlysubqueries in timely-beliefs are missing the optional filters that are also applied to the main query.Unfortunately, I also found an incompatibility. Two of those filters only roughly filter within the query, and only properly filter on the query results: namely
beliefs_afterandbeliefs_before. They do so because they require taking into account the sensor's knowledge horizon function, which are python functions and not suitable for injection into the SQL query. The SQL filters essentially introduce some slack around the restricted range of belief times, with the size of this slack equal to the bounds of the specific knowledge horizon function. For example, the knowledge horizon for events of a day-ahead market with a gate closure at noon is normally bounded between 12 and 36 hours (to account for possibly double daylight savings time transitions this is even widened to between 10 and 38 hours). In any case, this messes up selecting the minimum horizon in the subquery, especially for economic sensors with non-trivial knowledge horizons (physical sensors are more straightforward and typically lead to no slack being introduced)._Originally posted by @Flix6x in https://github.com/FlexMeasures/flexmeasures/pull/312#discussion_r784872448_
My suggested approach to resolve this is:
most_recent_beliefs_onlysubquery in the special case of querying with themost_recent_beliefs_onlyandbeliefs_after(and/orbeliefs_before) filters on sensors with unsuitable knowledge horizon functions, and usebelief_utils.select_most_recent_beliefon the query results instead. Resolved in #117.