This PR fixes a problem that occurs (inconsistently) within the get_load_sid function, specifically within the postprocessing of the flux query result. The problem propagates itself to other parts of openstef-dbc, because get_load_sid function is used by many other functions, e.g. get_load_pid and get_model_input.
The flux query, which this PR is about, returns two dataframes in a list. One of these dataframes contains the load and the other dataframe contains "the number of entries (nEntries)", which refers to the number of different system_id's for which the load is retrieved. The order of these dataframes within the returned list, is not always the same. The existing code was written under the assumption that the first dataframe in the list always is the dataframe with the load. However, sometimes the second dataframe contains the load.
The consequence of this bug was that nEntries, which is constant for all timestamps, was mistaken for the load. So, our models would sometimes receive a constant (flatliner) load to train on and make (fallback) forecasts on, while in actuality the load was not constant.
The new code recognizes dynamically which of the two output dataframes in the list contains the load.
This PR fixes a problem that occurs (inconsistently) within the
get_load_sidfunction, specifically within the postprocessing of the flux query result. The problem propagates itself to other parts of openstef-dbc, becauseget_load_sidfunction is used by many other functions, e.g.get_load_pidandget_model_input.The flux query, which this PR is about, returns two dataframes in a list. One of these dataframes contains the load and the other dataframe contains "the number of entries (
nEntries)", which refers to the number of different system_id's for which the load is retrieved. The order of these dataframes within the returned list, is not always the same. The existing code was written under the assumption that the first dataframe in the list always is the dataframe with the load. However, sometimes the second dataframe contains the load.The consequence of this bug was that
nEntries, which is constant for all timestamps, was mistaken for the load. So, our models would sometimes receive a constant (flatliner) load to train on and make (fallback) forecasts on, while in actuality the load was not constant.The new code recognizes dynamically which of the two output dataframes in the list contains the load.