[openei_tariff_analyzer.py] Entry not found due to API query LIMIT of 500

The OpenEI Tariff Analyzer python script can be used as an interface to access the OpenEI database. https://github.com/LBNL-ETA/elecprice/blob/2b78ea809c7f3f95c7fbe115fe9ed03fba2dcde8/electricitycostcalculator/openei_tariff/openei_tariff_analyzer.py By providing arguments such as utility_id, sector and tariff_rate_of_interest, corresponding entries from the database can be queried.

Expected behaviour: Requesting tariff_rate_of_interest with the value Time of Use, General Service, Demand Metered, Option D: GS-2 TOU D, Three Phase (Under 2 kV) should return 8 entries, including the newest one with startdate=2020-06-01T08:00:00.000Z and label= 5ed6bcba5457a3a405dd15aa as found in https://apps.openei.org/USURDB/rate/view/5ed6bcba5457a3a405dd15aa.

Actual behaviour: The result JSON file contains only one entry with startdate=2015-02-28T09:00:00.000Z and label=5c9114ea5457a37e5307bc77.

Error cause: Due to client side filtering and no option to filter on server side, the crappy OpenEI API filters only by utility_id, sector, returning the first 500 entries. Then on the client side, the tariff_rate_of_interest is filtered from those 500 entries. Unfortunately, the actual tariff_rate_of_interest of interest is within the following 138 entries and thus is never received at the client.

Minimal example:

from electricitycostcalculator.electricity_rate_manager.rate_manager import ElectricityRateManager
from electricitycostcalculator.openei_tariff.openei_tariff_analyzer import *

tariff_openei_apidata = OpenEI_tariff(utility_id='17609',
                                      sector='Commercial',
                                      tariff_rate_of_interest='Time of Use, General Service, Demand Metered, Option D: GS-2 TOU D, Three Phase (Under 2 kV)',
                                      distrib_level_of_interest=None,
                                   # it is at the secondary level, so not specified in the name
                                      phasewing=None,
                                   # the word 'Poly' is to be excluded, because the names may omit this info ..
                                      tou=True,
                                      option_exclusion=['Poly'])  # Need to reject the option X and W

tariff_openei_apidata.call_api(store_as_json=False)
entry_count = len(tariff_openei_apidata.data_openei)
print(entry_count)
for entry in tariff_openei_apidata.data_openei:
    print(entry['label'])

Output:

1
5c9114ea5457a37e5307bc77

LBNL-ETA / elecprice

[openei_tariff_analyzer.py] Entry not found due to API query LIMIT of 500 #8