search

LBNL-ETA / elecprice

This project aims to provide a tool for calculating a customer's electricity bill based on a specific utility tariff. An example is given for various California utility tariffs, retrieved from the OpenEI API.
Other
2 stars 1 forks source link

Access the API without the LIMIT of 500 entries #9

Closed mquintus closed 2 years ago

mquintus commented 2 years ago

The API of the OpenEI database only let's queries request 500 entries, but filtering for e.g. tariff names happens on the client side. This leads to missed results, as mentioned in #8 and fixed in this pull request. How does it work? In this pull request, the number (count) of returned ENTRIES of the API QUERY is counted. If the COUNT equals the LIMIT, then the API will be queried a second time with an additional OFFSET (which equals the LIMIT) to receive the rest of the entries.

rebrownlbnl commented 2 years ago

This solution sounds fine in principle, but I can't really vouch for the actual code. Hoping @smurthy0211 can do that and merge the PR.

smurthy0211 commented 2 years ago

The code looks fine. Will pull down the PR, test it and confirm. @mquintus - could you share a sample tariff that has over 500 entries?

mquintus commented 2 years ago

Hi @smurthy0211 thank you, yes the minimal example is in https://github.com/LBNL-ETA/elecprice/issues/8 The problem with

OpenEI_tariff(utility_id='17609',
    sector='Commercial',
    tariff_rate_of_interest='GS-2 TOU D'

is that utility_id='17609', sector='Commercial' is filtered server side and yields 623 entries but only 500 are submitted to the client.

old behavior drawio

My proposed fix is to repeat the query whenever 500 results are returned, because obviously the LIMIT has been hit and the actual tariff of interest might not have been fetched yet.

new-behavor drawio

smurthy0211 commented 2 years ago

I tried to merge this to the main branch but was unable to do that because it said I don't have write access. @rebrownlbnl - I have tested this code and it works correctly. Could you try to merge this?

rebrownlbnl commented 2 years ago

@anand thanks for handling this

Rich Brown Building Technology and Urban Systems Division Lawrence Berkeley National Laboratory Office: (510) 486-5896 Mobile: (510) 502-5968 @.*** www.lbl.gov/~rebrown

On Mon, Feb 28, 2022 at 3:13 PM Anand Prakash @.***> wrote:

Merged #9 https://github.com/LBNL-ETA/elecprice/pull/9 into main.

— Reply to this email directly, view it on GitHub https://github.com/LBNL-ETA/elecprice/pull/9#event-6158114695, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLORHY3M3IFZ5EJ4FCS3WLU5P6RPANCNFSM5OY5XVSA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>