Old timeframe switching logic for monthly-type meters (to try to get data to load):
1 Week > 1 Month > repeat
New timeframe switching logic for monthly-type meters:
1 Year > 1 Month > 2 Years > 1 Month > 1 Day > 1 Month > 1 Week > 1 Month > repeat
The timeframe logic for yearly-type meters should be the same as it was:
2 Year > 1 Year > repeat
Also fixed issue with meter menu opening logic (stop opening meter menu for the current meter, once current meter showed loading screen successfully)
Tried to do some comment cleanup (maybe save variable renaming and bigger refactors for another PR?)
TODOS
[ ] More testing of the script at various times of day to see if the changes make the scraper more robust to errors
[ ] Comment / logs cleanup
[ ] Any other issues or fixes that come up for the timeframe issue
For instance, there's probably a better data structure or control flow for the monthly list of timeframes that doesn't involve just repeating "1 month" 4 times but I just went with the first solution that worked
Not sure if increasing maxAttempts from 5 to 8 across the board is ideal (although it would improve reliability in general). It could be an option to make multiple "maxAttempts" variables for different events (e.g. max of 5 attempts for login retries vs 8 for timeframe retries)
Should the "Monthly Top Not Found" messages be tweaked / hidden in case of an intentional throw (throwing for odd timeframeIterator, not reading this value although it is valid)?
[ ] See current TODO comments in code as well (note "in progress" vs "future PR" keywords)
Testing
I included a log excerpt below since this error (top row of meter data aka monthly_top not detected) triggers somewhat randomly. For more robust testing, be sure to test this several times and various times of day.
Suggested: Use command node readPP.js --no-upload > logs/something.txt, check back later (Ctrl F Monthly Top not found, try again, Attempt in the text file specified)
Excerpt from my own logs below (May 21, 2024, ~6 pm).
39
Meter Menu Opened
New Meter Opened
Loading Screen Found
33986 HIGHWAY 34 BYPASS CORVALLIS OR (Item #241) (Meter #78831398 )
PP Meter ID: 78831398
Monthly Top not found.
One Week Option Found
One Year Found
One Year Clicked
Monthly Top not found, try again
Attempt 1 of 8
39
Meter Menu Opened
New Meter Opened
Loading Screen not found.
Loading Screen not found, trying again
39
33986 HIGHWAY 34 BYPASS CORVALLIS OR (Item #241) (Meter #78831398 )
PP Meter ID: 78831398
Monthly Top Found
throwing for odd timeframeiterator, not reading this value although valid
Monthly Top not found.
One Week Option Found
One Month Found
One Month Clicked
Monthly Top not found, try again
Attempt 2 of 8
39
33986 HIGHWAY 34 BYPASS CORVALLIS OR (Item #241) (Meter #78831398 )
PP Meter ID: 78831398
Monthly Top Found
Monthly Data Top Row Found, getting table top row value
Data is not yearly. Data is probably monthly.
===
Period2024-05-20Average Temperature52.5Usage(kwh)51.74Est. Rounded Amount$8
Issue
Changes
TODOS
maxAttemptsfrom 5 to 8 across the board is ideal (although it would improve reliability in general). It could be an option to make multiple "maxAttempts" variables for different events (e.g. max of 5 attempts for login retries vs 8 for timeframe retries)throwing for odd timeframeIterator, not reading this value although it is valid)?Testing
I included a log excerpt below since this error (top row of meter data aka
monthly_topnot detected) triggers somewhat randomly. For more robust testing, be sure to test this several times and various times of day.Suggested: Use command
node readPP.js --no-upload > logs/something.txt, check back later (Ctrl FMonthly Top not found, try again,Attemptin the text file specified)Excerpt from my own logs below (May 21, 2024, ~6 pm).