Matt-Brigida / EIAdata

R Wrapper for the Energy Information Administration (EIA) API
16 stars 15 forks source link

Speed improvements when converting to a data.frame #6

Closed sy2816 closed 6 years ago

sy2816 commented 6 years ago

I found that:

df <- data.frame(
    date = sapply(doc["//data/row/date"], xmlValue),
    value = sapply(doc["//data/row/value"], xmlValue)
)

is a much faster way to turn the data into a data.frame than the current method. On my machines. performance of the getEIA() function roughly doubles doing it this way for large data sets such as hourly and daily data. The change on the quarterly or annual data is negligible since so little processing time is spent on converting it to a data.frame in the first place.

Matt-Brigida commented 6 years ago

Thanks for the pull request. I ran some tests, and confirmed your results:

###testing base version----
microbenchmark(getEIA('EBA.CAL-ALL.DF.H', key = key), times = 5)
## Unit: seconds
##                                   expr      min       lq     mean   median
##  getEIA("EBA.CAL-ALL.DF.H", key = key) 11.17112 11.73119 12.01121 12.32057
##        uq      max neval
##  12.34497 12.48821     5

microbenchmark(getEIA('PET.RCLC1.D', key = key), times = 5)
## Unit: seconds
##                              expr      min       lq     mean  median       uq
##  getEIA("PET.RCLC1.D", key = key) 4.243039 4.281874 4.406932 4.42847 4.469302
##       max neval
##  4.611974     5

## testing sy2816 version----
microbenchmark(getEIA('EBA.CAL-ALL.DF.H', key = key), times = 5)
## Unit: seconds
##                                   expr      min       lq     mean   median
##  getEIA("EBA.CAL-ALL.DF.H", key = key) 6.287817 6.356803 6.499777 6.456629
##        uq      max neval
##  6.543836 6.853801     5

microbenchmark(getEIA('PET.RCLC1.D', key = key), times = 5)
## Unit: seconds
##                              expr      min       lq     mean   median       uq
##  getEIA("PET.RCLC1.D", key = key) 2.566363 2.578678 2.658892 2.614786 2.760034
##     max neval
##  2.7746     5

I'll run a few more tests, and likely merge your changes into master in a day or so.

Matt-Brigida commented 6 years ago

Looks good, thanks again.