ropensci / tradestatistics

R package to access Open Trade Statistics API
https://docs.ropensci.org/tradestatistics
Apache License 2.0
76 stars 14 forks source link

using arrow for larger tables #49

Closed pachadotdev closed 2 years ago

pachadotdev commented 2 years ago

I made some changes in the API, which offer a 50% time reduction by reading parquet instead of JSON from the API

library(microbenchmark)

microbenchmark(
  ots_create_tidy_data(table = "yrpc-parquet", reporters = "can"),
  ots_create_tidy_data(table = "yrpc", reporters = "can"),
  times = 10L
)

Unit: seconds
                                                            expr      min       lq     mean   median       uq      max neval
 ots_create_tidy_data(table = "yrpc-parquet", reporters = "can") 1.390202 1.894805 2.006423 1.989092 2.309964 2.427866    10
         ots_create_tidy_data(table = "yrpc", reporters = "can") 3.548397 3.750299 3.891013 3.889045 3.950003 4.364702    10

I already sent the PR to allow Parquet serialization in the plumber package, api.tradestatistics.io is now using plumber development version