Many AquaCache functions are designed to be run on a regular basic (hourly, daily, or weekly) to import and process data. Most of these can benefit from speed and memory use improvements, possibly using data.table or even by passing C++ code using Rcpp.
The package is made data.table aware in function dbGetQueryDT (.databale.aware <- TRUE), plus import calls in the NAMESPACE for data.table operators.
What's been done already:
dbGetQueryDT exists to return a data.table object from a DBI::dbGetQuery operation.
calculate_stats uses a bit of data.table where the function must operate row-by-row. It may be possible to improve performance elsewhere in the function.
downloadWSC fetches data using data.table::fread and returns a data.table.
Priority functions to improve:
synchronize
calculate_stats
In addition, function downloadWSC relies on weathercan::weather_dl to import data and this is quite slow. There's a lot of overhead in weather_dl that isn't necessary.
Many AquaCache functions are designed to be run on a regular basic (hourly, daily, or weekly) to import and process data. Most of these can benefit from speed and memory use improvements, possibly using data.table or even by passing C++ code using Rcpp.
The package is made data.table aware in function dbGetQueryDT (.databale.aware <- TRUE), plus import calls in the NAMESPACE for data.table operators.
What's been done already:
Priority functions to improve:
In addition, function downloadWSC relies on weathercan::weather_dl to import data and this is quite slow. There's a lot of overhead in weather_dl that isn't necessary.