Big Data Processing Tools

microsoft / finnts

Microsoft Finance Time Series Forecasting Framework (FinnTS) is a forecasting package that utilizes cutting-edge time series forecasting and parallelization on the cloud to produce accurate forecasts for financial data.

https://microsoft.github.io/finnts

Other

190 stars 34 forks source link

Big Data Processing Tools #84

Closed mitokic closed 1 year ago

mitokic commented 3 years ago

Investigate how to leverage tools like dtplyr , arrow, and sparklyr packages to do larger data manipulation within the package.

Could create standard functions for mutate, select, group by, summarize that call different packages based on the hist data size.

0-1M rows: dplyr
1M-50M rows: dtplyr
100m+ rows: sparklyr

https://www.business-science.io/code-tools/2019/08/15/big-data-dtplyr.html https://arrow.apache.org/docs/r/articles/dataset.html https://spark.rstudio.com/