ropensci / rb3

A bunch of downloaders and parsers for data delivered from B3
https://docs.ropensci.org/rb3/
Other
71 stars 28 forks source link

Improve the way data is cached #37

Closed wilsonfreitas closed 2 years ago

wilsonfreitas commented 2 years ago

The cached data used the template name and a hash that is built based on the arguments used to generate it. Further, data is always stored in rds format.

One interesting alternative is the use of parquet files to store data. With that duckdb could be used to query data on these files and the cache would form a database.

To get this done, and easy to debug and follow, a name formation rule should be interesting for cached files.

A naming function should be created to name cached data.

This naming function would be declared in the template file, the same way is done with downloader and reader.