search

judgelord / regulationsdotgov

a package to get data from regulations.gov
https://judgelord.github.io/regulationsdotgov/
MIT License
0 stars 0 forks source link

avoiding large objects in memory #5

Closed judgelord closed 2 weeks ago

judgelord commented 3 months ago

We need to be careful not to create objects much larger than the output inside functions.

For example, the raw results for 1000 calls for comment details are about 15 MB, whereas the data frame we return for 12,000 comments is only .2 MB.

Thus, loops should be designed to hold the final result and overwrite larger objects that are no longer needed rather than making large lists and then extracting data from them.

judgelord commented 1 month ago

I think we are good on this front, with no more than 20 raw results stored in memory at a time, correct?