rstudio / pins-r

Pin, discover, and share resources
https://pins.rstudio.com
Other
312 stars 63 forks source link

Are some things too big to pin? #547

Closed sellorm closed 2 years ago

sellorm commented 2 years ago

There used to be a vignette with advice on appropriate data sizes for pins.

Could it, or something like it, be reinstated?

https://github.com/rstudio/pins/blob/8ff11887bf18b372f123454b70e2f6a7588c5745/vignettes/advanced-too-big.Rmd

It was a helpful rule of thumb for many when deciding what they should or should not pin.

jthomasmock commented 2 years ago

I wonder if this would be most appropriate under the Connect docs/Pro Tips? Tag @AshleyHenry15

There's a small note in the pins Pro Tips:

Pins are best suited for objects up to a few hundred megabytes in size.

hadley commented 2 years ago

I have the following in https://pins.rstudio.com/articles/pins.html:

Note that when the data lives elsewhere, pins takes care of downloading and caching so that it’s only re-downloaded when needed. That said, most boards transmit pins over HTTP, and this is going to be slow and possibly unreliable for very large pins. As a general rule of thumb, we don’t recommend using pins with files over 500 MB. If you find yourself routinely pinning data larger that this, you might need to reconsider your data engineering pipeline.

Not sure how to make it more likely for people to find it at the right time.

AshleyHenry15 commented 2 years ago

@jthomasmock - I will add the following note to the Pro Tip: 2021-11-05_10-30-41

hadley commented 2 years ago

Thanks @AshleyHenry15!

github-actions[bot] commented 2 years ago

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.