rstudio / pins-python

https://rstudio.github.io/pins-python/
MIT License
52 stars 12 forks source link

Creating a local cached copy for a remote dataset? #184

Closed henningsway closed 1 year ago

henningsway commented 1 year ago

I have used pins before and was wondering, if there was a way to use this to maintain a remote set of datasets, that can be accessed by the whole team, while at the same time avoiding having to download the dataset every time I run or develop a script.

To me a local_cache=True Options would seems desirable, which would then download a local copy of the dataset, so that network connection is no longer required and a bigger dataset wouldn't have to be downloaded.

Is see, that it's difficult to implement (you would have to "manage the local file", e.g. set a path, check if it's still up to date) and that redownloading the file each time seems straightforward.

Would be very happy to hear your thoughts on this, though. :)

henningsway commented 1 year ago

Nevermind: https://rstudio.github.io/pins-python/getting_started.html#caching ;-)

juliasilge commented 1 year ago

Let us know if you have further questions or run into problems with the caching @henningsway! 🙌