A `raw` or `path` pin_write api

[slodge-work]

We've got a use case at present where we have a large number of files generated by a C# job which we'd like to upload to Connect pins.

These files are csv and parquet and we are seeing/thinking about:

• some processing overhead in this operation - e.g. we do some unnecessary data.table::fread csv parsing and then pins-r does some unnecessary utils::write.csv writing... (note that we use fread because it is significantly faster)
• some small data and format changes - because of small differences between fread and write.csv and because of numeric and NA representations of the csv content.

To avoid these, we're currently looking at changing our R code to use the "lower-level" API of pin_store .

However, I just thought I'd also ask: is there a "user-level" API that could also be considered? - i.e. could there be a variation on pin_write that takes a path instead of x? Or, if not, is there any way pins could expose options/overrides for its object_write handling? https://github.com/rstudio/pins-r/blob/3f10fdb590e8c190571e2f801bc8f2ba6fd4e174/R/pin-read-write.R#L154C1-L169C2

Not entirely sure how common a use case this is going to be for us or anyone else in the future - but thought I'd ask anyways.

[slodge-work]

Update.... now looking at pin_upload from https://pins.rstudio.com/articles/managing-custom-formats.html 👍

We're looking at starting with that - but maybe modifying it so that we can use types other than file in it

[juliasilge]

I would definitely recommend pin_upload() / pin_download() for a user-level API that lets you deal with paths and specific file formats. And that specific vignette you linked to shows you how to create custom functions (perhaps that you put in an internal R package) that can store and read files specifically how you need to, with any needed controls or customization. 👍

[slodge]

Thanks

I think we will take those functions and modify them slightly - we want to override this type for the files: https://github.com/rstudio/pins-r/blob/3f10fdb590e8c190571e2f801bc8f2ba6fd4e174/R/pin-upload-download.R#L61

[juliasilge]

You probably have already seen this, but if you want to store custom metadata including more specific "what kind of file is this?" than just "file", this is the approach we recommend: https://pins.rstudio.com/articles/customize-pins-metadata.html

It would get set in a separate, user slot but that would still be available when downloading, still get surfaced in a Connect preview page, etc.

[juliasilge]

Let us know if you have further questions! 🙌

[github-actions[bot]]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.