rstudio / pins-python

https://rstudio.github.io/pins-python/
MIT License
49 stars 12 forks source link

feat: support writing multiple dataframes/objects to the same pin #311

Open dbkegley opened 2 weeks ago

dbkegley commented 2 weeks ago

I spoke to some pins users at posit::conf who are interested in the ability to read/write multiple dataframes to the same pin. The primary use-case for this is when using board_connect. The ACL controls imposed by Connect mean that if a user wants to store >1 related dataframes on Connect then they must use multiple pins. This is cumbersome because they must also maintain ACL's for multiple content items. My recommendation for now is to use groups in Connect and update group membership but YMMV depending on the configured Auth provider in Connect.

I'm not that familiar with how pins stores data but my guess is that some of this is already possible when using the json storage type for Python or the json/rds types for R but the user would need combine their dataframes first.

It would be nice if pins supported APIs for writing multiple dataframes/objects to the same pin. I'm envisioning something like this:

board = pins.board_connect()
board.pin_write({"sales": tidy_sales_data, "other": my_other_dataframe}, "dbkegley/sales-summary", type="parquet")

dfs = board.pin_read("dbkegley/sales-summary")
sales_data = dfs['sales']
other_data = dfs['other']

This would store 2 separate parquet files (or CSVs) under the hood, one for each dataframe.

isabelizimm commented 2 weeks ago

Thank you for the report! I know something like this is available on the R side using board.pin_upload() to upload multiple files. Right now, pins for Python has the ability to upload 1 file at a time, but not multiple. I think a reasonable first step would be to implement pin_upload() for multiple files; that would enable people to do something like: board.pin_write([tidy_sales_data.to_parquet(), other.to_parquet()]).

Does that match what you would expect in this scenario, at least partially?

dbkegley commented 1 week ago

I think that would work! Thanks @isabelizimm