MinIO S3 storage

[tomasjanikds]

Would it be possible to implement connection to MinIO S3? The same way as the board_s3 function works, but also for MinIO SR instead for AWS only. I am using Dell ECS S3 storage (AWS competitor) and I can connect via MinIO. It would be awesome if I could use a board there.

[juliasilge]

I see that there is a minio.s3 R package that is on GitHub but not on CRAN, which could possibly be used as a backend if it made it to CRAN.

I also see that the Arrow C++ library has a filesystem interface for S3 that can be used with, for example, MinIO. It might be a good option to create a new board for S3-like filesystems using this, or change the existing board_s3() to work more generically with all filesystems that emulate S3. I haven't thought super deeply about this one yet (could we still write CSV, binary RDS, etc?).

[tomasjanikds]

It turns out that board_s3() is quite generic after all.

I did install paws package and with some additional arguments it works:

library(pins)
board <- board_s3(bucket = "<minio_bucket>",
                  endpoint = "<minio_endpoint>",
                  access_key = "<access_key>",
                  secret_access_key = "<secret_key>",
                  region = "us-east-1")

endpoint argument takes a full url (including https:// prefix) of MinIO endpoint. Also, when using MinIO, region argument needs to be us-east-1.

Now I can use full potential of pins with MinIO storage. Thanks a lot for this package.

[github-actions[bot]]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.