Move test data to an S3 bucket in the Hubverse AWS account

hubverse-org / hubData

Tools for accessing and working with hubverse Hub data

https://hubverse-org.github.io/hubData/

Other

3 stars 4 forks source link

Move test data to an S3 bucket in the Hubverse AWS account #54

Open bsweger opened 3 weeks ago

bsweger commented 3 weeks ago

The hubData test suite uses small cloud-based hubs to test S3 integration:

➜ aws s3 ls hubverse/hubutils/testhubs/ --no-sign-request
                           PRE empty/
                           PRE parquet/
                           PRE simple/

Now that the Hubverse has a dedicated S3 account, we should consider moving the above objects to it and modifying the packages that use these files for testing (to ensure broader access).

annakrystalli commented 2 weeks ago

Sure! Happy to help with this.

It would be nice if I could have access to the S3 account so I can make modifications when developing without needing assistance from others.

bsweger commented 2 weeks ago

Sure! Happy to help with this.

It would be nice if I could have access to the S3 account so I can make modifications when developing without needing assistance from others.

Of course! I propose an S3 bucket called something like hubverse-test-data that hubverse devs have read/write access to. Will ping the team to weigh in.

nickreich commented 2 weeks ago

No concerns! This seems like a sensible idea.