search

2i2c-org / infrastructure

Infrastructure for configuring and deploying our community JupyterHubs.
https://infrastructure.2i2c.org
BSD 3-Clause "New" or "Revised" License
108 stars 65 forks source link

[catalystproject-af, bhki] Enable object storage #4216

Closed jnywong closed 5 months ago

jnywong commented 5 months ago

Context

The bhki community would benefit from the recommended data transfer workflow proposed in #4213 to get started with analysing ~GBs of .fastq files for RNA sequencing.

Proposal

Set up a persistent storage bucket so that they can upload genomics data from a local server.

Updates and actions

No response

sgibson91 commented 5 months ago

I'm looking at this next.

Since we haven't got the full "LEAP" style setup in AWS, we will also need to figure out the "magic incantation" users will need to get their credentials to use to push to the bucket locally.

sgibson91 commented 5 months ago

There is now a persistent bucket set up for the BHKI community

sgibson91 commented 5 months ago

"Magic incantation" is here: https://github.com/scottyhq/jupyter-cloud-scoped-creds/blob/f9e30953f20bc3fb6971077f847c7021b88c728a/jupyter_cloud_scoped_creds/__init__.py#L16

I will find an appropriate hub to test this on and create a minimal example of pushing a file from local to the hub.

@jnywong For this to work, the AWS CLI needs to be installed in the image and it currently isn't as this community are using the Jupyter SciPy image. The Pangeo image ships with the AWS CLI installed, so we should communicate that that is the better image choice for this particular workflow.

sgibson91 commented 5 months ago

I recorded a video and shared directly with @jnywong on how to use the magic incantation to upload data from a local machine to an AWS S3 bucket and then seen the data from the hub. Hopefully that is enough to create some documentation for the community, but I am happy to answer further questions.

jnywong commented 5 months ago

Amazing 😍 I am looking forward to sharing a Catalyst demo with you soon because of your work 👏