2i2c-org / infrastructure

Infrastructure for configuring and deploying our community JupyterHubs.
https://infrastructure.2i2c.org
BSD 3-Clause "New" or "Revised" License
103 stars 63 forks source link

[Support] Reduce awi-ciroh Filestore to 3 TB #4844

Open jmunroe opened 1 week ago

jmunroe commented 1 week ago

The Freshdesk ticket link

https://2i2c.freshdesk.com/a/tickets/2170

Ticket request type

Other

Ticket impact

🟨 Medium

Short ticket description

The filestore size for awi-ciroh has decreased from 9TB to 4.7TB. To avoid charges for unused storage, the community wants to update the filestore capacity to ~5TB~ 3TB.

(Optional) Investigation results

Since this require creating a new filestore and transferring data over plus coordinating with the community, I think it should be schedule in the next engineering sprint.

sgibson91 commented 5 days ago

I had to request a quota increase to accommodate the size of the second filestore, so let's see how long that takes to come back.

sgibson91 commented 5 days ago

Quota increase approved

sgibson91 commented 5 days ago

Because AWI-CIROH are running in their own GCP project, I do not have the right access permissions to SSH into the VM I created to mount the 2 filestores and copy the data over.

Insufficient IAM permissions. The instance belongs to an external organization. You must be granted the roles/compute.osLoginExternalUser IAM role on the external organization to configure POSIX account information.

sgibson91 commented 5 days ago

I worked around the above restriction by modifying our deployer exec root-homes command to mount the two filestores. The data copying has begun!

jmunroe commented 15 hours ago

Community has revised their request to be 3TB.

sgibson91 commented 13 hours ago

I am having problems with my workaround modifying the deployer exec root-homes command to mount both filestores. The pod seems to be getting killed long before the transfer has completed.