Closed kbergin closed 5 years ago
kbergin
commented
5 years ago ➤ Saman Ehsan commented:
We should keep the hca-dcp-sc-pipelines-test-data bucket or move it to the production google project – it contains the reference files that we use for Optimus and SS2.
kbergin
commented
5 years ago We should definitely move that to the production google project. The references are used for production! I’m not sure if that will break any automated tests on skylab, due to permission issues. Thanks Saman!
kbergin
commented
5 years ago ➤ Chengchen Wang commented:
The bucket broad-dsde-mint-dev-credentials ( https://console.cloud.google.com/storage/browser/broad-dsde-mint-dev-credentials?project=broad-dsde-mint-dev ) needs to be kept, which hosts our service account key for dev.
The bucket broad-dsde-mint-dev-cromwell-execution ( https://console.cloud.google.com/storage/browser/broad-dsde-mint-dev-cromwell-execution?project=broad-dsde-mint-dev ) theoretically could be cleaned up since we shut down the mint-dev Cromwell already!
kbergin
commented
5 years ago ➤ Chengchen Wang commented:
Also be careful with the bucket https://console.cloud.google.com/storage/browser/hca-dcp-mint-test-data?project=broad-dsde-mint-dev ( https://console.cloud.google.com/storage/browser/hca-dcp-mint-test-data?project=broad-dsde-mint-dev ) since it hosts a lot of random stuff that might be used by a lot of random scripts, such as the monitoring script, or the benchmarking scripts in skylab-analysis repo
kbergin
commented
5 years ago ➤ Nick Barkas commented:
I want to point out that the following resources are still in use:
Buckets:
hca-dcp-mint-test-data ( https://console.cloud.google.com/storage/browser/hca-dcp-mint-test-data?project=broad-dsde-mint-dev ) (in progress of being phased out but a lot of useful data are there)
hca-dcp-sc-pipelines-test-data ( https://console.cloud.google.com/storage/browser/hca-dcp-sc-pipelines-test-data?project=broad-dsde-mint-dev ) (new bucket I created a few months ago with clear provenance of data)
VMs:
nb-devbox-1: my older devbox, I keep it off as I don’t use it. The only reason it exists is that I haven’t checked if I need any data in it
nb-devbox-2: my current devbox, I use for all current work that needs to be interactive and can’t run on my laptop
mint-dev-www ( https://console.cloud.google.com/compute/instancesDetail/zones/us-central1-c/instances/mint-dev-www?project=broad-dsde-mint-dev ): This is the mint web server I set up early on to share data in reports still use it if I need to share apps
kbergin
commented
5 years ago ➤ Rhian Anthony commented:
I copied the contents of hca-dcp-sc-pipelines-test-data to the bucket named hca-dcp-analysis-pipelines-reference in the hca-dcp-pipelines-prod google project (our production HCA project)
kbergin
commented
5 years ago ➤ Rhian Anthony commented:
I deleted broad-dsde-mint-dev-cromwell-execution since as Rex indicated we no longer use that cromwell.
kbergin
commented
5 years ago [~accountid:5a663714390d0d1daa454b88] I linked the ticket about organizing the references, was there anything else we wanted to do as fallout of this work?
kbergin
commented
5 years ago ➤ Nick Barkas commented:
FYI, I have changed my devbox instance to use a 5 TB instead of 10TB drive so that should help too. There are a few more dev instances that might be possible to cleanup.
kbergin
commented
5 years ago ➤ Rhian Anthony commented:
Closing ticket to open targeted subtickets
We’re currently spending too much money per month on the storage under the google project broad-dsde-mint-dev
There is likely a lot that can be cleaned up, policies we can put in place to avoid manual cleanup in the future, and possibly some things we should keep.
If you have something you need to keep, please comment below by 7/31.
┆Issue is synchronized with this Jira Ops Needs