search

pangeo-data / pangeo-cmip6-cloud

Documentation for Pangeo CMIP6 data stored in GCP/AWS cloud
https://pangeo-data.github.io/pangeo-cmip6-cloud/
17 stars 9 forks source link

Size of pangeo cloud holding #37

Closed jbusecke closed 9 months ago

jbusecke commented 2 years ago

@aradhakrishnanGFDL had questions about the size of the cloud holdings. Do we have a method of checking on this currently?

cc @naomi-henderson @rabernat

aradhakrishnanGFDL commented 2 years ago

@jbusecke From the AWS management console, S3 and then S3 storage lens provides quick highlights including the size. Include only the current versions as a filter. Hope that helps!

jbusecke commented 2 years ago

it does. Thanks a lot. Ill try to work on this next week, since I have to get the credentials first. Does anybody know if comparable tools exist for GCS?

delgadom commented 2 years ago

Not sure if this is what you're looking for, but you can view the size of a bucket & object count on the monitoring dashboard:

https://console.cloud.google.com/monitoring/dashboards/resourceList/gcs_bucket?project=PROJECT_ID

jbusecke commented 9 months ago

Moved to https://github.com/leap-stc/cmip6-leap-feedstock/issues/93

durack1 commented 9 months ago

Not sure if this is what you're looking for, but you can view the size of a bucket & object count on the monitoring dashboard:

https://console.cloud.google.com/monitoring/dashboards/resourceList/gcs_bucket?project=PROJECT_ID

@delgadom dumb question, do you need certain perms within the project for the URL to resolve? I get a Failed to load. error

delgadom commented 8 months ago

You definitely do need pretty broad permissions. Just checking - did you replace or drop the project id parameter? But yeah you’ll need a pretty wide set of permissions on the project. I’m not sure how grant projects work - you could check with your rep to see if this is possible. Hope that’s helpful!

durack1 commented 8 months ago

You definitely do need pretty broad permissions. Just checking - did you replace or drop the project id parameter? But yeah you’ll need a pretty wide set of permissions on the project. I’m not sure how grant projects work - you could check with your rep to see if this is possible. Hope that’s helpful!

I literally used the exact url listed above https://github.com/pangeo-data/pangeo-cmip6-cloud/issues/37#issuecomment-1917621152 - so maybe I need to switchout "csiro" or whatever the correct ID is?