Open rabernat opened 5 years ago
Specifically, one will use gsutil with a command like
gsutil -m cp -r gcs://pangeo-data/llc4320_surface/SST .
Hi @glemson, have you given this a try?
I have not had the time to work on this yet.
-----Original Message----- From: Ryan Abernathey [mailto:notifications@github.com] Sent: Wednesday, October 17, 2018 3:12 PM To: pangeo-data/poseidon poseidon@noreply.github.com Cc: glemson gerard.lemson@gmail.com; Mention mention@noreply.github.com Subject: Re: [pangeo-data/poseidon] transfer LLC4320 zarr data to SciServer (#1)
Hi @glemson https://github.com/glemson , have you given this a try?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo- data/poseidon/issues/1#issuecomment-430752846 , or mute the thread https://github.com/notifications/unsubscribe- auth/AB64l39Oc0pZFiEmPbCpLE3PIQQlcvVdks5ul4EDgaJpZM4XaJui . https://github.com/notifications/beacon/AB64l5hD4-0MLSVMG3IYcl- dCyxPbpIIks5ul4EDgaJpZM4XaJui.gif
Problem with gsutil, most likely due to incorrect installation. $ filedb02:/srv/data01/ocean/LLC4320:50$ ~/gsutil/gsutil -m cp -r gcs://pangeo-data/llc4320_surface/SST . InvalidUrlError: Unrecognized scheme "gcs".
Use gs:// instead of gcs:///
On Thu, Dec 13, 2018 at 7:44 AM glemson notifications@github.com wrote:
Problem with gsutil, most likely due to incorrect installation. $ filedb02:/srv/data01/ocean/LLC4320:50$ ~/gsutil/gsutil -m cp -r gcs://pangeo-data/llc4320_surface/SST . InvalidUrlError: Unrecognized scheme "gcs".
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/poseidon/issues/1#issuecomment-446955462, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJFJkhLIVRymp1KhU-HU4aAzjKgAuaOks5u4kucgaJpZM4XaJui .
Thanks, it seems to be working.
-----Original Message----- From: Ryan Abernathey [mailto:notifications@github.com] Sent: Thursday, December 13, 2018 8:57 AM To: pangeo-data/poseidon poseidon@noreply.github.com Cc: glemson gerard.lemson@gmail.com; Mention mention@noreply.github.com Subject: Re: [pangeo-data/poseidon] transfer LLC4320 zarr data to SciServer (#1)
Use gs:// instead of gcs:///
On Thu, Dec 13, 2018 at 7:44 AM glemson notifications@github.com wrote:
Problem with gsutil, most likely due to incorrect installation. $ filedb02:/srv/data01/ocean/LLC4320:50$ ~/gsutil/gsutil -m cp -r gcs://pangeo-data/llc4320_surface/SST . InvalidUrlError: Unrecognized scheme "gcs".
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/poseidon/issues/1#issuecomment-4469554 62, or mute the thread https://github.com/notifications/unsubscribe-auth/ABJFJkhLIVRymp1KhU- HU4aAzjKgAuaOks5u4kucgaJpZM4XaJui .
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo- data/poseidon/issues/1#issuecomment-446976888 , or mute the thread https://github.com/notifications/unsubscribe- auth/AB64l3O4UOZjmZ0ArKox6cN-KtzeKTqGks5u4lyvgaJpZM4XaJui . https://github.com/notifications/beacon/AB64l0UGp8cr1IHioWb- 2NH51Q3M7jGsks5u4lyvgaJpZM4XaJui.gif
files have been downloaded to /SciServer/filedb02-01/ocean/LLC4320. I asked sysadmins to create a linux group 'poseidon' that should have access to this data. all/most of us will be member of that group. problem is currently I cannot chown to Poseidon, because apparently I am in too many groups. We'll solve that somehow.
Hello! Just adding to this trend. Only SST has been transferred to filedb02, and we are still missing (on SciServer) the rest of the surface variables available on the pangeo cloud, along with the GRID data. Now that I have access to the group directory, I will finish downloading the rest of the data.
@rabernat I tried downloading the rest of the surface variables onto SciServer, but was only able to download SSS and the grid. When I tried SSH, SSU, SSV or grid instead of SSS (or even SST) I get the following error
-bash-4.2$ ~/gsutil/gsutil -m cp -r gs://pangeo-data/llc4320_surface/SSH . CommandException: No URLs matched: gs://pangeo-data/llc4320_surface/SSH CommandException: 1 file/object could not be transferred.
I get the same error for SSU, and SSV (I also tried lower case). Do you know what is the correct URLs for these variables?
All the info is cataloger here: https://github.com/pangeo-data/pangeo-datastore/blob/master/intake-catalogs/ocean/llc4320.yaml
However, we need to pause these downloads for a moment. Your transfers have racked up nearly $2000 in egress fees in the past few days.
If I put the bucket into requester-pays mode, can you pay the remaining egress fees from your account? I think you should have lots of money in your CSSI budget for this. (We only have $5K a year.)
-Ryan
Ok, I was not aware of these fees. Another alternative that I am not very familiar is Globus Connect. Tom mentioned that we have access to an account from JHU to finish transferring the files. I will look into this confirm with you once I have this set up.
-Miguel
From: Ryan Abernathey notifications@github.com Sent: Tuesday, November 26, 2019 12:10 PM To: pangeo-data/poseidon poseidon@noreply.github.com Cc: Miguel Jimenez-Urias mjimen17@jhu.edu; Comment comment@noreply.github.com Subject: Re: [pangeo-data/poseidon] transfer LLC4320 zarr data to SciServer (#1)
All the info is cataloger here: https://github.com/pangeo-data/pangeo-datastore/blob/master/intake-catalogs/ocean/llc4320.yaml
However, we need to pause these downloads for a moment. Your transfers have racked up nearly $2000 in egress fees in the past few days.
If I put the bucket into requester-pays mode, can you pay the remaining egress fees from your account? I think you should have lots of money in your CSSI budget for this. (We only have $5K a year.)
-Ryan
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/pangeo-data/poseidon/issues/1?email_source=notifications&email_token=AB64CSLPFXIIQLHUN57DTMDQVVKBFA5CNFSM4F3ITORKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFGYJ2A#issuecomment-558728424, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB64CSOJVY3XUVUQF5KY3JDQVVKBFANCNFSM4F3ITORA.
Globus will not solve the problem. But I thought that this deal was supposed to help with the egress situation: https://www.internet2.edu/blogs/detail/14984
I'll ping our contacts at google for more info.
cc @lila
All of the data is cataloged here, with links to google cloud: http://pangeo.io/catalog.html