Can't download Response file

[parshadkp]

Downloading "cosi-pipeline-public/COSI-SMEX/DC2/ResponsesSMEXv12.Continuum.HEALPixO3_10bins_log_flat.binnedimaging.imagingresponse.nonsparse_nside8.area.good_chunks_unzip.h5.zip" from wasabi gives an error.

The actual download results in this xml file.

[ckarwin]

Hi @parshadkp, It looks like you may be missing / after Responses in your path. So the full path should be:

cosi-pipeline-public/COSI-SMEX/DC2/Responses/SMEXv12.Continuum.HEALPixO3_10bins_log_flat.binnedimaging.imagingresponse.nonsparse_nside8.area.good_chunks_unzip.h5.zip

If this does not fix the problem, then can you please give the exact command that you're running and the error?

[parshadkp]

Hi @ckarwin, sorry that is just a typo in my post. I get that error from downloading through wasabi's website.

I also tried the command (which gives me the error below): (COSI) parshadkp@Parshads-MacBook-Pro cosi-data-challenge-2 % AWS_ACCESS_KEY_ID=GBAL6XATQZNRV3GFH9Y4 AWS_SECRET_ACCESS_KEY=GToOczY5hGX3sketNO2fUwiq4DJoewzIgvTCHoOv aws s3api get-object --bucket cosi-pipeline-public --key cosi-pipeline-public/COSI-SMEX/DC2/Responses/SMEXv12.Continuum.HEALPixO3_10bins_log_flat.binnedimaging.imagingresponse.nonsparse_nside8.area.good_chunks_unzip.h5.zip --endpoint-url=https://s3.us-west-1.wasabisys.com SMEXv12.Continuum.HEALPixO3_10bins_log_flat.binnedimaging.imagingresponse.nonsparse_nside8.area.good_chunks_unzip.h5.zip

An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

[ckarwin]

The --key should not include cosi-pipeline-public/. That is specified in the --bucket. So the command is:

AWS_ACCESS_KEY_ID=GBAL6XATQZNRV3GFH9Y4 AWS_SECRET_ACCESS_KEY=GToOczY5hGX3sketNO2fUwiq4DJoewzIgvTCHoOv aws s3api get-object --bucket cosi-pipeline-public --key COSI-SMEX/DC2/Responses/SMEXv12.Continuum.HEALPixO3_10bins_log_flat.binnedimaging.imagingresponse.nonsparse_nside8.area.good_chunks_unzip.h5.zip --endpoint-url=https://s3.us-west-1.wasabisys.com SMEXv12.Continuum.HEALPixO3_10bins_log_flat.binnedimaging.imagingresponse.nonsparse_nside8.area.good_chunks_unzip.h5.zip

I just tested it, and it works for me. Please try and let me know if it works.

[parshadkp]

Ah, it works. Thank you so much!

Is it not possible to download from the wasabi website though? It results in the .xml file when I try to.

[ckarwin]

It should also be possible to download from the website, although I have gotten a similar error at times in the past. I'm not really sure why this happens. Usually the issue can be resolved by logging out and then logging in again. If that doesn't work, then try downloading the file again at a later time (e.g. maybe in an hour or two). Please let me know if you are eventually able to download the file from the website or if the issue persist.

[parshadkp]

Tried logging in and out. Still the same .xml file that is downloaded. It only happens with the zip files. I'm only a Mac M1 if that helps at all.

[ckarwin]

Hi @parshadkp Thank you for the feedback. To clarify, do the .gz files work ok, and the only issue is with the .zip files?

[parshadkp]

Just tried a .gz file, doesn't work either. Getting the same .xml file when trying to download any files from wasabi's website. The command line works fine though.

[ckarwin]

Ok, I'll look into this and get back to you.

[ckarwin]

Hi @parshadkp, I have clarified the download instructions on the DC2 git page. In summary, for users outside of the COSI collaboration, the data can be downloaded using the command line prompt. Developers within the COSI collaboration can also access the data on wasabi, but they will need the appropriate logon credentials. Can you clarify for me: are you in the COSI collaboration? If so, are you able to download the files now from wasabi (using the needed logon info), or is this still an issue?

[parshadkp]

Hi @ckarwin, I am on the COSI collaboration but I do not have the login credentials for the Wasabi website.

[ckarwin]

@parshadkp Can you send me a direct message on COSI slack and I'll send you the login credentials for Wasabi?

[ckarwin]

I sent @parshadkp all the logon info needed for wasabi, and the files can now be downloaded ok. That closes this issue.