search

drphilmarshall / SpaceWarps

Science Team Website Development and Analysis
MIT License
12 stars 18 forks source link

SW-DES data location at the Zooniverse end #230

Open anumore opened 8 years ago

anumore commented 8 years ago

Where and how to transfer the SW-DES beta data and then actual data for the launch? For CFHTLS, we transferred the data to amazon servers using globus online interface.

aprajita commented 8 years ago

I already spoke to cam about this, as we cannot do it trough the project builder. they will help us when the time comes. I don't think they use that anymore but will check.

On Wednesday, 11 May 2016, Anupreeta notifications@github.com wrote:

Where and how to transfer the SW-DES beta data and then actual data for the launch? For CFHTLS, we transferred the data to amazon servers using globus online interface.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/230

anumore commented 8 years ago

yes, i remember that you mentioned something like that. but from the DES side, we would like to have more specific info for how to go about this if possible, so that we can begin setting things up in parallel to other tasks.

aprajita commented 8 years ago

tell me how many images you want for beta testing and i'll enquire

On 12 May 2016, at 00:39, Anupreeta notifications@github.com wrote:

yes, i remember that you mentioned something like that. but from the DES side, we would like to have more specific info for how to go about this if possible, so that we can begin setting things up in parallel to other tasks.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub

anumore commented 8 years ago

just to give you a number i'd say 500-1000 but we don't know yet for sure. also, for the beta sample, we can just use some ftp option but its good to have set things up and test with the beta sample already. thanks!

aprajita commented 8 years ago

Cam said this "Yeah, having the data on s3 and providing a CSV manifest (SRC URL, metadata) to us and we can import it. We can help with the s3 transfer too if needed" so should be straight forward.

If we want randomised subject IDs in the metadata - we will need to take of that from our end before delivery - these will then given a subject Id int eh db of the form .e.g. 2056727. if we were to only people uploading at the time all the panoptest subject IDs would be sequential.

e.g. one of the test data i just uploaded looks like https://www.zooniverse.org/projects/aprajita/swdemo/talk/subjects/2056728 but it's metadata ID is '2'

anumore commented 8 years ago

Thanks @aprajita
What is s3? I don't quite understand the reply though. Does Cam mean we can do the same thing as before?

aprajita commented 8 years ago

yes, s3 i think is the amazon data cloud http://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html

On 17 May 2016 at 23:57, Anupreeta notifications@github.com wrote:

Thanks @aprajita https://github.com/aprajita

What is s3? I don't quite understand the reply though. Does Cam mean we can do the same thing as before?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/230#issuecomment-219878837

aprajita commented 8 years ago

they can help with teh S3 transfer too not sure if it's different to what you did before