Open anumore opened 8 years ago
I already spoke to cam about this, as we cannot do it trough the project builder. they will help us when the time comes. I don't think they use that anymore but will check.
On Wednesday, 11 May 2016, Anupreeta notifications@github.com wrote:
Where and how to transfer the SW-DES beta data and then actual data for the launch? For CFHTLS, we transferred the data to amazon servers using globus online interface.
— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/230
yes, i remember that you mentioned something like that. but from the DES side, we would like to have more specific info for how to go about this if possible, so that we can begin setting things up in parallel to other tasks.
tell me how many images you want for beta testing and i'll enquire
On 12 May 2016, at 00:39, Anupreeta notifications@github.com wrote:
yes, i remember that you mentioned something like that. but from the DES side, we would like to have more specific info for how to go about this if possible, so that we can begin setting things up in parallel to other tasks.
— You are receiving this because you commented. Reply to this email directly or view it on GitHub
just to give you a number i'd say 500-1000 but we don't know yet for sure. also, for the beta sample, we can just use some ftp option but its good to have set things up and test with the beta sample already. thanks!
Cam said this "Yeah, having the data on s3 and providing a CSV manifest (SRC URL, metadata) to us and we can import it. We can help with the s3 transfer too if needed" so should be straight forward.
If we want randomised subject IDs in the metadata - we will need to take of that from our end before delivery - these will then given a subject Id int eh db of the form .e.g. 2056727. if we were to only people uploading at the time all the panoptest subject IDs would be sequential.
e.g. one of the test data i just uploaded looks like https://www.zooniverse.org/projects/aprajita/swdemo/talk/subjects/2056728 but it's metadata ID is '2'
Thanks @aprajita
What is s3? I don't quite understand the reply though. Does Cam mean we can do the same thing as before?
yes, s3 i think is the amazon data cloud http://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html
On 17 May 2016 at 23:57, Anupreeta notifications@github.com wrote:
Thanks @aprajita https://github.com/aprajita
What is s3? I don't quite understand the reply though. Does Cam mean we can do the same thing as before?
— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/230#issuecomment-219878837
they can help with teh S3 transfer too not sure if it's different to what you did before
Where and how to transfer the SW-DES beta data and then actual data for the launch? For CFHTLS, we transferred the data to amazon servers using globus online interface.