search

MIRC-UofSC / xsede-development

0 stars 0 forks source link

Data transfer #2

Open zonca opened 3 years ago

zonca commented 3 years ago

@wilsbacher @junzhoum

While I wait for feedback on streaming from object store (#1), I can start evaluating your data transfer needs.

In the workplan, we wrote:

how do we efficiently transfer the current amount of data and future capability (e.g., exposing videos to analysis by neural network to detect information in the videos will require either (1) use of a higher quality video (Prores or FFv1) or (2) directories of DPX/TIFF files. Streaming will be performed by mp4 contributions to the archive. We will explore Globus and other parallel transfer options.

Can you explain here more in detail what you would like to achieve? For example:

I think it is useful to have some ideas written down here, then we can schedule a call to discuss more.

zonca commented 3 years ago

I have an old tutorial I wrote about configuring Globus I could start from and update (https://zonca.dev/2017/04/globus-gridftp-local.html)

junzhoum commented 3 years ago

Hi Andrea,

Thanks for the update. I need discuss with Greg and the student for all the answers. It has been extremely slow in the recent weeks due to final exam weeks and graduation season. The student is leaving. I will get back to you as soon as I get the answers.

Thank you.

Jun