search

cms-dpoa / cloud-processing

Exploring the usage of public cloud resources for CMS open data processing
GNU General Public License v3.0
0 stars 0 forks source link

Quantify copy time from the processing pod to NFS Vs BUCKET #48

Open subash-taranga opened 2 weeks ago

subash-taranga commented 2 weeks ago

image

Logs can be find here processed file copy to bucket logs.txt

image

image image image image image

image image image subash@MSI:~/cloud-processing/standard-gke-cluster-gcs/argo$ argo logs @latest -n argo | grep 1342814727

image image

image

image

image

image

image image

image image

image

image

We can see file copy starts as soon as Closed file take place and copy end with "sub-process exited"

subash@MSI:~/cloud-processing/standard-gke-cluster-nfs/argo$ argo logs pfnano-process-ww642 -n argo | grep 1995231051

Can get full logs here processed file copy to nfs logs 1.txt processed file copy to nfs logs 2.txt image

image

image

image

image

image

image

tomcordruw commented 5 days ago

@subash-taranga I can confirm your observation from my end. When testing a dataset processing 3 million events with both gcs bucket and nfs, output files were around 800MB in size and they took around 40 seconds to copy.

Testing a whole dataset, 29 million events, output files were ~2.0-2.5GB files and it took around 2 minutes.

subash-taranga commented 3 days ago

@tomcordruw Yes it is correct....Thanks a lot