Closed gordonwatts closed 3 months ago
(venv) [bash][gwatts]:idap-200gbps-atlas > python servicex/servicex_materialize_branches.py -v --distributed-client scheduler --dask-scheduler 'tcp://dask-gwatts-2e1782e2-0.af-jupyter:8786' --dask-profile --dataset mc_1TB --query xaod_all --num-files 0
0000.0504 - INFO - root - Using release 22.2.107 for type information.
0000.7540 - INFO - root - Running over 1 datasets, 1.222 TB and 136458000 events.
0000.7542 - INFO - root - Building ServiceX query
0000.7546 - INFO - root - Querying dataset mc20_13TeV:mc20_13TeV.364157.Sherpa_221_NNPDF30NNLO_Wmunu_MAXHTPTV0_70_CFilterBVeto.deriv.DAOD_PHYSLITE.e5340_s3681_r13145_p6026
0000.7546 - INFO - root - Running on the full dataset(s).
0000.7547 - INFO - root - Starting ServiceX query
0000.7711 - INFO - servicex.servicex_client - Returning code generators from cache
Transform ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/?
Download/URLs ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/? 0002.5382 - INFO - servicex.query - ServiceX Transform speed_test_mc20_13TeV:mc20_13TeV.364157.Sherpa_221_NNPD
Transform ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/?
Transform ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/?
Download/URLs ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1213/1211 06:01
0571.2979 - INFO - root - Event rate for ServiceX: 00:09:30 time, 239.17 kHz, Data rate: 17.13 Gbits/s
0571.2980 - INFO - root - Dataset speed_test_mc20_13TeV:mc20_13TeV.364157.Sherpa_221_NNPDF30NNLO_Wmunu_MAXHTPTV0_70_CFilterBVeto.deriv.DAOD_PHYSLITE.e5340_s3681_r has 1165 files
0580.0327 - INFO - root - Number of skimmed events: 130668000 (skim percent: 95.7569%)
0580.1648 - INFO - root - Using `uproot.dask` to open files (splitting files 2 ways).
0580.1649 - INFO - root - Starting build of DASK graphs
0580.6522 - INFO - root - Computing the total count
0635.8339 - INFO - root - Event rate for DASK Calculation: 00:00:55 time, 2472.90 kHz, Data rate: 177.16 Gbits/s
0635.8340 - INFO - root - DASK event rate over actual events: 2367.98 kHz
0635.8341 - INFO - root - speed_test_mc20_13TeV:mc20_13TeV.364157.Sherpa_221_NNPDF30NNLO_Wmunu_MAXHTPTV0_70_CFilterBVeto.deriv.DAOD_PHYSLITE.e5340_s3681_r: result = 130,668,000
Duration: 49.68 s
Tasks Information
number of tasks: 7682
compute time: 3hr 39m
transfer time: 92.79 s
Was pre-set with 100 workers
No idea what the second bump was in the xcache rate - but note we never get above 16 Gbps. We had about 100-300 workers requested, but likely they weren't able to get to work by the time it finished.
The key here is that we don't even get up to speed in the xcache output - the ramping is too slow.