Test the 1 TB dataset for rates

[gordonwatts]

The key here is that we don't even get up to speed in the xcache output - the ramping is too slow.

[gordonwatts]

(venv) [bash][gwatts]:idap-200gbps-atlas > python servicex/servicex_materialize_branches.py -v --distributed-client scheduler --dask-scheduler 'tcp://dask-gwatts-2e1782e2-0.af-jupyter:8786' --dask-profile --dataset mc_1TB --query xaod_all --num-files 0
0000.0504 - INFO - root - Using release 22.2.107 for type information.
0000.7540 - INFO - root - Running over 1 datasets, 1.222 TB and 136458000 events.
0000.7542 - INFO - root - Building ServiceX query
0000.7546 - INFO - root - Querying dataset mc20_13TeV:mc20_13TeV.364157.Sherpa_221_NNPDF30NNLO_Wmunu_MAXHTPTV0_70_CFilterBVeto.deriv.DAOD_PHYSLITE.e5340_s3681_r13145_p6026
0000.7546 - INFO - root - Running on the full dataset(s).
0000.7547 - INFO - root - Starting ServiceX query
0000.7711 - INFO - servicex.servicex_client - Returning code generators from cache
Transform     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/?  
Download/URLs ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/?  0002.5382 - INFO - servicex.query - ServiceX Transform speed_test_mc20_13TeV:mc20_13TeV.364157.Sherpa_221_NNPD
Transform     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/?            
Transform     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0/?            
Download/URLs ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1213/1211 06:01
0571.2979 - INFO - root - Event rate for ServiceX: 00:09:30 time, 239.17 kHz, Data rate: 17.13 Gbits/s
0571.2980 - INFO - root - Dataset speed_test_mc20_13TeV:mc20_13TeV.364157.Sherpa_221_NNPDF30NNLO_Wmunu_MAXHTPTV0_70_CFilterBVeto.deriv.DAOD_PHYSLITE.e5340_s3681_r has 1165 files
0580.0327 - INFO - root - Number of skimmed events: 130668000 (skim percent: 95.7569%)
0580.1648 - INFO - root - Using `uproot.dask` to open files (splitting files 2 ways).
0580.1649 - INFO - root - Starting build of DASK graphs
0580.6522 - INFO - root - Computing the total count
0635.8339 - INFO - root - Event rate for DASK Calculation: 00:00:55 time, 2472.90 kHz, Data rate: 177.16 Gbits/s
0635.8340 - INFO - root - DASK event rate over actual events: 2367.98 kHz
0635.8341 - INFO - root - speed_test_mc20_13TeV:mc20_13TeV.364157.Sherpa_221_NNPDF30NNLO_Wmunu_MAXHTPTV0_70_CFilterBVeto.deriv.DAOD_PHYSLITE.e5340_s3681_r: result = 130,668,000

Duration: 49.68 s
Tasks Information
number of tasks: 7682
compute time: 3hr 39m
transfer time: 92.79 s

Was pre-set with 100 workers

[gordonwatts]

No idea what the second bump was in the xcache rate - but note we never get above 16 Gbps. We had about 100-300 workers requested, but likely they weren't able to get to work by the time it finished.