search

numbbo / coco

Numerical Black-Box Optimization Benchmarking Framework
https://numbbo.github.io/coco
Other
261 stars 86 forks source link

loading times for data #1980

Open nikohansen opened 4 years ago

nikohansen commented 4 years ago

EDIT: these have become first-time loading times. After the first time, the max loading time is now <0.15 seconds.

The loading times for the bbob data sets range from 2.5 to 50 seconds on a 2019 Macbook Pro. Below is the full list, useful to find quick-to-load datasets for testing or playing.

ds_list = cocopp.load('13/CMA')

is the quickest-to-load sauber bbob dataset.

ds_list = cocopp.load('b/2009/bay')

is the fastest to load sauber 2009 data set with the 2009 instance repetitions 3 * [1,2,3,4,5].

import cocopp, cma

loading_times = previous_times
for name in cocopp.bbob:
    time = cma.utilities.utils.ElapsedWCTime()
    dsl = cocopp.load('bbob/' + name)
    loading_times += [(time.toc, name)]
print(sorted(loading_times))  # some appear twice to get a dispersion estimate

Timing measures (click on Details)

``` [(2.5123581886291504, '2009/DIRECT_posik_noiseless.tgz'), (2.8592660427093506, '2009/DIRECT_posik_noiseless.tgz'), (3.224440097808838, '2013/CMAES_Hutter_hutter_noiseless.tgz'), (3.2957026958465576, '2013/CMAES_Hutter_hutter_noiseless.tgz'), (3.577888011932373, '2009/MCS_huyer_noiseless.tgz'), (3.6324799060821533, '2009/GLOBAL_pal_noiseless.tgz'), (3.748338222503662, '2013/IPOP400D_auger_noiseless.tgz'), (3.8477797508239746, '2013/IPOP400D_auger_noiseless.tgz'), (3.8557822704315186, '2009/GLOBAL_pal_noiseless.tgz'), (3.865396022796631, '2009/MCS_huyer_noiseless.tgz'), (3.9278602600097656, '2013/fminunc_pal_noiseless.tgz'), (3.9708099365234375, '2013/fmincon_pal_noiseless.tgz'), (4.042601108551025, '2013/fminunc_pal_noiseless.tgz'), (4.070181131362915, '2013/simplex_pal_noiseless.tgz'), (4.078040838241577, '2013/simplex_pal_noiseless.tgz'), (4.119091033935547, '2013/fmincon_pal_noiseless.tgz'), (4.24816632270813, '2009/BAYEDA_gallagher_noiseless.tgz'), (4.305064916610718, '2013/HMLSL_pal_noiseless.tgz'), (4.309692144393921, '2013/HMLSL_pal_noiseless.tgz'), (4.373304843902588, '2013/OQNLP_pal_noiseless.tgz'), (4.3840651512146, '2009/BAYEDA_gallagher_noiseless.tgz'), (4.4418230056762695, '2013/MLSL_pal_noiseless.tgz'), (4.477442026138306, '2013/HCMA_loshchilov_noiseless.tgz'), (4.54229998588562, '2010/1komma2ser_brockhoff_noiseless.tar.gz'), (4.553889036178589, '2010/1komma4mirser_brockhoff_noiseless.tar.gz'), (4.603099822998047, '2010/1komma4ser_brockhoff_noiseless.tar.gz'), (4.636629819869995, '2010/1komma2mirser_brockhoff_noiseless.tar.gz'), (4.676295757293701, '2012/BIPOPsaACM_loshchilov_noiseless.tgz'), (4.717749118804932, '2010/1komma2mir_brockhoff_noiseless.tar.gz'), (4.731079816818237, '2010/1komma2_brockhoff_noiseless.tar.gz'), (4.740843057632446, '2010/1komma4_brockhoff_noiseless.tar.gz'), (4.7526631355285645, '2009/FULLNEWUOA_ros_noiseless.tgz'), (4.76731014251709, '2009/NELDERDOERR_doerr_noiseless.tgz'), (4.769866466522217, '2010/1komma4mir_brockhoff_noiseless.tar.gz'), (4.990481853485107, '2009/PSO_el-abd_noiseless.tgz'), (5.0055859088897705, '2009/MA-LS-CHAIN_molina_noiseless.tgz'), (5.0066399574279785, '2009/BAYEDA_gallagher_noiseless.tgz'), (5.014755010604858, '2012/IPOPsaACM_loshchilov_noiseless.tgz'), (5.035427093505859, '2013/DE_Pal_pal_noiseless.tgz'), (5.041011333465576, '2013/CGA-ring100_holtschulte_noiseless.tgz'), (5.080881118774414, '2013/CGA-grid100_holtschulte_noiseless.tgz'), (5.1218650341033936, '2013/CGA-grid16_holtschulte_noiseless.tgz'), (5.155517816543579, '2009/BAYEDA_gallagher_noiseless.tgz'), (5.202517032623291, '2010/1plus1_brockhoff_noiseless.tar.gz'), (5.275317907333374, '2010/1plus2mirser_brockhoff_noiseless.tar.gz'), (5.280163764953613, '2013/U-zero_tran_noiseless.tgz'), (5.330496072769165, '2013/U-DCN_tran_noiseless.tgz'), (5.335342884063721, '2009/PSO_Bounds_el-abd_noiseless.tgz'), (5.343286752700806, '2013/HILL_holtschulte_noiseless.tgz'), (5.352894306182861, '2009/VNS_garcia-martinez_noiseless.tgz'), (5.35558295249939, '2013/CGA-ring16_holtschulte_noiseless.tgz'), (5.399998903274536, '2013/GA-100_holtschulte_noiseless.tgz'), (5.406499147415161, '2009/DE-PSO_garcia-nieto_noiseless.tgz'), (5.4136271476745605, '2010/oPOEMS_kubalic_noiseless.tar.gz'), (5.4550461769104, '2012/CMAma_brockhoff_noiseless.tgz'), (5.472886800765991, '2012/CMAES_posik_noiseless.tgz'), (5.49519681930542, '2009/EDA-PSO_el-abd_noiseless.tgz'), (5.495300054550171, '2012/DEctpb_posik_noiseless.tgz'), (5.5057971477508545, '2009/POEMS_kubalik_noiseless.tgz'), (5.550126075744629, '2010/NBC-CMA_preuss_noiseless.tar.gz'), (5.5731120109558105, '2012/xNES_schaul_noiseless.tgz'), (5.579071998596191, '2012/CMAa_brockhoff_noiseless.tgz'), (5.616408109664917, '2012/SNES_schaul_noiseless.tgz'), (5.6214001178741455, '2013/P-DCN_tran_noiseless.tgz'), (5.6292197704315186, '2010/DE-F-AUC_fialho_noiseless.tar.gz'), (5.669055938720703, '2009/LSfminbnd_posik_noiseless.tgz'), (5.685362815856934, '2012/CMA_brockhoff_noiseless.tgz'), (5.708168029785156, '2013/BIPOP-saACM-k_loshchilov_noiseless.tgz'), (5.7241339683532715, '2012/CMAm_brockhoff_noiseless.tgz'), (5.725514888763428, '2012/DE-BFGS_voglis_noiseless.tgz'), (5.735372066497803, '2010/pPOEMS_kubalic_noiseless.tar.gz'), (5.76050591468811, '2012/DEAE_posik_noiseless.tgz'), (5.764648199081421, '2010/PM-AdapSS-DE_fialho_noiseless.tar.gz'), (5.768438100814819, '2012/CMAmh_brockhoff_noiseless.tgz'), (5.771177053451538, '2010/RCGA_tran_noiseless.tar.gz'), (5.799239873886108, '2013/P-zero_tran_noiseless.tgz'), (5.801631927490234, '2012/CMAmah_brockhoff_noiseless.tgz'), (5.83924412727356, '2009/CMA-ESPLUSSEL_auger_noiseless.tgz'), (5.871999025344849, '2012/PSO-BFGS_voglis_noiseless.tgz'), (5.876449108123779, '2009/LSstep_posik_noiseless.tgz'), (5.876784086227417, '2013/BIPOP-aCMA-STEP_loshchilov_noiseless.tgz'), (5.884450912475586, '2012/JADEctpb_posik_noiseless.tgz'), (5.891248941421509, '2013/IP_liao_noiseless.tgz'), (5.903611898422241, '2010/DEuniform_fialho_noiseless.tar.gz'), (5.906591892242432, '2012/DE_posik_noiseless.tgz'), (5.906638860702515, '2013/tany_liao_noiseless.tgz'), (5.931397199630737, '2012/DE-AUTO_voglis_noiseless.tgz'), (5.942446231842041, '2013/IP-10DDr_liao_noiseless.tgz'), (5.9566428661346436, '2012/NIPOPaCMA_loshchilov_noiseless.tgz'), (5.982802152633667, '2012/JADE_posik_noiseless.tgz'), (5.997680902481079, '2010/MOS_torre_noiseless.tar.gz'), (6.032480001449585, '2012/JADEb_posik_noiseless.tgz'), (6.0379319190979, '2013/IP-500_liao_noiseless.tgz'), (6.067556858062744, '2012/NBIPOPaCMA_loshchilov_noiseless.tgz'), (6.069911003112793, '2009/BIPOP-CMA-ES_hansen_noiseless.tgz'), (6.0803210735321045, '2012/DE-SIMPLEX_voglis_noiseless.tgz'), (6.085437059402466, '2013/texp_liao_noiseless.tgz'), (6.08591103553772, '2009/Rosenbrock_posik_noiseless.tgz'), (6.099390029907227, '2012/ACOR_liao_noiseless.tgz'), (6.180742025375366, '2012/xNESas_schaul_noiseless.tgz'), (6.206459283828735, '2013/PRCGA_sawyerr_noiseless.tgz'), (6.236809968948364, '2012/DE-ROLL_voglis_noiseless.tgz'), (6.246721982955933, '2012/DEb_posik_noiseless.tgz'), (6.263749122619629, '2012/DBRCGA_chuang_noiseless.tgz'), (6.297060012817383, '2009/Cauchy-EDA_posik_noiseless.tgz'), (6.314383029937744, '2010/IPOP-CMA-ES_ros_noiseless.tar.gz'), (6.336417198181152, '2010/IPOP-ACTCMA-ES_ros_noiseless.tar.gz'), (6.361392021179199, '2010/CMAEGS_finck_noiseless.tar.gz'), (6.389548063278198, '2009/AMALGAM_bosman_noiseless.tgz'), (6.391831159591675, '2013/MEMPSODE_voglis_noiseless.tgz'), (6.495636940002441, '2009/CMA-ESPLUSSEL_auger_noiseless.tgz'), (6.501638174057007, '2009/AMALGAM_bosman_noiseless.tgz'), (6.588334798812866, '2009/ALPS_hornby_noiseless.tgz'), (6.629495859146118, '2009/ALPS_hornby_noiseless.tgz'), (6.723217010498047, '2009/ALPS_hornby_noiseless.tgz'), (6.814644813537598, '2009/BIPOP-CMA-ES_hansen_noiseless.tgz'), (6.822392225265503, '2012/MVDE_melo_noiseless.tgz'), (6.882508993148804, '2010/ABC_elabd_noiseless.tar.gz'), (7.025951147079468, '2009/AMALGAM_bosman_noiseless.tgz'), (7.0407068729400635, '2009/G3PCX_posik_noiseless.tgz'), (7.2292869091033936, '2009/GA_nicolau_noiseless.tgz'), (7.237746238708496, '2009/RANDOMSEARCH_auger_noiseless.tgz'), (7.588737964630127, '2009/iAMALGAM_bosman_noiseless.tgz'), (7.666553020477295, '2009/ONEFIFTH_auger_noiseless.tgz'), (7.862884759902954, '2009/IPOP-SEP-CMA-ES_ros_noiseless.tgz'), (9.915147066116333, '2010/AVGNEWUOA_ros_noiseless.tar.gz'), (11.55675220489502, '2009/NEWUOA_ros_noiseless.tgz'), (11.664323091506958, '2009/BFGS_ros_noiseless.tgz'), (12.476078987121582, '2009/BFGS_ros_noiseless.tgz'), (14.19924283027649, '2009/NELDER_hansen_noiseless.tgz'), (16.020208835601807, '2013/SMAC-BBOB_hutter_noiseless.tgz'), (25.383677005767822, '2013/lmm-CMA-ES_auger_noiseless.tgz'), (35.69036602973938, '2010/SPSA_finck_noiseless.tar.gz'), (40.0265748500824, '2012/BIPOPaCMA_loshchilov_noiseless.tgz'), (52.210891008377075, '2009/DASA_korosec_noiseless.tgz')] ```

EDIT: loading times after the first load with get_DataSetList are now below 0.15 seconds (yet to be pushed):

``` [(0.019949674606323242, '2009/DIRECT_posik_noiseless.tgz'), (0.02297806739807129, '2014-others/UNIF-cocopf_Baudis.tgz'), (0.024075984954833984, '2015-CEC/R-SHADE-10e2_Tanabe.tgz'), (0.024497032165527344, '2015-GECCO/BrentSTEPqi_Posik.tgz'), (0.025784015655517578, '2017/EvoSpace-PSO-GA_Garcia-Valdez.tgz'), (0.025806665420532227, '2015-CEC/R-DE-10e2_Tanabe.tgz'), (0.02611088752746582, '2015-CEC/RL-SHADE-10e2_Tanabe.tgz'), (0.026412010192871094, '2015-CEC/MATSUMOTO_Brockhoff.tgz'), (0.026892900466918945, '2014-others/SLSQP-scipy_Baudis.tgz'), (0.02772212028503418, '2015-GECCO/BrentSTEPrr_Posik.tgz'), (0.027846813201904297, '2014-others/Powell-scipy_Baudis.tgz'), (0.02857375144958496, '2014-others/L-BFGS-B-scipy_Baudis.tgz'), (0.02871990203857422, '2013/SMAC-BBOB_hutter_noiseless.tgz'), (0.0295102596282959, '2015-GECCO/LHD-10xDefault-MATSuMoTo_Brockhoff.tgz'), (0.029691219329833984, '2015-GECCO/RAND-2xDefault-MATSuMoTo_Brockhoff.tgz'), (0.0300600528717041, '2015-GECCO/BrentSTEPifeg_Posik.tgz'), (0.03011608123779297, '2015-GECCO/STEPif_Posik.tgz'), (0.03015589714050293, '2009/MCS_huyer_noiseless.tgz'), (0.030373096466064453, '2015-GECCO/BrentSTEPif_Posik.tgz'), (0.030406713485717773, '2014-others/BFGS-scipy_Baudis.tgz'), (0.030468225479125977, '2022/EXPLO2_Huntsman.tgz'), (0.030916213989257812, '2014-others/Nelder-Mead-scipy_Baudis.tgz'), (0.03265690803527832, '2009/FULLNEWUOA_ros_noiseless.tgz'), (0.03302502632141113, '2015-GECCO/STEPrr_Posik.tgz'), (0.03306078910827637, '2013/fmincon_pal_noiseless.tgz'), (0.0354008674621582, '2009/POEMS_kubalik_noiseless.tgz'), (0.03591799736022949, '2014-others/EG50-cocopf_Baudis.tgz'), (0.037081003189086914, '2009/VNS_garcia-martinez_noiseless.tgz'), (0.03709673881530762, '2009/LSstep_posik_noiseless.tgz'), (0.03750801086425781, '2009/GLOBAL_pal_noiseless.tgz'), (0.03844404220581055, '2009/NELDERDOERR_doerr_noiseless.tgz'), (0.03876090049743652, '2018/BFGS-P-StPt_Blelly.tgz'), (0.038812875747680664, '2010/1komma4ser_brockhoff_noiseless.tgz'), (0.03983306884765625, '2013/OQNLP_pal_noiseless.tgz'), (0.040451765060424805, '2010/1komma2ser_brockhoff_noiseless.tgz'), (0.040609121322631836, '2015-CEC/SOO_Derbel.tgz'), (0.04061603546142578, '2010/AVGNEWUOA_ros_noiseless.tgz'), (0.0406947135925293, '2015-CEC/R-DE-10e5_Tanabe.tgz'), (0.0409398078918457, '2009/Rosenbrock_posik_noiseless.tgz'), (0.04110598564147949, '2009/LSfminbnd_posik_noiseless.tgz'), (0.04118919372558594, '2020/SLSQP-11-scipy_Hansen.tgz'), (0.04147911071777344, '2010/1komma4mirser_brockhoff_noiseless.tgz'), (0.04174208641052246, '2018/BFGS-P-range_Blelly.tgz'), (0.04188704490661621, '2013/simplex_pal_noiseless.tgz'), (0.04216814041137695, '2010/1plus1_brockhoff_noiseless.tgz'), (0.04218769073486328, '2013/U-zero_tran_noiseless.tgz'), (0.042207956314086914, '2018/BFGS-P-Instances_Blelly.tgz'), (0.0423741340637207, '2018/BFGS-P-09_Blelly.tgz'), (0.042472124099731445, '2018/BFGS-M-17_Blelly.tgz'), (0.04251289367675781, '2013/HILL_holtschulte_noiseless.tgz'), (0.042685747146606445, '2015-CEC/R-SHADE-10e5_Tanabe.tgz'), (0.04295778274536133, '2019/RS-20-initIn0_Brockhoff.tgz'), (0.04296588897705078, '2014-others/CG-Fletcher-Reeves-scipy_Baudis.tgz'), (0.04347109794616699, '2010/1plus2mirser_brockhoff_noiseless.tgz'), (0.04350996017456055, '2010/1komma2mirser_brockhoff_noiseless.tgz'), (0.043557167053222656, '2013/U-DCN_tran_noiseless.tgz'), (0.04409909248352051, '2019/RS-10-initIn0_Brockhoff.tgz'), (0.044216156005859375, '2013/P-zero_tran_noiseless.tgz'), (0.04434776306152344, '2019/BFGS-scipy-2019_Varelas.tgz'), (0.04472708702087402, '2010/1komma4mir_brockhoff_noiseless.tgz'), (0.044951677322387695, '2010/1komma4_brockhoff_noiseless.tgz'), (0.045018911361694336, '2015-GECCO/RF5-CMAES_Bajer_2013instances.tgz'), (0.04510688781738281, '2019/RS-2_Brockhoff.tgz'), (0.045120954513549805, '2021/SHADE-LM_Okulewicz.tgz'), (0.04529380798339844, '2015-CEC/RL-SHADE-10e5_Tanabe.tgz'), (0.04554486274719238, '2019/RS-5-initIn0_Brockhoff.tgz'), (0.0456540584564209, '2019/RS-6-initIn0_Brockhoff.tgz'), (0.045686960220336914, '2019/RS-1-initIn0_Brockhoff.tgz'), (0.045744895935058594, '2019/RS-0p5-initIn0_Brockhoff.tgz'), (0.04582405090332031, '2019/RS-4-initIn0_Brockhoff.tgz'), (0.04587507247924805, '2015-GECCO/IPOPCMAv3p61_Bajer_2013instances.tgz'), (0.046054840087890625, '2009/PSO_Bounds_el-abd_noiseless.tgz'), (0.04608511924743652, '2019/RS-3_Brockhoff.tgz'), (0.046324968338012695, '2020/SLSQP+lq-CMA-ES_Hansen.tgz'), (0.04648113250732422, '2010/oPOEMS_kubalic_noiseless.tgz'), (0.046694278717041016, '2013/tany_liao_noiseless.tgz'), (0.046732187271118164, '2019/RS-2-initIn0_Brockhoff.tgz'), (0.046767234802246094, '2019/RS-1_Brockhoff.tgz'), (0.04684710502624512, '2013/P-DCN_tran_noiseless.tgz'), (0.04709291458129883, '2010/1komma2mir_brockhoff_noiseless.tgz'), (0.04717898368835449, '2015-GECCO/RF1-CMAES_Bajer.tgz'), (0.04731106758117676, '2019/RS-3-initIn0_Brockhoff.tgz'), (0.04732394218444824, '2013/CGA-ring16_holtschulte_noiseless.tgz'), (0.04764533042907715, '2019/RS-20_Brockhoff.tgz'), (0.0476531982421875, '2019/RS-0p5_Brockhoff.tgz'), (0.047966957092285156, '2013/texp_liao_noiseless.tgz'), (0.04897308349609375, '2019/RS-10_Brockhoff.tgz'), (0.04912900924682617, '2013/GA-100_holtschulte_noiseless.tgz'), (0.04959988594055176, '2015-GECCO/RF5-CMAES_Bajer.tgz'), (0.04968404769897461, '2019/SLSQP-scipy-2019_Varelas.tgz'), (0.049989938735961914, '2009/EDA-PSO_el-abd_noiseless.tgz'), (0.050421953201293945, '2009/MA-LS-CHAIN_molina_noiseless.tgz'), (0.050520896911621094, '2012/JADEctpb_posik_noiseless.tgz'), (0.0505976676940918, '2017/Ord-H-DTS-CMA-ES_Pitra.tgz'), (0.05083894729614258, '2015-GECCO/GP5-CMAES_Bajer.tgz'), (0.05100584030151367, '2010/ABC_elabd_noiseless.tgz'), (0.05101490020751953, '2018-others/uDE-rand_Vodopija.tgz'), (0.05139517784118652, '2013/CGA-grid100_holtschulte_noiseless.tgz'), (0.0514531135559082, '2009/DE-PSO_garcia-nieto_noiseless.tgz'), (0.051782846450805664, '2009/BAYEDA_gallagher_noiseless.tgz'), (0.05186295509338379, '2015-GECCO/IPOPCMAv3p61_Bajer.tgz'), (0.05201077461242676, '2022/oMads-2N_Dahito.tgz'), (0.05203700065612793, '2012/DE-ROLL_voglis_noiseless.tgz'), (0.05204892158508301, '2012/DE-BFGS_voglis_noiseless.tgz'), (0.0523219108581543, '2015-GECCO/GP1-CMAES_Bajer.tgz'), (0.05245685577392578, '2012/DE-AUTO_voglis_noiseless.tgz'), (0.05257296562194824, '2012/DEb_posik_noiseless.tgz'), (0.05301833152770996, '2013/DE_Pal_pal_noiseless.tgz'), (0.05357098579406738, '2009/NEWUOA_ros_noiseless.tgz'), (0.05371594429016113, '2012/DE-SIMPLEX_voglis_noiseless.tgz'), (0.05388998985290527, '2013/PRCGA_sawyerr_noiseless.tgz'), (0.053997039794921875, '2010/pPOEMS_kubalic_noiseless.tgz'), (0.053999900817871094, '2019/TNC-scipy-2019_Varelas.tgz'), (0.05400705337524414, '2022/oMads-Neg_Dahito.tgz'), (0.0541079044342041, '2012/JADEb_posik_noiseless.tgz'), (0.054203033447265625, '2016/PSAaSmD-CMA-ES_Nishida.tgz'), (0.054259300231933594, '2019/GNN-CMA-ES_Faury.tgz'), (0.05443406105041504, '2012/MVDE_melo_noiseless.tgz'), (0.054534912109375, '2013/MLSL_pal_noiseless.tgz'), (0.05459403991699219, '2018-others/uDE-best_Vodopija.tgz'), (0.05460214614868164, '2010/PM-AdapSS-DE_fialho_noiseless.tgz'), (0.054798126220703125, '2013/lmm-CMA-ES_auger_noiseless.tgz'), (0.05483508110046387, '2018/DE-rand_Vodopija.tgz'), (0.0552830696105957, '2013/HCMA_loshchilov_noiseless.tgz'), (0.05607771873474121, '2009/IPOP-SEP-CMA-ES_ros_noiseless.tgz'), (0.05611896514892578, '2013/IP-10DDr_liao_noiseless.tgz'), (0.05648064613342285, '2018-others/uDE-ttb_Vodopija.tgz'), (0.056603193283081055, '2012/DEAE_posik_noiseless.tgz'), (0.05674481391906738, '2017-others/RANDOMSEARCH-4p5-1e7D_Brockhoff.tgz'), (0.0567779541015625, '2017-others/RANDOMSEARCH-4-1e7D_Brockhoff.tgz'), (0.05681729316711426, '2012/DE_posik_noiseless.tgz'), (0.05693697929382324, '2019/L-BFGS-B-scipy-2019_Varelas.tgz'), (0.05713629722595215, '2016/PSAaLmC-CMA-ES_Nishida.tgz'), (0.057168006896972656, '2017/DTS-CMA-ES_Pitra.tgz'), (0.0572969913482666, '2012/PSO-BFGS_voglis_noiseless.tgz'), (0.057456016540527344, '2010/DEuniform_fialho_noiseless.tgz'), (0.05765891075134277, '2018-others/uBBDE_Vodopija.tgz'), (0.05797314643859863, '2017-others/RANDOMSEARCH-5-1e7D_Brockhoff.tgz'), (0.05811572074890137, '2018-others/DTS-CMA-ES_005-2pop_v26_1model_Bajer.tgz'), (0.059165000915527344, '2009/iAMALGAM_bosman_noiseless.tgz'), (0.05923199653625488, '2010/SPSA_finck_noiseless.tgz'), (0.059293270111083984, '2009/G3PCX_posik_noiseless.tgz'), (0.059307098388671875, '2018/DE-best_Vodopija.tgz'), (0.05979514122009277, '2012/DBRCGA_chuang_noiseless.tgz'), (0.059964895248413086, '2010/DE-F-AUC_fialho_noiseless.tgz'), (0.060585975646972656, '2019/Powell-scipy-2019_Varelas.tgz'), (0.060658931732177734, '2015-GECCO/STEPifeg_Posik.tgz'), (0.06068921089172363, '2018/BBDE-best_Vodopija.tgz'), (0.06143307685852051, '2018-others/uBBDE-ttb_Vodopija.tgz'), (0.06154203414916992, '2018/BBDE-N_Vodopija.tgz'), (0.06184029579162598, '2010/MOS_torre_noiseless.tgz'), (0.061975717544555664, '2017/SSEABC_Aydin.tgz'), (0.061985015869140625, '2016/PSAaSmC-CMA-ES_Nishida.tgz'), (0.062406063079833984, '2018/BBDE_Vodopija.tgz'), (0.06264925003051758, '2010/RCGA_tran_noiseless.tgz'), (0.0627279281616211, '2013/IPOP400D_auger_noiseless.tgz'), (0.0628058910369873, '2020/HE-ES_Glasmachers.tgz'), (0.06284904479980469, '2009/PSO_el-abd_noiseless.tgz'), (0.06318116188049316, '2021/SHADE-LM-POP4-to-10_Okulewicz.tgz'), (0.06351590156555176, '2015-GECCO/CMA-MSR_Atamna.tgz'), (0.06356501579284668, '2009/ONEFIFTH_auger_noiseless.tgz'), (0.06368279457092285, '2015-GECCO/LHD-2xDefault-MATSuMoTo_Brockhoff.tgz'), (0.0639181137084961, '2018-others/uBBDE-best_Vodopija.tgz'), (0.0642540454864502, '2012/BIPOPsaACM_loshchilov_noiseless.tgz'), (0.06457710266113281, '2009/NELDER_hansen_noiseless.tgz'), (0.06504201889038086, '2013/IP_liao_noiseless.tgz'), (0.06599211692810059, '2012/JADE_posik_noiseless.tgz'), (0.06605005264282227, '2013/CGA-ring100_holtschulte_noiseless.tgz'), (0.06630992889404297, '2009/CMA-ESPLUSSEL_auger_noiseless.tgz'), (0.06746125221252441, '2018/PSA-CMA-ES_Nishida.tgz'), (0.06797504425048828, '2009/BFGS_ros_noiseless.tgz'), (0.06840991973876953, '2009/AMALGAM_bosman_noiseless.tgz'), (0.06946992874145508, '2013/IP-500_liao_noiseless.tgz'), (0.06999015808105469, '2019/DE-scipy-2019_Varelas.tgz'), (0.0705251693725586, '2013/fminunc_pal_noiseless.tgz'), (0.07097434997558594, '2009/BIPOP-CMA-ES_hansen_noiseless.tgz'), (0.07122206687927246, '2010/IPOP-ACTCMA-ES_ros_noiseless.tgz'), (0.07155776023864746, '2009/Cauchy-EDA_posik_noiseless.tgz'), (0.07194709777832031, '2012/CMA_brockhoff_noiseless.tgz'), (0.07218623161315918, '2012/NBIPOPaCMA_loshchilov_noiseless.tgz'), (0.07223391532897949, '2012/CMAm_brockhoff_noiseless.tgz'), (0.07321405410766602, '2013/CGA-grid16_holtschulte_noiseless.tgz'), (0.07332086563110352, '2019/Adaptive-Two-Mode_Bodner.tgz'), (0.07374405860900879, '2020/CMA-ES-2019_Hansen.tgz'), (0.07455706596374512, '2018-others/uBBDE-N_Vodopija.tgz'), (0.0757589340209961, '2012/CMAmah_brockhoff_noiseless.tgz'), (0.0769813060760498, '2012/DEctpb_posik_noiseless.tgz'), (0.07725882530212402, '2010/1komma2_brockhoff_noiseless.tgz'), (0.07784223556518555, '2010/NBC-CMA_preuss_noiseless.tgz'), (0.07849431037902832, '2018/CMAES-APOP-Var1_Nguyen.tgz'), (0.08042287826538086, '2012/SNES_schaul_noiseless.tgz'), (0.08051609992980957, '2017/Ord-Q-DTS-CMA-ES_Pitra.tgz'), (0.08080387115478516, '2009/RANDOMSEARCH_auger_noiseless.tgz'), (0.08089876174926758, '2009/DASA_korosec_noiseless.tgz'), (0.08278203010559082, '2019/IPOP-CMA-ES-2019_Faury.tgz'), (0.08316183090209961, '2015-GECCO/RF1-CMAES_Bajer_2013instances.tgz'), (0.08382010459899902, '2019/Nelder-Mead-scipy-2019_Varelas.tgz'), (0.08390998840332031, '2017/Ord-N-DTS-CMA-ES_Pitra.tgz'), (0.08416581153869629, '2019/CG-scipy-2019_Varelas.tgz'), (0.08483028411865234, '2017/KL-IPOP-CMA-ES_Yamaguchi.tgz'), (0.08529305458068848, '2012/ACOR_liao_noiseless.tgz'), (0.08662986755371094, '2009/ALPS_hornby_noiseless.tgz'), (0.0881800651550293, '2015-GECCO/CMA-CSA_Atamna.tgz'), (0.08857011795043945, '2014-others/CMA-ES-python_Baudis.tgz'), (0.08897519111633301, '2015-GECCO/GP1-CMAES_Bajer_2013instances.tgz'), (0.08899211883544922, '2013/BIPOP-aCMA-STEP_loshchilov_noiseless.tgz'), (0.0898900032043457, '2019/adapt-Nelder-Mead-scipy-2019_Varelas.tgz'), (0.09134221076965332, '2018/DE-ttb_Vodopija.tgz'), (0.09272909164428711, '2019/RS-6_Brockhoff.tgz'), (0.0941319465637207, '2016/PSAaLmD-CMA-ES_Nishida.tgz'), (0.09463810920715332, '2019/COBYLA-scipy-2019_Varelas.tgz'), (0.09487795829772949, '2013/CMAES_Hutter_hutter_noiseless.tgz'), (0.09647417068481445, '2018/BBDE-ttb_Vodopija.tgz'), (0.09653806686401367, '2013/HMLSL_pal_noiseless.tgz'), (0.09803295135498047, '2013/MEMPSODE_voglis_noiseless.tgz'), (0.09825420379638672, '2015-GECCO/CMA-TPA_Atamna.tgz'), (0.09978866577148438, '2015-GECCO/GP5-CMAES_Bajer_2013instances.tgz'), (0.1013479232788086, '2020/lq-CMA-ES_Hansen.tgz'), (0.10264706611633301, '2018/PSA-CMA-ESwRS_Nishida.tgz'), (0.10499978065490723, '2009/GA_nicolau_noiseless.tgz'), (0.10649490356445312, '2012/BIPOPaCMA_loshchilov_noiseless.tgz'), (0.10952901840209961, '2012/IPOPsaACM_loshchilov_noiseless.tgz'), (0.11119890213012695, '2018/CMAES-APOP-Var3_Nguyen.tgz'), (0.11581897735595703, '2017/KL-Restart-CMA-ES_Yamaguchi.tgz'), (0.11582326889038086, '2012/CMAma_brockhoff_noiseless.tgz'), (0.11684989929199219, '2012/NIPOPaCMA_loshchilov_noiseless.tgz'), (0.11746001243591309, '2018/CMAES-APOP-Var2_Nguyen.tgz'), (0.12190008163452148, '2012/CMAa_brockhoff_noiseless.tgz'), (0.12219691276550293, '2012/CMAmh_brockhoff_noiseless.tgz'), (0.12264490127563477, '2013/BIPOP-saACM-k_loshchilov_noiseless.tgz'), (0.12273526191711426, '2017/CMAES-APOP_Nguyen.tgz'), (0.12284708023071289, '2010/CMAEGS_finck_noiseless.tgz'), (0.13104891777038574, '2010/IPOP-CMA-ES_ros_noiseless.tgz'), (0.13462615013122559, '2017/KL-BIPOP-CMA-ES_Yamaguchi.tgz'), (0.14850497245788574, '2012/CMAES_posik_noiseless.tgz')] ```
brockho commented 4 years ago

Many thanks for the data, @nikohansen.

Just a remark: the longest-to-load data sets (SMAC-BBOB, lmm-CMA-ES, SPSA, BIPOPaCMA and DASA) have each instance stored in a single file. Looks like that matters a lot when loading.

nikohansen commented 4 years ago

And that is on a Macbook Pro, that is, even without a spinning disk.