search

AIworx-Labs / chocolate

A fully decentralized hyperparameter optimization framework
http://chocolate.readthedocs.io
BSD 3-Clause "New" or "Revised" License
121 stars 41 forks source link

Disappearance of Search Space #34

Closed jlevy44 closed 5 years ago

jlevy44 commented 5 years ago

Hi there,

I'm optimizing using Bayes. For some reason, after 50 iterations, my search space disappeared (no job has successfully run since) and degraded from: 

43|0.530283581420485|0.594507535272973|0.0956133794822555|0.183174425952106|0.324464855577522|0.18321976289463|0.447247428760473|0.494862238036925|0.594291236564883|0.13365381153193|0.621074537333421|0.985043314092192|0.673105557564046|0.596371641856695|0.239961433973444|0.875564411418133|0.68697429593277|0.816370003321507|0.522079915259265|0.299775777059482|0.317955724207088|0.254591595114779|42|0.287920729412387
44|0.0171569182510461|0.523320898649622|0.69997076052281|0.727165793748551|0.936893202909011|0.832087118258521|0.817707066616694|0.043945601495308|0.172307370813185|0.604412420534715|0.698271229512247|0.714185614077469|0.268339034524112|0.864922001768117|0.11006805138535|0.157309044576634|0.234912451561038|0.603408986782483|0.167489595204554|0.924554238220891|0.584603069868852|0.257061923179376|43|
45|0.380566059855455|0.933064088981559|0.933714630646887|0.0744530249769997|0.388265600889486|0.944433675981516|0.0569716535292384|0.622625741906597|0.824726859177882|0.286916684031226|0.0461006760876181|0.367887978131571|0.143214333369612|0.608785397554655|0.355639695538943|0.443461587061983|0.0395272464760753|0.432397938687212|0.916889202326262|0.788378929495989|0.966677421598842|0.203127582215792|44|
46|0.253411368785518|0.0777907741815148|0.876500079587662|0.321163552224698|0.261745976080779|0.567233304826923|0.614005484432338|0.561732707323025|0.303126185751605|0.108269578069825|0.742713447522607|0.668533849457911|0.0944285609757195|0.0105585222315285|0.0661138279019602|0.408501621538934|0.0442106972122589|0.555647870999506|0.964949584291862|0.92392717661155|0.971449433224484|0.487447863638309|45|0.310609024949372
47|0.0298706617302658|0.240152555751103|0.090250446753992|0.579542641511725|0.950262983355336|0.511885716113274|0.644236640943123|0.581846488931566|0.713375881433834|0.667854374379905|0.460655785871363|0.565685697090619|0.297101361852681|0.333863237438923|0.914182987219078|0.781396972753748|0.789599736757716|0.329955374513446|0.19818814113627|0.502965587267886|0.4976784227606|0.0491609756261113|46|
48|0.00885569460517077|0.118405123983955|0.557886102430734|0.0952691739938708|0.08662104381455|0.536737574589037|0.400385295217095|0.627107524687692|0.810342690307004|0.663091784708001|0.38270787596168|0.0886371654368058|0.406951667476664|0.349650275208202|0.786681378478588|0.908111690347794|0.843372779162317|0.151473114416399|0.100487967891708|0.817909681798678|0.895492049701731|0.184908076406235|47|0.254542940994725
49|0.0283817259954816|0.34723398483473|0.0874992440489671|0.264315861727955|0.444942900253486|0.21283369509211|0.00847858108027655|0.707910239646895|0.563197022676976|0.200059736335963|0.984806237389409|0.253012615556054|0.192273064061049|0.93755194497991|0.0270019825026756|0.642695827742226|0.154103034542351|0.910466695738626|0.44607513890935|0.983591745982839|0.269964783615805|0.363716220251394|48|

into:

52|0.999999|0.999999|0.0|0.999999|0.0|0.666662114854188|0.0|0.999999|0.999999|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|51|
53|0.0|0.999999|0.0|0.0|0.237397505392564|0.333353925188991|0.999999|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.99480679509398|52|
54|0.999999|0.999999|0.999999|0.0|0.0|0.330118936455318|0.0|0.0|0.999999|0.0|0.0|0.0|0.666662174539248|0.0|0.999999|0.0|0.0|0.0|0.0|0.0|0.999999|0.999999|53|
55|0.0|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|0.0|0.999999|0.999999|0.999999|0.999999|0.0|0.0|0.0|0.0|0.999999|0.0|0.999999|0.999999|0.999999|54|
56|0.999999|0.0|0.999999|0.999999|0.0|0.152723969423227|0.999999|0.0|0.0|0.0|0.0|0.0|0.333335688983986|0.0|0.0|0.999999|0.0|0.0|0.0|0.0|0.999999|0.0|55|
57|0.999999|0.0|0.999999|0.0|0.999999|0.38312686338175|0.0|0.0526761582384528|0.0|0.0|0.0|0.0|0.333323312054163|0.0|0.900502413181104|0.0|0.0|0.0|0.0|0.967615307563784|0.0|0.0|56|
58|0.999999|0.0|0.999999|0.999999|0.0|0.666661372626746|0.0|0.0|0.0|0.999999|0.0|0.999999|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.999999|0.0|57|
59|0.999999|0.0|0.0|0.999999|0.999999|0.666688133500132|0.0|0.0|0.0|0.0|0.0|0.0|0.719368432365075|0.0|0.999999|0.999999|0.0|0.0|0.0|0.0|0.999999|0.999999|58|
60|0.999999|0.0|0.971209972397585|0.999999|0.999999|0.666666492771594|0.999999|0.0|0.0|0.0|0.0|0.0|0.941648128350468|0.0|0.0|0.0|0.0|0.0|0.999999|0.999999|0.999999|0.999999|59|
61|0.0|0.999999|0.0|0.999999|0.999999|0.333278684727739|0.0|0.0|0.0|0.0|0.0|0.0|0.241231958085866|0.0|0.999999|0.999999|0.0|0.0|0.0|0.0|0.999999|0.999999|60|
62|0.0|0.999999|0.0|0.0|0.0|0.0|0.0|0.999999|0.999999|0.0|0.0|0.0|0.33341907011072|0.999999|0.0|0.0|0.0|0.0|0.0|0.999999|0.999999|0.0|61|
63|0.0|0.0|0.999999|0.999999|0.0|0.872050888368733|0.0|0.0|0.0|0.0|0.0|0.0|0.666671709977091|0.0|0.0|0.0|0.0|0.999999|0.0|0.999999|0.0|0.999999|62|
64|0.999999|0.0|0.999999|0.999999|0.0|0.666666435941093|0.0|0.0|0.0|0.999999|0.0|0.0|0.999999|0.0|0.0|0.0|0.999999|0.999999|0.999999|0.0|0.0|0.0|63|
65|0.0|0.999999|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|-2.77555756156289e-17|0.999999|64|
66|0.0|0.999999|0.999999|0.999999|0.999999|0.354959525945384|0.999999|0.0|0.0|0.0|0.0|0.0|0.333398139070798|0.999999|0.0|0.0|0.0|0.0|0.0|0.0|0.999999|0.999999|65|
67|0.0|0.999999|0.930855008554571|0.0|0.0414986078171875|0.666666692338916|0.0|0.999999|0.999999|0.0|0.0|0.0|0.999999|0.0|0.0|0.0|0.999999|0.58272672066777|0.0|0.0|0.938542037908141|0.84414554050701|66|
68|0.0|0.999999|0.999999|0.0|0.999999|0.0|0.999999|0.0|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|0.999999|0.0|0.0|0.0|0.999999|0.0|0.999999|67|
69|0.0|0.0|0.999999|0.0|0.999999|0.333322400996303|0.0|0.999999|0.999999|0.0|0.0|0.0|0.391332714852607|0.999999|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.999999|68|
70|0.999999|0.651042976081047|0.0|0.999999|0.999999|0.70726506425955|0.0|0.0|0.0|0.74419997053051|0.999999|0.999999|0.666692486118609|0.0|0.0|0.0|0.999999|0.999999|0.0|0.0|0.0|0.999999|69|
71|0.999999|0.0|0.999999|-1.38777878078145e-17|0.999999|0.999999|0.0|0.0|0.0|0.0|0.999999|0.0|0.999999|0.0|0.0|0.0|0.999999|0.999999|0.0|0.999999|0.999999|0.0|70|
72|0.999999|0.999999|0.999999|0.0|0.0|0.333417588213429|0.0|0.0|0.999999|0.0|0.0|0.0|0.676891573223725|0.0|0.0|0.0|0.0|0.916211729683123|0.999999|0.0|0.999999|0.999999|71|
73|0.0|0.999999|0.0|0.0|0.999999|0.33341080482224|0.999999|0.0|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|0.0|0.0|72|
74|0.999999|0.0|0.999999|0.999999|0.0|0.999999|0.0|0.0|0.0|0.999999|0.0|0.0|0.999999|0.0|0.0|0.0|0.999999|0.999999|0.999999|0.0|0.0|0.0|73|
75|0.999999|0.999999|0.0|0.0|0.999999|1.38777878078145e-17|0.0|0.999999|0.0|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|0.0|0.0|0.0|0.0|0.999999|0.999999|74|
76|0.0|0.0|0.0|0.999999|0.0|0.35731087627765|0.0|0.999999|0.0|0.0|0.0|0.0|0.667162676980283|0.763568451269825|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|75|
77|0.0|0.999999|0.0|0.999999|0.0|0.0|0.999999|0.0|0.0|0.0|0.0|0.0|0.665914393042038|0.999999|0.0|0.0|0.0|0.0|0.0|0.999999|0.0|0.999999|76|
78|0.999999|0.0|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|0.0|0.999999|0.999999|0.0|77|
79|0.0|0.999999|0.999999|0.999999|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.0|0.999999|0.0|0.0|0.0|0.999999|0.999999|0.999999|0.999999|0.0|0.999999|78|

Note that a lot of losses are not returned, this is due to the nature of the algorithm which can fail sometimes under memory constraints. But these final failures seem to be due to a bad search space that causes failures for reasons not pertaining to memory. Help much appreciated!