"failing_datasets": ["LL1_penn_fudan_pedestrian", "31_urbansound", "LL0_186_braziltourism", "6_86_com_DBLP", "60_jester", "LL1_EDGELIST_net_nomination_seed", "LL1_net_nomination_seed", "49_facebook", "59_umls", "LL1_736_stock_market", "1491_one_hundred_plants_margin_clust", "56_sunspots", "DS01876", "LL1_336_MS_Geolife_transport_mode_prediction", "313_spectrometer", "6_70_com_amazon"]

[liangmuxin]

In seed_dataset

[liangmuxin]

    "Traceback (most recent call last):\n",
    "  File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate\n",
    "    self._push_candidate(result, candidate, candidate_cache)\n",
    "  File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate\n",
    "    value = result['test_metrics'][0]['value'] if result else None\n",
    "KeyError: 0\n",

[RqS]

If we comment out the cache thing in the previous run. We are actually failing less.

The result of that run is inside 0726-nocache. Note the problem of thetext metric file written by that run is not yet fixed, we are not writing the best for that run.

On Fri, Jul 27, 2018 at 10:32 AM pada notifications@github.com wrote:

In seed_dataset

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/usc-isi-i2/dsbox-ta2/issues/147, or mute the thread https://github.com/notifications/unsubscribe-auth/AGPpwbLWXg3kIR0bDUWJamvMjDPmEv2mks5uK06tgaJpZM4VkQno .

[liangmuxin]

OK. got it

[proska]

Would you please give me more information on when this failure happens? Have changed the location in which we store test metric results?

[liangmuxin]

This failure is in cached version

[liangmuxin]

` ubuntu@ip-172-31-42-171:/dsbox_efs/runs/Muxin_7_27_seed/failure$ python3 -m json.tool ./56_sunspots_out_summary.json { "out_file": "/dsbox_efs/runs/seed/56_sunspots/supporting_files/logs/out.txt", "traces": { "Trace_3": [ "128 Traceback (most recent call last):", "129 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "130 self._push_candidate(result, candidate, candidate_cache)", "131 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "132 value = result['test_metrics'][0]['value'] if result else None", "133 KeyError: 0", "" ], "Trace_7": [ "280 Traceback (most recent call last):", "281 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "282 self._push_candidate(result, candidate, candidate_cache)", "283 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "284 value = result['test_metrics'][0]['value'] if result else None", "285 KeyError: 0", "" ], "Trace_4": [ "170 Traceback (most recent call last):", "171 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "172 self._push_candidate(result, candidate, candidate_cache)", "173 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "174 value = result['test_metrics'][0]['value'] if result else None", "175 KeyError: 0", "" ], "Trace_6": [ "246 Traceback (most recent call last):", "247 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "248 self._push_candidate(result, candidate, candidate_cache)", "249 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "250 value = result['test_metrics'][0]['value'] if result else None", "251 KeyError: 0", "" ], "Trace_2": [ "94 Traceback (most recent call last):", "95 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "96 self._push_candidate(result, candidate, candidate_cache)", "97 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "98 value = result['test_metrics'][0]['value'] if result else None", "99 KeyError: 0", "" ], "Trace_12": [ "474 Traceback (most recent call last):", "475 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "476 self._push_candidate(result, candidate, candidate_cache)", "477 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "478 value = result['test_metrics'][0]['value'] if result else None", "479 KeyError: 0", "" ], "Trace_8": [ "322 Traceback (most recent call last):", "323 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "324 self._push_candidate(result, candidate, candidate_cache)", "325 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "326 value = result['test_metrics'][0]['value'] if result else None", "327 KeyError: 0", "" ], "Trace_1": [ "60 Traceback (most recent call last):", "61 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "62 self._push_candidate(result, candidate, candidate_cache)", "63 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "64 value = result['test_metrics'][0]['value'] if result else None", "65 KeyError: 0", "" ], "Trace_5": [ "204 Traceback (most recent call last):", "205 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "206 self._push_candidate(result, candidate, candidate_cache)", "207 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "208 value = result['test_metrics'][0]['value'] if result else None", "209 KeyError: 0", "" ], "Trace_9": [ "356 Traceback (most recent call last):", "357 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "358 self._push_candidate(result, candidate, candidate_cache)", "359 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "360 value = result['test_metrics'][0]['value'] if result else None", "361 KeyError: 0", "" ], "Trace_13": [ "508 Traceback (most recent call last):", "509 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "510 self._push_candidate(result, candidate, candidate_cache)", "511 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "512 value = result['test_metrics'][0]['value'] if result else None", "513 KeyError: 0", "" ], "Trace_10": [ "398 Traceback (most recent call last):", "399 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "400 self._push_candidate(result, candidate, candidate_cache)", "401 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "402 value = result['test_metrics'][0]['value'] if result else None", "403 KeyError: 0", "" ], "Trace_0": [ "37 Traceback (most recent call last):", "38 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "39 self._push_candidate(result, candidate, candidate_cache)", "40 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "41 value = result['test_metrics'][0]['value'] if result else None", "42 KeyError: 0", "" ], "Trace_11": [ "432 Traceback (most recent call last):", "433 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate", "434 self._push_candidate(result, candidate, candidate_cache)", "435 File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate", "436 value = result['test_metrics'][0]['value'] if result else None", "437 KeyError: 0", "" ] }, "last_few_lines": [ "--------------------\n", "[INFO] Worker started, id: <_MainProcess(MainProcess, started)> , True\n", "[INFO] Will use normal train-test mode ( n = 1 ) to choose best primitives.\n", "[INFO] Hit@cache: ('d3m.primitives.dsbox.Denormalize', -2341604804659697464)\n", "[INFO] Hit@cache: ('d3m.primitives.datasets.DatasetToDataFrame', -8515364664864555464)\n", "[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', 2821400207488637039)\n", "[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', -4642014206590463659)\n", "[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKExtraTreesRegressor', 2012231772836712242)\n", "[INFO] Testing finish.!!!\n", "The dataset no need to split of split failed, will not train again.\n", "!!!! No test_dataset1\n", "{'cross_validation_metrics': [],\n", " 'fitted_pipeline': <dsbox.pipeline.fitted_pipeline.FittedPipeline object at 0x7f3f0711b198>,\n", " 'test_metrics': {'metric': 'rootMeanSquaredError', 'value': 0.0},\n", " 'total_runtime': 0.4031658172607422,\n", " 'training_metrics': {'metric': 'rootMeanSquaredError', 'value': 0.0}}\n", "!!!!\n", "Traceback (most recent call last):\n", " File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 382, in setup_initial_candidate\n", " self._push_candidate(result, candidate, candidate_cache)\n", " File \"/user_opt/dsbox/dsbox-ta2/python/dsbox/template/search.py\", line 132, in _push_candidate\n", " value = result['test_metrics'][0]['value'] if result else None\n", "KeyError: 0\n", "[ERROR] Initial Pipeline failed, Trying a random pipeline ...\n", "{'denormalize_step': {'hyperparameters': {},\n", " 'primitive': 'd3m.primitives.dsbox.Denormalize'},\n", " 'extract_attribute_step': {'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Attribute',)},\n", " 'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes'},\n", " 'extract_target_step': {'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/TrueTarget',)},\n", " 'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes'},\n", " 'model_step': {'hyperparameters': {},\n", " 'primitive': 'd3m.primitives.sklearn_wrap.SKExtraTreesRegressor'},\n", " 'to_dataframe_step': {'hyperparameters': {},\n", " 'primitive': 'd3m.primitives.datasets.DatasetToDataFrame'}}\n", "--------------------\n", "[ERROR] - dsbox.controller.controller - [ERROR] not candidate!\n", "[INFO] - dsbox.controller.controller - \u001b[30m\u001b[42m[INFO] report: -inf\n", "[INFO] - dsbox.controller.controller - \u001b[30m\u001b[42m[INFO] UCT updated: [37.0281143399872, 125.21909989321647]\n", "[INFO] - dsbox.controller.controller - \u001b[30m\u001b[42m[INFO] cache size: 6, candidates: 0\n", "++++++++++\n", "[INFO] - dsbox.controller.controller - [INFO] Start test function\n", "[INFO] - dsbox.controller.controller - [INFO] No specified pipeline ID found, will load the latest crated pipeline.\n", "The following pipeline file will be loaded:\n", "/dsbox_efs/runs/seed/56_sunspots/pipelines/f088a0d3-c156-49f4-b9a1-93ebd3c096f1.json\n", "[INFO] - dsbox.controller.controller - [INFO] Pipeline load finished\n", "[INFO] - dsbox.controller.controller - [INFO] testing data\n", "[INFO] - dsbox.controller.controller - [INFO] Finished: prediction results saving finished\n", "[INFO] - dsbox.controller.controller - [INFO] The prediction results is stored at: /dsbox_efs/runs/seed/56_sunspots/predictions/f088a0d3-c156-49f4-b9a1-93ebd3c096f1\n", "[INFO] Testing Done\n", "[INFO] The time used for running program is 112.46378200000001 seconds.\n" ], "debugs": {}, "templates": [ "14 [INFO] Template choices:", "15 Template ' SRI_Mean_Baseline_Template ' has been added to template base.", "16 ** Template ' Default_Time_Series_Forcasting_Template ' has been added to template base.", "" ], "errors": { "[ERROR] - dsbox.controller.controller - [ERROR] not candidate!\n": 7, "[ERROR] Initial Pipeline failed, Trying a random pipeline ...\n": 14 } } ubuntu@ip-172-31-42-171:/dsbox_efs/runs/Muxin_7_27_seed/failure$

` One significant failure is like this

[kyao]

Looks like we are not storing the a list of metrics in some cases. I will fix the bug.

[RqS]

KeyError fixed