search

usc-isi-i2 / dsbox-ta2

The DSBox TA2 component
MIT License
11 stars 6 forks source link

Some datasets don't return a cross validation value #81

Closed serbanstan closed 6 years ago

serbanstan commented 6 years ago

When running python ta2-search /nas/home/stan/dsbox/runs2/config-seed/26_radon_seed_config.json the above error pops up in the stack trace. 

(dsbox-devel-710) [stan@dsbox01 python]$ python ta2-search /nas/home/stan/dsbox/runs2/config-seed/26_radon_seed_config.json
Namespace(configuration_file='/nas/home/stan/dsbox/runs2/config-seed/26_radon_seed_config.json', cpus=-1, debug=False, output_prefix=None, timeout=-1)
Using configuation:
{'cpus': '10',
 'dataset_schema': '/nfs1/dsbox-repo/data/datasets/seed_datasets_current/26_radon_seed/26_radon_seed_dataset/datasetDoc.json',
 'executables_root': '/nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed/executables',
 'pipeline_logs_root': '/nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed/logs',
 'problem_root': '/nfs1/dsbox-repo/data/datasets/seed_datasets_current/26_radon_seed/26_radon_seed_problem',
 'problem_schema': '/nfs1/dsbox-repo/data/datasets/seed_datasets_current/26_radon_seed/26_radon_seed_problem/problemDoc.json',
 'ram': '10Gi',
 'saved_pipeline_ID': '',
 'saving_folder_loc': '/nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed',
 'temp_storage_root': '/nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed/temp',
 'timeout': 9,
 'training_data_root': '/nfs1/dsbox-repo/data/datasets/seed_datasets_current/26_radon_seed/26_radon_seed_dataset'}
[INFO] No test data config found! Will split the data.
[INFO] - dsbox.controller.controller - Top level output directory: /nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed
[INFO] Succesfully parsed test data
{'structural_type': <class 'd3m.container.pandas.DataFrame'>, 'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Table', 'https://metadata.datadrivendiscovery.org/types/DatasetEntryPoint'), 'dimension': {'name': 'rows', 'semantic_types': ('https://metadata.datadrivendiscovery.org/types/TabularRow',), 'length': 736}}
{'dimension': <FrozenOrderedDict OrderedDict([('name', 'rows'), ('semantic_types', ('https://metadata.datadrivendiscovery.org/types/TabularRow',)), ('length', 736)])>,
 'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Table',
                    'https://metadata.datadrivendiscovery.org/types/DatasetEntryPoint'),
 'structural_type': <class 'd3m.container.pandas.DataFrame'>}
{'structural_type': <class 'd3m.container.pandas.DataFrame'>, 'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Table', 'https://metadata.datadrivendiscovery.org/types/DatasetEntryPoint'), 'dimension': {'name': 'rows', 'semantic_types': ('https://metadata.datadrivendiscovery.org/types/TabularRow',), 'length': 183}}
{'dimension': <FrozenOrderedDict OrderedDict([('name', 'rows'), ('semantic_types', ('https://metadata.datadrivendiscovery.org/types/TabularRow',)), ('length', 183)])>,
 'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Table',
                    'https://metadata.datadrivendiscovery.org/types/DatasetEntryPoint'),
 'structural_type': <class 'd3m.container.pandas.DataFrame'>}
[INFO] Template choices:
Template ' Default_regression_template ' has been added to template base.
[INFO] Template 0:Default_regression_template Selected. UCT:[100.0]
[INFO] Worker started, id: <_MainProcess(MainProcess, started)>
/nfs1/dsbox-repo/stan/miniconda/envs/dsbox-devel-710/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
[INFO] Push@cache: ('d3m.primitives.dsbox.Denormalize', 587284634364934697)
/nfs1/dsbox-repo/stan/miniconda/envs/dsbox-devel-710/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
[INFO] Push@cache: ('d3m.primitives.datasets.DatasetToDataFrame', 587284634364934697)
[INFO] Push@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', 977600147975090788)
[INFO] Push@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', -1966569371269715752)
[INFO] Push@cache: ('d3m.primitives.dsbox.Profiler', 131337691952325357)
/nfs1/dsbox-repo/stan/dsbox-profiling/dsbox/datapreprocessing/profiler/dependencies/date_extractor.py:408: UserWarning: DateExtractor: Failed to set timezone as America/Los_Angeles. Catch offset must be a timedelta representing a whole number of minutes, not datetime.timedelta(-1, 58022).
  warn('DateExtractor: Failed to set timezone as ' + str(self.default_tz) + '. Catch ' + str(e))
/nfs1/dsbox-repo/stan/miniconda/envs/dsbox-devel-710/lib/python3.6/re.py:212: FutureWarning: split() requires a non-empty pattern match.
  return _compile(pattern, flags).split(string, maxsplit)
[INFO] Push@cache: ('d3m.primitives.dsbox.CleaningFeaturizer', 131337691952325357)
[INFO] Push@cache: ('d3m.primitives.dsbox.CorexText', -7031971290585165178)
[INFO] Push@cache: ('d3m.primitives.dsbox.Encoder', -7031971290585165178)
[INFO] Push@cache: ('d3m.primitives.sklearn_wrap.SKImputer', -6931622793779467230)
[INFO] Push@cache: ('d3m.primitives.sklearn_wrap.SKARDRegression', -757582090584309155)
******************
[INFO] Writing results
{'fitted_pipeline': <dsbox.pipeline.fitted_pipeline.FittedPipeline object at 0x7fe753c705f8>, 'training_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3689347674245992}], 'cross_validation_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.37602206929480897, 'values': [0.45196794796525436, 0.6903608008599122, 0.5287042975078422, 0.2943527117932023, 0.3448464219578232, 0.2712624762647298, 0.31806732987922665, 0.29137569219893433, 0.31950582085577206, 0.24977719366539228], 'targets': []}], 'test_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3576875175951625}]}
{'denormalize_step': {'primitive': 'd3m.primitives.dsbox.Denormalize', 'hyperparameters': {}}, 'to_dataframe_step': {'primitive': 'd3m.primitives.datasets.DatasetToDataFrame', 'hyperparameters': {}}, 'extract_attribute_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Attribute',)}}, 'profiler_step': {'primitive': 'd3m.primitives.dsbox.Profiler', 'hyperparameters': {}}, 'clean_step': {'primitive': 'd3m.primitives.dsbox.CleaningFeaturizer', 'hyperparameters': {}}, 'corex_step': {'primitive': 'd3m.primitives.dsbox.CorexText', 'hyperparameters': {}}, 'encoder_step': {'primitive': 'd3m.primitives.dsbox.Encoder', 'hyperparameters': {}}, 'impute_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKImputer', 'hyperparameters': {}}, 'extract_target_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Target', 'https://metadata.datadrivendiscovery.org/types/SuggestedTarget')}}, 'model_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKARDRegression', 'hyperparameters': {}}} 0.3576875175951625
Training rootMeanSquaredError = 0.3689347674245992
CV rootMeanSquaredError = 0.37602206929480897
Test rootMeanSquaredError = 0.3576875175951625
******************
[INFO] Saving training results in /nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed.txt
[INFO] report: 0.3576875175951625
[INFO] UCT updated: [18.921517921357168]
[INFO] cache size: 10
[INFO] Template 0:Default_regression_template Selected. UCT:[18.921517921357168]
[INFO] Worker started, id: <_MainProcess(MainProcess, started)>
[INFO] Hit@cache: ('d3m.primitives.dsbox.Denormalize', 587284634364934697)
[INFO] Hit@cache: ('d3m.primitives.datasets.DatasetToDataFrame', 587284634364934697)
[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', 977600147975090788)
[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', -1966569371269715752)
[INFO] Hit@cache: ('d3m.primitives.dsbox.Profiler', 131337691952325357)
[INFO] Hit@cache: ('d3m.primitives.dsbox.CleaningFeaturizer', 131337691952325357)
[INFO] Hit@cache: ('d3m.primitives.dsbox.CorexText', -7031971290585165178)
[INFO] Hit@cache: ('d3m.primitives.dsbox.Encoder', -7031971290585165178)
[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKImputer', -6931622793779467230)
[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKARDRegression', -757582090584309155)
******************
[INFO] Writing results
{'fitted_pipeline': <dsbox.pipeline.fitted_pipeline.FittedPipeline object at 0x7fe50c61c4a8>, 'training_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3689347674245992}], 'cross_validation_metrics': [], 'test_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3576875175951625}]}
{'denormalize_step': {'primitive': 'd3m.primitives.dsbox.Denormalize', 'hyperparameters': {}}, 'to_dataframe_step': {'primitive': 'd3m.primitives.datasets.DatasetToDataFrame', 'hyperparameters': {}}, 'extract_attribute_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Attribute',)}}, 'profiler_step': {'primitive': 'd3m.primitives.dsbox.Profiler', 'hyperparameters': {}}, 'clean_step': {'primitive': 'd3m.primitives.dsbox.CleaningFeaturizer', 'hyperparameters': {}}, 'corex_step': {'primitive': 'd3m.primitives.dsbox.CorexText', 'hyperparameters': {}}, 'encoder_step': {'primitive': 'd3m.primitives.dsbox.Encoder', 'hyperparameters': {}}, 'impute_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKImputer', 'hyperparameters': {}}, 'extract_target_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Target', 'https://metadata.datadrivendiscovery.org/types/SuggestedTarget')}}, 'model_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKARDRegression', 'hyperparameters': {}}} 0.3576875175951625
Training rootMeanSquaredError = 0.3689347674245992
Test rootMeanSquaredError = 0.3576875175951625
******************
[INFO] Saving training results in /nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed.txt
Traceback (most recent call last):
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 409, in search_template
    f.write(str(candidate.data['cross_validation_metrics'][0]['value']) + "\n")
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 531, in train
    template, candidate=self.exec_history.iloc[idx]['candidate'], cache=cache)
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 414, in search_template
    '[ERROR] Save training results Failed!')
d3m.exceptions.NotSupportedError: [ERROR] Save training results Failed!
[INFO] Template 0:Default_regression_template Selected. UCT:[18.921517921357168]
[INFO] Worker started, id: <_MainProcess(MainProcess, started)>
[INFO] Hit@cache: ('d3m.primitives.dsbox.Denormalize', 587284634364934697)
[INFO] Hit@cache: ('d3m.primitives.datasets.DatasetToDataFrame', 587284634364934697)
[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', 977600147975090788)
[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', -1966569371269715752)
[INFO] Hit@cache: ('d3m.primitives.dsbox.Profiler', 131337691952325357)
[INFO] Hit@cache: ('d3m.primitives.dsbox.CleaningFeaturizer', 131337691952325357)
[INFO] Hit@cache: ('d3m.primitives.dsbox.CorexText', -7031971290585165178)
[INFO] Hit@cache: ('d3m.primitives.dsbox.Encoder', -7031971290585165178)
[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKImputer', -6931622793779467230)
[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKARDRegression', -757582090584309155)
******************
[INFO] Writing results
{'fitted_pipeline': <dsbox.pipeline.fitted_pipeline.FittedPipeline object at 0x7fe53c0f0b70>, 'training_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3689347674245992}], 'cross_validation_metrics': [], 'test_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3576875175951625}]}
{'denormalize_step': {'primitive': 'd3m.primitives.dsbox.Denormalize', 'hyperparameters': {}}, 'to_dataframe_step': {'primitive': 'd3m.primitives.datasets.DatasetToDataFrame', 'hyperparameters': {}}, 'extract_attribute_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Attribute',)}}, 'profiler_step': {'primitive': 'd3m.primitives.dsbox.Profiler', 'hyperparameters': {}}, 'clean_step': {'primitive': 'd3m.primitives.dsbox.CleaningFeaturizer', 'hyperparameters': {}}, 'corex_step': {'primitive': 'd3m.primitives.dsbox.CorexText', 'hyperparameters': {}}, 'encoder_step': {'primitive': 'd3m.primitives.dsbox.Encoder', 'hyperparameters': {}}, 'impute_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKImputer', 'hyperparameters': {}}, 'extract_target_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Target', 'https://metadata.datadrivendiscovery.org/types/SuggestedTarget')}}, 'model_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKARDRegression', 'hyperparameters': {}}} 0.3576875175951625
Training rootMeanSquaredError = 0.3689347674245992
Test rootMeanSquaredError = 0.3576875175951625
******************
[INFO] Saving training results in /nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed.txt
Traceback (most recent call last):
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 409, in search_template
    f.write(str(candidate.data['cross_validation_metrics'][0]['value']) + "\n")
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 531, in train
    template, candidate=self.exec_history.iloc[idx]['candidate'], cache=cache)
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 414, in search_template
    '[ERROR] Save training results Failed!')
d3m.exceptions.NotSupportedError: [ERROR] Save training results Failed!
[INFO] Template 0:Default_regression_template Selected. UCT:[18.921517921357168]
[INFO] Worker started, id: <_MainProcess(MainProcess, started)>
[INFO] Hit@cache: ('d3m.primitives.dsbox.Denormalize', 587284634364934697)
[INFO] Hit@cache: ('d3m.primitives.datasets.DatasetToDataFrame', 587284634364934697)
[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', 977600147975090788)
[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', -1966569371269715752)
[INFO] Hit@cache: ('d3m.primitives.dsbox.Profiler', 131337691952325357)
[INFO] Hit@cache: ('d3m.primitives.dsbox.CleaningFeaturizer', 131337691952325357)
[INFO] Hit@cache: ('d3m.primitives.dsbox.CorexText', -7031971290585165178)
[INFO] Hit@cache: ('d3m.primitives.dsbox.Encoder', -7031971290585165178)
[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKImputer', -6931622793779467230)
[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKARDRegression', -757582090584309155)
******************
[INFO] Writing results
{'fitted_pipeline': <dsbox.pipeline.fitted_pipeline.FittedPipeline object at 0x7fe51475f9b0>, 'training_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3689347674245992}], 'cross_validation_metrics': [], 'test_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3576875175951625}]}
{'denormalize_step': {'primitive': 'd3m.primitives.dsbox.Denormalize', 'hyperparameters': {}}, 'to_dataframe_step': {'primitive': 'd3m.primitives.datasets.DatasetToDataFrame', 'hyperparameters': {}}, 'extract_attribute_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Attribute',)}}, 'profiler_step': {'primitive': 'd3m.primitives.dsbox.Profiler', 'hyperparameters': {}}, 'clean_step': {'primitive': 'd3m.primitives.dsbox.CleaningFeaturizer', 'hyperparameters': {}}, 'corex_step': {'primitive': 'd3m.primitives.dsbox.CorexText', 'hyperparameters': {}}, 'encoder_step': {'primitive': 'd3m.primitives.dsbox.Encoder', 'hyperparameters': {}}, 'impute_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKImputer', 'hyperparameters': {}}, 'extract_target_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Target', 'https://metadata.datadrivendiscovery.org/types/SuggestedTarget')}}, 'model_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKARDRegression', 'hyperparameters': {}}} 0.3576875175951625
Training rootMeanSquaredError = 0.3689347674245992
Test rootMeanSquaredError = 0.3576875175951625
******************
[INFO] Saving training results in /nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed.txt
Traceback (most recent call last):
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 409, in search_template
    f.write(str(candidate.data['cross_validation_metrics'][0]['value']) + "\n")
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 531, in train
    template, candidate=self.exec_history.iloc[idx]['candidate'], cache=cache)
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 414, in search_template
    '[ERROR] Save training results Failed!')
d3m.exceptions.NotSupportedError: [ERROR] Save training results Failed!
[INFO] Template 0:Default_regression_template Selected. UCT:[18.921517921357168]
[INFO] Worker started, id: <_MainProcess(MainProcess, started)>
[INFO] Hit@cache: ('d3m.primitives.dsbox.Denormalize', 587284634364934697)
[INFO] Hit@cache: ('d3m.primitives.datasets.DatasetToDataFrame', 587284634364934697)
[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', 977600147975090788)
[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', -1966569371269715752)
[INFO] Hit@cache: ('d3m.primitives.dsbox.Profiler', 131337691952325357)
[INFO] Hit@cache: ('d3m.primitives.dsbox.CleaningFeaturizer', 131337691952325357)
[INFO] Hit@cache: ('d3m.primitives.dsbox.CorexText', -7031971290585165178)
[INFO] Hit@cache: ('d3m.primitives.dsbox.Encoder', -7031971290585165178)
[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKImputer', -6931622793779467230)
[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKARDRegression', -757582090584309155)
******************
[INFO] Writing results
{'fitted_pipeline': <dsbox.pipeline.fitted_pipeline.FittedPipeline object at 0x7fe53c0c6940>, 'training_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3689347674245992}], 'cross_validation_metrics': [], 'test_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3576875175951625}]}
{'denormalize_step': {'primitive': 'd3m.primitives.dsbox.Denormalize', 'hyperparameters': {}}, 'to_dataframe_step': {'primitive': 'd3m.primitives.datasets.DatasetToDataFrame', 'hyperparameters': {}}, 'extract_attribute_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Attribute',)}}, 'profiler_step': {'primitive': 'd3m.primitives.dsbox.Profiler', 'hyperparameters': {}}, 'clean_step': {'primitive': 'd3m.primitives.dsbox.CleaningFeaturizer', 'hyperparameters': {}}, 'corex_step': {'primitive': 'd3m.primitives.dsbox.CorexText', 'hyperparameters': {}}, 'encoder_step': {'primitive': 'd3m.primitives.dsbox.Encoder', 'hyperparameters': {}}, 'impute_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKImputer', 'hyperparameters': {}}, 'extract_target_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Target', 'https://metadata.datadrivendiscovery.org/types/SuggestedTarget')}}, 'model_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKARDRegression', 'hyperparameters': {}}} 0.3576875175951625
Training rootMeanSquaredError = 0.3689347674245992
Test rootMeanSquaredError = 0.3576875175951625
******************
[INFO] Saving training results in /nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed.txt
Traceback (most recent call last):
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 409, in search_template
    f.write(str(candidate.data['cross_validation_metrics'][0]['value']) + "\n")
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 531, in train
    template, candidate=self.exec_history.iloc[idx]['candidate'], cache=cache)
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 414, in search_template
    '[ERROR] Save training results Failed!')
d3m.exceptions.NotSupportedError: [ERROR] Save training results Failed!
[INFO] Template 0:Default_regression_template Selected. UCT:[18.921517921357168]
[INFO] Worker started, id: <_MainProcess(MainProcess, started)>
[INFO] Hit@cache: ('d3m.primitives.dsbox.Denormalize', 587284634364934697)
[INFO] Hit@cache: ('d3m.primitives.datasets.DatasetToDataFrame', 587284634364934697)
[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', 977600147975090788)
[INFO] Hit@cache: ('d3m.primitives.data.ExtractColumnsBySemanticTypes', -1966569371269715752)
[INFO] Hit@cache: ('d3m.primitives.dsbox.Profiler', 131337691952325357)
[INFO] Hit@cache: ('d3m.primitives.dsbox.CleaningFeaturizer', 131337691952325357)
[INFO] Hit@cache: ('d3m.primitives.dsbox.CorexText', -7031971290585165178)
[INFO] Hit@cache: ('d3m.primitives.dsbox.Encoder', -7031971290585165178)
[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKImputer', -6931622793779467230)
[INFO] Hit@cache: ('d3m.primitives.sklearn_wrap.SKARDRegression', -757582090584309155)
******************
[INFO] Writing results
{'fitted_pipeline': <dsbox.pipeline.fitted_pipeline.FittedPipeline object at 0x7fe75afeb278>, 'training_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3689347674245992}], 'cross_validation_metrics': [], 'test_metrics': [{'metric': 'rootMeanSquaredError', 'value': 0.3576875175951625}]}
{'denormalize_step': {'primitive': 'd3m.primitives.dsbox.Denormalize', 'hyperparameters': {}}, 'to_dataframe_step': {'primitive': 'd3m.primitives.datasets.DatasetToDataFrame', 'hyperparameters': {}}, 'extract_attribute_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Attribute',)}}, 'profiler_step': {'primitive': 'd3m.primitives.dsbox.Profiler', 'hyperparameters': {}}, 'clean_step': {'primitive': 'd3m.primitives.dsbox.CleaningFeaturizer', 'hyperparameters': {}}, 'corex_step': {'primitive': 'd3m.primitives.dsbox.CorexText', 'hyperparameters': {}}, 'encoder_step': {'primitive': 'd3m.primitives.dsbox.Encoder', 'hyperparameters': {}}, 'impute_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKImputer', 'hyperparameters': {}}, 'extract_target_step': {'primitive': 'd3m.primitives.data.ExtractColumnsBySemanticTypes', 'hyperparameters': {'semantic_types': ('https://metadata.datadrivendiscovery.org/types/Target', 'https://metadata.datadrivendiscovery.org/types/SuggestedTarget')}}, 'model_step': {'primitive': 'd3m.primitives.sklearn_wrap.SKARDRegression', 'hyperparameters': {}}} 0.3576875175951625
Training rootMeanSquaredError = 0.3689347674245992
Test rootMeanSquaredError = 0.3576875175951625
******************
[INFO] Saving training results in /nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed.txt
Traceback (most recent call last):
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 409, in search_template
    f.write(str(candidate.data['cross_validation_metrics'][0]['value']) + "\n")
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 531, in train
    template, candidate=self.exec_history.iloc[idx]['candidate'], cache=cache)
  File "/nfs1/dsbox-repo/stan/dsbox-ta2/python/dsbox/controller/controller.py", line 414, in search_template
    '[ERROR] Save training results Failed!')
d3m.exceptions.NotSupportedError: [ERROR] Save training results Failed!
*+*+*+*+*+*+*+*+*+*+
[INFO] Start test function
[INFO] No specified pipeline ID found, will load the latest crated pipeline.
The following pipeline file will be loaded:
/nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed/pipelines/377264fe-f820-4f03-b748-ee7fd551cef3.json
[INFO] Pipeline load finished
[INFO] testing data:
[INFO] Finished: prediction results saving finished
[INFO] The prediction results is stored at:  /nfs1/dsbox-repo/stan/dsbox-ta2/python/output/26_radon_seed/predictions/377264fe-f820-4f03-b748-ee7fd551cef3
[INFO] Testing Done
[INFO] The time used for running program is 2261.52 seconds.
serbanstan commented 6 years ago

Similar behavior for running python ta2-search /nas/home/stan/dsbox/runs2/config-ll0/LL0_531_boston_config.json

serbanstan commented 6 years ago

The error seems to occur when cross validation is not performed.

serbanstan commented 6 years ago

It seems that after the system reads in a template it doesn't perform cross validation once more. Pushing a fix and closing issue.