search

openml / automlbenchmark

OpenML AutoML Benchmarking Framework
https://openml.github.io/automlbenchmark
MIT License
399 stars 132 forks source link

[GAMA] Error: NoResultError:'ignore' #489

Closed cynthiamaia closed 2 years ago

cynthiamaia commented 2 years ago

Hi, I'm trying to run some experiments on GAMA and it's giving the following error: NoResultError:'ignore' . I'm using branch: master and the command is: python runbenchmark.py gama example test

PGijsbers commented 2 years ago

Hi! Can you post the full output? Which OS are you using? I am currently having some issues reproducing the result.

cynthiamaia commented 2 years ago

Yes! Error: [ERROR] [amlb.benchmark:09:57:43.137] 'ignore' Traceback (most recent call last): File "/home/cmm3/Downloads/automlbenchmark-2.0.5/amlb/benchmark.py", line 543, in run meta_result = self.benchmark.framework_module.run(self._dataset, task_config) File "/home/cmm3/Downloads/automlbenchmark-2.0.5/frameworks/GAMA/init.py", line 31, in run input_data=data, dataset=dataset, config=config, options=options) File "/home/cmm3/Downloads/automlbenchmark-2.0.5/frameworks/shared/caller.py", line 142, in run_in_venv raise NoResultError(res.error_message) amlb.results.NoResultError: 'ignore'

I'm using Linux!

PGijsbers commented 2 years ago

I still did not manage to reproduce (on MacOS, though I require :latest there because PyPI GAMA does not currently work on Apple silicon). Can you give the full outlog, from command to error? (See my example below) And, if that's not an issue, try gama:latest instead? On my end, I am trying to reproduce it on Docker but experience some issue building the image.

(venv) pietergijsbers@TUE027303 automlbenchmark % python runbenchmark.py gama:latest example test       
/Users/pietergijsbers/repositories/automlbenchmark/venv/lib/python3.8/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
Running benchmark `gama:latest` on `example` framework in `local` mode.
Loading frameworks definitions from ['/Users/pietergijsbers/repositories/automlbenchmark/resources/frameworks.yaml'].
Loading benchmark constraint definitions from ['/Users/pietergijsbers/repositories/automlbenchmark/resources/constraints.yaml'].
Loading benchmark definitions from /Users/pietergijsbers/repositories/automlbenchmark/resources/benchmarks/example.yaml.
[MONITORING] [Python [2118]] CPU Utilization: 40.4%
[MONITORING] [Python [2118]] Memory Usage: 56.3%

------------------------------------------------
Starting job local.example.test.credit-g.0.GAMA.
[MONITORING] [Python [2118]] Disk Usage: 15.8%
Assigning 4 cores (total=10) for new task credit-g.
Assigning 5116 MB (total=16384 MB) for new credit-g task.
Running task credit-g on framework GAMA with config:
TaskConfig({'framework': 'GAMA', 'framework_params': {}, 'framework_version': '21.0.1.dev', 'type': 'classification', 'name': 'credit-g', 'fold': 0, 'metric': 'auc', 'metrics': ['auc', 'logloss', 'acc', 'balacc'], 'seed': 676333333, 'job_timeout_seconds': 1200, 'max_runtime_seconds': 600, 'cores': 4, 'max_mem_size_mb': 5116, 'min_vol_size_mb': -1, 'input_dir': '/Users/pietergijsbers/.openml', 'output_dir': '/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905', 'output_predictions_file': '/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/predictions/credit-g/0/predictions.csv', 'ext': {}, 'type_': 'binary', 'output_metadata_file': '/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/predictions/credit-g/0/metadata.json'})
Running cmd `/Users/pietergijsbers/repositories/automlbenchmark/frameworks/GAMA/venv/bin/python -W ignore /Users/pietergijsbers/repositories/automlbenchmark/frameworks/GAMA/exec.py`
INFO:__main__:
**** GAMA [v21.0.1.dev] ****
INFO:__main__:sklearn == 1.1.2
INFO:__main__:category_encoders == 2.5.0
INFO:__main__:Running GAMA with a maximum time of 600s on 4 cores, optimizing roc_auc.
INFO:gama.gama:Using GAMA version 21.0.1.dev.
INFO:gama.gama:INIT:GamaClassifier(scoring=roc_auc,regularize_length=True,max_pipeline_length=None,random_state=676333333,max_total_time=600,max_eval_time=None,n_jobs=4,max_memory_mb=5116,verbosity=30,search=AsyncEA(),post_processing=BestFitPostProcessing(),output_directory=/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/logs/credit-g/0,store=logs)
INFO:gama.utilities.generic.timekeeper:START: preprocessing default
INFO:gama.utilities.generic.timekeeper:STOP: preprocessing default after 0.0487s.
INFO:gama.utilities.generic.timekeeper:START: search AsyncEA
INFO:gama.search_methods.async_ea:Starting EA with new population.
[MONITORING] [Python [2118]] CPU Utilization: 43.5%
[MONITORING] [Python [2118]] Memory Usage: 57.5%
[MONITORING] [Python [2118]] Disk Usage: 15.8%
[MONITORING] [Python [2118]] CPU Utilization: 51.2%
[MONITORING] [Python [2118]] Memory Usage: 57.4%
[MONITORING] [Python [2118]] Disk Usage: 16.3%
[MONITORING] [Python [2118]] CPU Utilization: 54.2%
[MONITORING] [Python [2118]] Memory Usage: 60.6%
[MONITORING] [Python [2118]] Disk Usage: 16.7%
[MONITORING] [Python [2118]] CPU Utilization: 44.8%
[MONITORING] [Python [2118]] Memory Usage: 60.3%
[MONITORING] [Python [2118]] Disk Usage: 16.7%
INFO:gama.gama:Search phase evaluated 8832 individuals.
INFO:gama.utilities.generic.timekeeper:STOP: search AsyncEA after 539.0852s.
INFO:gama.utilities.generic.timekeeper:START: postprocess BestFitPostProcessing
INFO:gama.utilities.generic.timekeeper:STOP: postprocess BestFitPostProcessing after 0.0139s.
INFO:__main__:Predicting on the test set.
WARNING:amlb.utils.process:Terminating process psutil.Process(pid=2125, name='Python', status='running', started='14:49:10').
WARNING:amlb.utils.process:Killing process psutil.Process(pid=2125, name='Python', status='running', started='14:49:10').

Predictions preview:
          bad      good predictions truth
0   0.028887  0.971113        good  good
1   0.010681  0.989319        good  good
2   0.203783  0.796217        good  good
3   0.554724  0.445276         bad  good
4   0.047109  0.952891        good  good
5   0.134945  0.865055        good  good
6   0.507826  0.492174         bad  good
7   0.365412  0.634588        good  good
8   0.238253  0.761747        good  good
9   0.076210  0.923790        good  good
10  0.030079  0.969921        good  good
11  0.159762  0.840238        good  good
12  0.029414  0.970586        good  good
13  0.008550  0.991450        good  good
14  0.045369  0.954631        good  good
15  0.051723  0.948277        good  good
16  0.284580  0.715420        good  good
17  0.075925  0.924075        good  good
18  0.530637  0.469363         bad  good
19  0.049900  0.950100        good  good

Predictions saved to `/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/predictions/credit-g/0/predictions.csv`.
Loading metadata from `/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/predictions/credit-g/0/metadata.json`.
Loading predictions from `/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/predictions/credit-g/0/predictions.csv`.
Metric scores: { 'acc': 0.77,
  'app_version': 'dev [master, db00ecf]',
  'auc': 0.8352380952380953,
  'balacc': 0.7023809523809523,
  'constraint': 'test',
  'duration': nan,
  'fold': 0,
  'framework': 'GAMA',
  'id': 'openml.org/t/31',
  'info': None,
  'logloss': 0.4558634141920356,
  'metric': 'auc',
  'mode': 'local',
  'models_count': 50,
  'params': '',
  'predict_duration': 0.010987997055053711,
  'result': 0.8352380952380953,
  'seed': 676333333,
  'task': 'credit-g',
  'training_duration': 539.1491913795471,
  'type': 'binary',
  'utc': '2022-08-15T12:58:14',
  'version': '21.0.1.dev'}
Job `local.example.test.credit-g.0.GAMA` executed in 548.971 seconds.

------------------------------------------------
Starting job local.example.test.credit-g.1.GAMA.
Assigning 4 cores (total=10) for new task credit-g.
Assigning 4952 MB (total=16384 MB) for new credit-g task.
Running task credit-g on framework GAMA with config:
TaskConfig({'framework': 'GAMA', 'framework_params': {}, 'framework_version': '21.0.1.dev', 'type': 'classification', 'name': 'credit-g', 'fold': 1, 'metric': 'auc', 'metrics': ['auc', 'logloss', 'acc', 'balacc'], 'seed': 676333334, 'job_timeout_seconds': 1200, 'max_runtime_seconds': 600, 'cores': 4, 'max_mem_size_mb': 4952, 'min_vol_size_mb': -1, 'input_dir': '/Users/pietergijsbers/.openml', 'output_dir': '/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905', 'output_predictions_file': '/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/predictions/credit-g/1/predictions.csv', 'ext': {}, 'type_': 'binary', 'output_metadata_file': '/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/predictions/credit-g/1/metadata.json'})
Running cmd `/Users/pietergijsbers/repositories/automlbenchmark/frameworks/GAMA/venv/bin/python -W ignore /Users/pietergijsbers/repositories/automlbenchmark/frameworks/GAMA/exec.py`
INFO:__main__:
**** GAMA [v21.0.1.dev] ****
INFO:__main__:sklearn == 1.1.2
INFO:__main__:category_encoders == 2.5.0
INFO:__main__:Running GAMA with a maximum time of 600s on 4 cores, optimizing roc_auc.
INFO:gama.gama:Using GAMA version 21.0.1.dev.
INFO:gama.gama:INIT:GamaClassifier(scoring=roc_auc,regularize_length=True,max_pipeline_length=None,random_state=676333334,max_total_time=600,max_eval_time=None,n_jobs=4,max_memory_mb=4952,verbosity=30,search=AsyncEA(),post_processing=BestFitPostProcessing(),output_directory=/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/logs/credit-g/1,store=logs)
INFO:gama.utilities.generic.timekeeper:START: preprocessing default
INFO:gama.utilities.generic.timekeeper:STOP: preprocessing default after 0.0409s.
INFO:gama.utilities.generic.timekeeper:START: search AsyncEA
INFO:gama.search_methods.async_ea:Starting EA with new population.
[MONITORING] [Python [2118]] CPU Utilization: 61.6%
[MONITORING] [Python [2118]] Memory Usage: 63.1%
[MONITORING] [Python [2118]] Disk Usage: 17.0%
[MONITORING] [Python [2118]] CPU Utilization: 54.1%
[MONITORING] [Python [2118]] Memory Usage: 63.7%
[MONITORING] [Python [2118]] Disk Usage: 17.1%
[MONITORING] [Python [2118]] CPU Utilization: 55.6%
[MONITORING] [Python [2118]] Memory Usage: 65.5%
[MONITORING] [Python [2118]] Disk Usage: 17.1%
[MONITORING] [Python [2118]] CPU Utilization: 54.0%
[MONITORING] [Python [2118]] Memory Usage: 65.3%
[MONITORING] [Python [2118]] Disk Usage: 17.1%
[MONITORING] [Python [2118]] CPU Utilization: 47.6%
[MONITORING] [Python [2118]] Memory Usage: 66.3%
[MONITORING] [Python [2118]] Disk Usage: 17.2%
INFO:gama.gama:Search phase evaluated 4300 individuals.
INFO:gama.utilities.generic.timekeeper:STOP: search AsyncEA after 539.1176s.
INFO:gama.utilities.generic.timekeeper:START: postprocess BestFitPostProcessing
INFO:gama.utilities.generic.timekeeper:STOP: postprocess BestFitPostProcessing after 0.0584s.
INFO:__main__:Predicting on the test set.
WARNING:amlb.utils.process:Terminating process psutil.Process(pid=4258, name='Python', status='running', started='14:58:17').
WARNING:amlb.utils.process:Killing process psutil.Process(pid=4258, name='Python', status='running', started='14:58:17').

Predictions preview:
          bad      good predictions truth
0   0.352740  0.647260        good  good
1   0.197036  0.802964        good  good
2   0.239601  0.760399        good  good
3   0.363618  0.636382        good  good
4   0.302108  0.697892        good  good
5   0.187865  0.812135        good  good
6   0.382321  0.617679        good  good
7   0.316587  0.683413        good  good
8   0.278292  0.721708        good  good
9   0.294422  0.705578        good  good
10  0.310809  0.689191        good  good
11  0.166840  0.833160        good  good
12  0.153164  0.846836        good  good
13  0.224196  0.775804        good  good
14  0.244106  0.755894        good  good
15  0.372425  0.627575        good  good
16  0.298907  0.701093        good  good
17  0.120793  0.879207        good  good
18  0.260522  0.739478        good  good
19  0.183240  0.816760        good  good

Predictions saved to `/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/predictions/credit-g/1/predictions.csv`.
Loading metadata from `/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/predictions/credit-g/1/metadata.json`.
Loading predictions from `/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/predictions/credit-g/1/predictions.csv`.
Metric scores: { 'acc': 0.71,
  'app_version': 'dev [master, db00ecf]',
  'auc': 0.7476190476190476,
  'balacc': 0.5166666666666667,
  'constraint': 'test',
  'duration': nan,
  'fold': 1,
  'framework': 'GAMA',
  'id': 'openml.org/t/31',
  'info': None,
  'logloss': 0.54075395717395,
  'metric': 'auc',
  'mode': 'local',
  'models_count': 50,
  'params': '',
  'predict_duration': 0.013705015182495117,
  'result': 0.7476190476190476,
  'seed': 676333334,
  'task': 'credit-g',
  'training_duration': 539.2179410457611,
  'type': 'binary',
  'utc': '2022-08-15T13:07:21',
  'version': '21.0.1.dev'}
Job `local.example.test.credit-g.1.GAMA` executed in 546.706 seconds.
All jobs executed in 1095.683 seconds.
[MONITORING] [Python [2118]] CPU Utilization: 42.1%
[MONITORING] [Python [2118]] Memory Usage: 62.8%
[MONITORING] [Python [2118]] Disk Usage: 17.2%
Processing results for gama.example.test.local.20220815T124905
Scores saved to `/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/scores/GAMA.benchmark_example.csv`.
Scores saved to `/Users/pietergijsbers/repositories/automlbenchmark/results/gama.example.test.local.20220815T124905/scores/results.csv`.
Scores saved to `/Users/pietergijsbers/repositories/automlbenchmark/results/results.csv`.
Summing up scores for current run:
             id     task fold framework constraint   result metric  duration      seed
openml.org/t/31 credit-g    0      GAMA       test 0.835238    auc     549.0 676333333
openml.org/t/31 credit-g    1      GAMA       test 0.747619    auc     546.7 676333334
cynthiamaia commented 2 years ago

Hi, yes! (venv) cmm3@george01:~/Downloads/automlbenchmark$ python runbenchmark.py gama:latest example test

Running benchmark `gama:latest` on `example` framework in `local` mode.
Loading frameworks definitions from ['/home/cmm3/Downloads/automlbenchmark/resources/frameworks.yaml'].
Loading benchmark constraint definitions from ['/home/cmm3/Downloads/automlbenchmark/resources/constraints.yaml'].
Loading benchmark definitions from /home/cmm3/Downloads/automlbenchmark/resources/benchmarks/example.yaml.

------------------------------------------------
Starting job local.example.test.credit-g.0.GAMA.
Assigning 16 cores (total=16) for new task credit-g.
Assigning 1058 MB (total=16033 MB) for new credit-g task.
[MONITORING] [python [23925]] CPU Utilization: 27.2%
[MONITORING] [python [23925]] Memory Usage: 93.4%
[MONITORING] [python [23925]] Disk Usage: 79.6%
Running task credit-g on framework GAMA with config:
TaskConfig({'framework': 'GAMA', 'framework_params': {}, 'framework_version': '21.0.1', 'type': 'classification', 'name': 'credit-g', 'fold': 0, 'metric': 'auc', 'metrics': ['auc', 'logloss', 'acc', 'balacc'], 'seed': 2110001463, 'job_timeout_seconds': 7200, 'max_runtime_seconds': 3600, 'cores': 16, 'max_mem_size_mb': 1058, 'min_vol_size_mb': -1, 'input_dir': '/home/cmm3/.cache/openml', 'output_dir': '/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953', 'output_predictions_file': '/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953/predictions/credit-g/0/predictions.csv', 'ext': {}, 'type_': 'binary', 'output_metadata_file': '/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953/predictions/credit-g/0/metadata.json'})
Running cmd `/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/bin/python -W ignore /home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/exec.py`
INFO:__main__:
**** GAMA [v21.0.1] ****
INFO:__main__:sklearn == 0.24.2
INFO:__main__:category_encoders == 2.5.0
INFO:__main__:Running GAMA with a maximum time of 3600s on 16 cores, optimizing roc_auc.
INFO:gama.gama:Using GAMA version 21.0.1.
INFO:gama.gama:INIT:GamaClassifier(scoring=roc_auc,regularize_length=True,max_pipeline_length=None,random_state=2110001463,max_total_time=3600,max_eval_time=None,n_jobs=16,max_memory_mb=1058,verbosity=30,search=AsyncEA(),post_processing=BestFitPostProcessing(),output_directory=/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953/logs/credit-g/0,store=logs,goal=simplicity)
INFO:gama.utilities.generic.timekeeper:START: preprocessing default
ERROR:frameworks.shared.callee:'ignore'
Traceback (most recent call last):

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/shared/callee.py", line 70, in call_run

    result = run_fn(ds, config)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/exec.py", line 79, in run

    gama_automl.fit(X_train, y_train)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/gama/GamaClassifier.py", line 134, in fit

    super().fit(x, y, *args, **kwargs)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/gama/gama.py", line 508, in fit

    x, is_classification

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/gama/utilities/preprocessing.py", line 63, in basic_encoding

    x_enc = encoding_pipeline.fit_transform(x, y=None)  # Is this dangerous?

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/sklearn/pipeline.py", line 387, in fit_transform

    return last_step.fit_transform(Xt, y, **fit_params_last_step)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/sklearn/base.py", line 699, in fit_transform

    return self.fit(X, **fit_params).transform(X)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/category_encoders/utils.py", line 295, in fit

    self._fit(X, y, **kwargs)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/category_encoders/one_hot.py", line 117, in _fit

    }[self.handle_missing]

KeyError: 'ignore'

'ignore'
Traceback (most recent call last):
  File "/home/cmm3/Downloads/automlbenchmark/amlb/benchmark.py", line 546, in run
    meta_result = self.benchmark.framework_module.run(self._dataset, task_config)
  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/__init__.py", line 31, in run
    input_data=data, dataset=dataset, config=config, options=options)
  File "/home/cmm3/Downloads/automlbenchmark/frameworks/shared/caller.py", line 150, in run_in_venv
    raise NoResultError(res.error_message)
amlb.results.NoResultError: 'ignore'
Loading metadata from `/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953/predictions/credit-g/0/metadata.json`.
Metric scores: { 'acc': nan,
  'app_version': 'dev [master, 88186b3]',
  'auc': nan,
  'balacc': nan,
  'constraint': 'test',
  'duration': nan,
  'fold': 0,
  'framework': 'GAMA',
  'id': 'openml.org/t/31',
  'info': "NoResultError: 'ignore'",
  'logloss': nan,
  'metric': 'auc',
  'mode': 'local',
  'models_count': nan,
  'params': '',
  'predict_duration': nan,
  'result': nan,
  'seed': 2110001463,
  'task': 'credit-g',
  'training_duration': nan,
  'type': 'binary',
  'utc': '2022-08-15T15:29:54',
  'version': '21.0.1'}
Job `local.example.test.credit-g.0.GAMA` executed in 0.894 seconds.

------------------------------------------------
Starting job local.example.test.credit-g.1.GAMA.
Assigning 16 cores (total=16) for new task credit-g.
Assigning 1047 MB (total=16033 MB) for new credit-g task.
Running task credit-g on framework GAMA with config:
TaskConfig({'framework': 'GAMA', 'framework_params': {}, 'framework_version': '21.0.1', 'type': 'classification', 'name': 'credit-g', 'fold': 1, 'metric': 'auc', 'metrics': ['auc', 'logloss', 'acc', 'balacc'], 'seed': 2110001464, 'job_timeout_seconds': 7200, 'max_runtime_seconds': 3600, 'cores': 16, 'max_mem_size_mb': 1047, 'min_vol_size_mb': -1, 'input_dir': '/home/cmm3/.cache/openml', 'output_dir': '/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953', 'output_predictions_file': '/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953/predictions/credit-g/1/predictions.csv', 'ext': {}, 'type_': 'binary', 'output_metadata_file': '/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953/predictions/credit-g/1/metadata.json'})
Running cmd `/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/bin/python -W ignore /home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/exec.py`
INFO:__main__:
**** GAMA [v21.0.1] ****
INFO:__main__:sklearn == 0.24.2
INFO:__main__:category_encoders == 2.5.0
INFO:__main__:Running GAMA with a maximum time of 3600s on 16 cores, optimizing roc_auc.
INFO:gama.gama:Using GAMA version 21.0.1.
INFO:gama.gama:INIT:GamaClassifier(scoring=roc_auc,regularize_length=True,max_pipeline_length=None,random_state=2110001464,max_total_time=3600,max_eval_time=None,n_jobs=16,max_memory_mb=1047,verbosity=30,search=AsyncEA(),post_processing=BestFitPostProcessing(),output_directory=/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953/logs/credit-g/1,store=logs,goal=simplicity)
INFO:gama.utilities.generic.timekeeper:START: preprocessing default
ERROR:frameworks.shared.callee:'ignore'
Traceback (most recent call last):

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/shared/callee.py", line 70, in call_run

    result = run_fn(ds, config)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/exec.py", line 79, in run

    gama_automl.fit(X_train, y_train)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/gama/GamaClassifier.py", line 134, in fit

    super().fit(x, y, *args, **kwargs)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/gama/gama.py", line 508, in fit

    x, is_classification

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/gama/utilities/preprocessing.py", line 63, in basic_encoding

    x_enc = encoding_pipeline.fit_transform(x, y=None)  # Is this dangerous?

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/sklearn/pipeline.py", line 387, in fit_transform

    return last_step.fit_transform(Xt, y, **fit_params_last_step)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/sklearn/base.py", line 699, in fit_transform

    return self.fit(X, **fit_params).transform(X)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/category_encoders/utils.py", line 295, in fit

    self._fit(X, y, **kwargs)

  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/venv/lib/python3.7/site-packages/category_encoders/one_hot.py", line 117, in _fit

    }[self.handle_missing]

KeyError: 'ignore'

'ignore'
Traceback (most recent call last):
  File "/home/cmm3/Downloads/automlbenchmark/amlb/benchmark.py", line 546, in run
    meta_result = self.benchmark.framework_module.run(self._dataset, task_config)
  File "/home/cmm3/Downloads/automlbenchmark/frameworks/GAMA/__init__.py", line 31, in run
    input_data=data, dataset=dataset, config=config, options=options)
  File "/home/cmm3/Downloads/automlbenchmark/frameworks/shared/caller.py", line 150, in run_in_venv
    raise NoResultError(res.error_message)
amlb.results.NoResultError: 'ignore'
Loading metadata from `/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953/predictions/credit-g/1/metadata.json`.
Metric scores: { 'acc': nan,
  'app_version': 'dev [master, 88186b3]',
  'auc': nan,
  'balacc': nan,
  'constraint': 'test',
  'duration': nan,
  'fold': 1,
  'framework': 'GAMA',
  'id': 'openml.org/t/31',
  'info': "NoResultError: 'ignore'",
  'logloss': nan,
  'metric': 'auc',
  'mode': 'local',
  'models_count': nan,
  'params': '',
  'predict_duration': nan,
  'result': nan,
  'seed': 2110001464,
  'task': 'credit-g',
  'training_duration': nan,
  'type': 'binary',
  'utc': '2022-08-15T15:29:55',
  'version': '21.0.1'}
Job `local.example.test.credit-g.1.GAMA` executed in 0.853 seconds.
All jobs executed in 1.757 seconds.
[MONITORING] [python [23925]] CPU Utilization: 15.9%
[MONITORING] [python [23925]] Memory Usage: 93.3%
[MONITORING] [python [23925]] Disk Usage: 79.6%
Processing results for gama.example.test.local.20220815T152953
Scores saved to `/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953/scores/GAMA.benchmark_example.csv`.
Scores saved to `/home/cmm3/Downloads/automlbenchmark/results/gama.example.test.local.20220815T152953/scores/results.csv`.
Scores saved to `/home/cmm3/Downloads/automlbenchmark/results/results.csv`.
Summing up scores for current run:
             id     task fold framework constraint metric  duration       seed                    info
openml.org/t/31 credit-g    0      GAMA       test    auc       0.9 2110001463 NoResultError: 'ignore'
openml.org/t/31 credit-g    1      GAMA       test    auc       0.9 2110001464 NoResultError: 'ignore'
PGijsbers commented 2 years ago

So it looks like you are running into this GAMA issue which is fixed on Github but not on PyPI. This would normally be resolved by using :latest but despite running :latest it looks like the previously installed stable version is used (from **** GAMA [v21.0.1] **** instead of **** GAMA [v21.0.1.dev] **** (please note that in this case .dev is indicates a later version)). I can't reproduce this issue - is it possible you already ran gama:latest before, a longer while ago? Then it could be using an outdated Github version. Using -s force with :latest in your command might alleviate this issue.

cynthiamaia commented 2 years ago

Thank you, it worked!