Tool does not stop if framework dependencies can not be installed

[PGijsbers]

Encountered this today, the (old) fixed numpy versions can not be installed on my M1 Mac. After failing to install the dependencies, the script carries on and simply reports a missing results with error:

Processing results for randomforest.test.test.local.20220714T132006
Summing up scores for current run:
             id task fold    framework constraint      metric  duration      seed                                                                                                                                                                                                     info
openml.org/t/59 iris    0 RandomForest       test neg_logloss       3.3 714307014 CalledProcessError: Command '/Users/pietergijsbers/repositories/automlbenchmark/frameworks/RandomForest/venv/bin/python -W ignore /Users/pietergijsbers/repositories/automlbenchmark/frameworks/RandomF…

a closer look at the error reveals:

Traceback (most recent call last):

  File "/Users/pietergijsbers/repositories/automlbenchmark/frameworks/RandomForest/exec.py", line 11, in <module>

    import psutil

ModuleNotFoundError: No module named 'psutil'

which is still not very informative.

I think we should immediately abort the experiment when we notice that there is a failure in the installation process. This way we can communicate the error better and faster.

[sebhrusen]

could you try to add

set -e

on top of frameworks/shared/setup.sh ? it should ensure that every framework's setup will exit on the first failed command

[sebhrusen]

there's probably additional work needed to get this aborted correctly in containers and in AWS though... (setup done separately there)

[PGijsbers]

That works (regardless of whether it's shared or framework-specific dependencies that fail to install). It gives the output below, we should add some logging around https://github.com/openml/automlbenchmark/blob/7729f0dcaa37b2d5c48ca7c63b66ddd5f729e786/amlb/benchmark.py#L123-L126

to make it easier to parse for new users. Seems like a good enough fix for the issue (in local mode).

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Command '['/Users/pietergijsbers/repositories/automlbenchmark/frameworks/RandomForest/setup.sh', 'stable']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "runbenchmark.py", line 182, in <module>
    bench.setup(amlb.SetupMode[args.setup])
  File "/Users/pietergijsbers/repositories/automlbenchmark/amlb/benchmark.py", line 123, in setup
    self.framework_module.setup(*self.framework_def.setup_args,
  File "/Users/pietergijsbers/repositories/automlbenchmark/frameworks/RandomForest/__init__.py", line 7, in setup
    call_script_in_same_dir(__file__, "setup.sh", *args, **kwargs)
  File "/Users/pietergijsbers/repositories/automlbenchmark/amlb/utils/process.py", line 259, in call_script_in_same_dir
    return run_script(script_path, *args, **kwargs)
  File "/Users/pietergijsbers/repositories/automlbenchmark/amlb/utils/process.py", line 253, in run_script
    return run_cmd(script_path, *args, **kwargs)
  File "/Users/pietergijsbers/repositories/automlbenchmark/amlb/utils/process.py", line 247, in run_cmd
    raise e
  File "/Users/pietergijsbers/repositories/automlbenchmark/amlb/utils/process.py", line 221, in run_cmd
    completed = run_subprocess(str_cmd if params.shell else full_cmd,
  File "/Users/pietergijsbers/repositories/automlbenchmark/amlb/utils/process.py", line 77, in run_subprocess
    raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/Users/pietergijsbers/repositories/automlbenchmark/frameworks/RandomForest/setup.sh', 'stable']' returned non-zero exit status 1.