search

owkin / benchmark_flamby

This is an example to showcase how one can use benchopt in combination with FLamby in order to be able to create and share reproducible reports about performance of new FL strategies.
MIT License
4 stars 1 forks source link

Benchopt install -e not working as expected #3

Open jeandut opened 1 year ago

jeandut commented 1 year ago

After having installed benchopt, the command benchopt install -e at the root of this repository is supposed to create a conda environment with all our benchmark's dependencies installed. In our case this would mainly install FLamby from github with all its dependencies. However currently this does not work as the CLI is plagued by import errors and it fails to fetch FLamby with all_extra option from github. Here is how to reproduce the issue.

git clone https://github.com/benchopt/benchopt.git
git clone https://github.com/owkin/FLamby.git
git clone https://github.com/owkin/benchmark_flamby.git
conda create -n benchopt_test python=3.9
cd benchopt
pip install -e .
which benchopt
/Users/jterrail/opt/anaconda3/envs/benchopt_test/bin/benchopt
cd ../benchmark_flamby
benchopt install -e .
FedDataset module not found error in benchmark_flamby/datasets/fed_isic2019.py (and all others datasets python file)

The error above is "normalish" as the import of FLamby is skipped because it fails (it is not installed). Then I install FLamby to avoid this error (which I shouldn't have to because I would like benchopt install -e to do that for me):

cd ../FLamby
pip install -e ".[all_extra]"
cd ../benchmark_flamby
benchopt install -e .
.../var/folders/hs/9wxtjrjx2k718tlbtzyfl1dm0000gp/T/tmp_r610f7x:8: no matches found: git+https://github.com/owkin/FLamby#egg=flamby[all_extra]
Traceback (most recent call last):
  File "/Users/jterrail/opt/anaconda3/envs/benchopt_test/bin/benchopt", line 8, in <module>
    sys.exit(benchopt())
  File "/Users/jterrail/opt/anaconda3/envs/benchopt_test/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jterrail/opt/anaconda3/envs/benchopt_test/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/jterrail/opt/anaconda3/envs/benchopt_test/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jterrail/opt/anaconda3/envs/benchopt_test/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jterrail/opt/anaconda3/envs/benchopt_test/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/jterrail/Desktop/benchopt/benchopt/cli/main.py", line 473, in install
    benchmark.install_all_requirements(
  File "/Users/jterrail/Desktop/benchopt/benchopt/benchmark.py", line 411, in install_all_requirements
    install_in_conda_env(
  File "/Users/jterrail/Desktop/benchopt/benchopt/utils/conda_env_cmd.py", line 213, in install_in_conda_env
    _run_shell_in_conda_env(
  File "/Users/jterrail/Desktop/benchopt/benchopt/utils/shell_cmd.py", line 129, in _run_shell_in_conda_env
    return _run_shell(
  File "/Users/jterrail/Desktop/benchopt/benchopt/utils/shell_cmd.py", line 67, in _run_shell
    raise RuntimeError(raise_on_error.format(output=output))
RuntimeError: Failed to conda install packages pip:git+https://github.com/owkin/FLamby#egg=flamby[all_extra]
Error:
jeandut commented 1 year ago

@tomMoral answered:

For the CLI to work you need to have all dependencies installed already for your imports to succeed otherwise you get import errors when launching this command aka you need to have a conda already set up. So what does this command bring ? (genuine question)

I am also confused by your question ^^ The core idea is that if you have installed python with conda and then install benchopt, you can install the dependencies of a benchmark, either in an existing env or in a new one (the -e option of benchopt install creates an env benchopt_flamby with dependencies to the the benchmark_flamby). The idea is that you can also easily install the specific dependencies of a dataset or solver. This is important so that benchmark users have an easy install procedure for the code that it runs. For this to work, the classes Solver/Dataset/Objective should be importable even without the dependencies: this is the purpose of the safe_import_context, which catches ImportError.

For the error you are reporting, this is strange as the repo should exist and can be install with this command. Which version of pip is present in the env? If you run benchopt install -e, there is no connection to the current environment so the pip install -e command run before should not have any impact.