search

facebookresearch / nevergrad

A Python toolbox for performing gradient-free optimization
https://facebookresearch.github.io/nevergrad/
MIT License
3.95k stars 353 forks source link

Several import errors when testing each optimizer, and false warnings about budget #1490

Closed nhansendev closed 1 year ago

nhansendev commented 1 year ago

Steps to reproduce

Run the following script:

import nevergrad as ng
import numpy as np

def square(x):
    return sum(np.sin(x))

# These create run-time import errors
bad_calls = ["FCMA", "NLOPT"]
err_list = []

for k, v in ng.optimizers.registry.items():
    try:
        if k not in bad_calls:
            print(k, end="")
            optimizer = v(parametrization=3, budget=100)
            recommendation = optimizer.minimize(square)
            print(": OK")
    except ModuleNotFoundError:
        # Catchable module not found errors
        print(": --------- ERR")
        err_list.append(k)
        pass

print("Modules with errors:")
for e in err_list:
    print(e)

Observed Results

RandomSearch: OK QORandomSearch: OK ORandomSearch: OK RandomSearchPlusMiddlePoint: OK MetaRecentering: OK MetaTuneRecentering: OK HullAvgMetaTuneRecentering: OK HullAvgMetaRecentering: OK AvgMetaRecenteringNoHull: OK HaltonSearch: OK HaltonSearchPlusMiddlePoint: OK LargeHaltonSearch: OK ScrHaltonSearch: OK ScrHaltonSearchPlusMiddlePoint: OK HammersleySearch: OK HammersleySearchPlusMiddlePoint: OK ScrHammersleySearchPlusMiddlePoint: OK ScrHammersleySearch: OK QOScrHammersleySearch: OK OScrHammersleySearch: OK CauchyScrHammersleySearch: OK LHSSearch: OK CauchyLHSSearch: OK DE: OK TwoPointsDE: OK RotatedTwoPointsDE: OK LhsDE: OK QrDE: OK NoisyDE: OK AlmostRotationInvariantDE: OK RotationInvariantDE: OK RecES: OK RecMixES: OK RecMutDE: OK ES: OK MixES: OK MutDE: OK NonNSGAIIES: OK NelderMead: OK CmaFmin2: OK Powell: OK RPowell: OK Cobyla: OK RCobyla: OK SQP: OK RSQP: OK /home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/recastlib.py:263: NevergradRuntimeWarning: with this optimizer, it is more efficient to set num_objectives before the optimization begins warnings.warn( PymooNSGA2: --------- ERR OnePlusOne: OK NoisyOnePlusOne: OK DiscreteOnePlusOne: OK PortfolioDiscreteOnePlusOne: OK DiscreteLenglerOnePlusOne: OK AdaptiveDiscreteOnePlusOne: OK AnisotropicAdaptiveDiscreteOnePlusOne: OK DiscreteBSOOnePlusOne: OK DiscreteDoerrOnePlusOne: OK CauchyOnePlusOne: OK OptimisticNoisyOnePlusOne: OK OptimisticDiscreteOnePlusOne: OK NoisyDiscreteOnePlusOne: OK DoubleFastGADiscreteOnePlusOne: OK SparseDoubleFastGADiscreteOnePlusOne: OK RecombiningPortfolioOptimisticNoisyDiscreteOnePlusOne: OK RecombiningPortfolioDiscreteOnePlusOne: OK CMA: OK DiagonalCMA: OK EDA: OK TBPSA: OK NaiveTBPSA: OK NoisyBandit: OK RealSpacePSO: OK PSO: OK SPSA: OK RescaledCMA: OK /home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/differentialevolution.py:106: InefficientSettingsWarning: DE algorithms are inefficient with budget < 60 warnings.warn( Portfolio: OK /home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/optimizerlib.py:834: InefficientSettingsWarning: PSO is inefficient with budget < 60 warnings.warn("PSO is inefficient with budget < 60", errors.InefficientSettingsWarning) ParaPortfolio: OK ASCMADEthird: OK MultiCMA: OK TripleCMA: OK PolyCMA: OK MultiScaleCMA: OK MetaModel: OK MetaModelOnePlusOne: OK /home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/differentialevolution.py:106: InefficientSettingsWarning: DE algorithms are inefficient with budget < 60 warnings.warn( SQPCMA: OK MultiDiscrete: OK CMandAS2: OK CMandAS3: OK CM: OK BO: OK BOSplit: OK PCABO: --------- ERR BayesOptimBO: --------- ERR /home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/differentialevolution.py:106: InefficientSettingsWarning: DE algorithms are inefficient with budget < 60 warnings.warn( GeneticDE: OK ChainCMAPowell: OK ChainMetaModelSQP: OK ChainMetaModelPowell: OK ChainDiagonalCMAPowell: OK ChainNaiveTBPSAPowell: OK ChainNaiveTBPSACMAPowell: OK cGA: OK NaiveIsoEMNA: OK NGOptBase: OK Shiwa: OK NGO: OK NGOpt4: OK NGOpt8: OK NGOpt10: OK NGOpt12: OK NGOpt13: OK NGOpt14: OK NGOpt15: OK NGOpt16: OK NGOpt21: OK NGOpt36: OK NGOpt38: OK NGOpt39: OK NGOpt: OK

Modules caught by the "ModuleNotFoundError" exception: PymooNSGA2 PCABO BayesOptimBO

Note the random warnings in the list about InefficientSettingsWarning: DE algorithms are inefficient with budget < 60 while the budget is clearly set to 100.

Additionally, FCMA and NLOPT could not be called without runtime import errors not caught by the "ModuleNotFoundError" exception:

Traceback (most recent call last):
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/recaster.py", line 66, in run
    self.output = self._caller(self._fake_callable, *self._args, **self._kwargs)
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/recastlib.py", line 78, in _optimization_function
    import nlopt
ModuleNotFoundError: No module named 'nlopt'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nate/Desktop/Atom/ManualStrat/ng_bug.py", line 14, in <module>
    recommendation = optimizer.minimize(square)
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/base.py", line 659, in minimize
    args = self.ask()
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/base.py", line 469, in ask
    candidate = self._internal_ask_candidate()
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/recaster.py", line 276, in _internal_ask_candidate
    self._check_error()
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/recaster.py", line 177, in _check_error
    raise RuntimeError(
RuntimeError: Recast optimizer raised an error:
No module named 'nlopt'

and

Traceback (most recent call last):
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/optimizerlib.py", line 454, in es
    from fcmaes import cmaes
ModuleNotFoundError: No module named 'fcmaes'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nate/Desktop/Atom/ManualStrat/ng_bug.py", line 15, in <module>
    recommendation = optimizer.minimize(square)
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/base.py", line 659, in minimize
    args = self.ask()
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/base.py", line 469, in ask
    candidate = self._internal_ask_candidate()
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/optimizerlib.py", line 469, in _internal_ask_candidate
    self._to_be_asked.extend(self.es.ask())
  File "/home/nate/.local/lib/python3.10/site-packages/nevergrad/optimization/optimizerlib.py", line 456, in es
    raise ImportError(
ImportError: Please install fcmaes (pip install fcmaes) to use FCMA optimizers

These dependencies are not mentioned in the documentation anywhere that I've seen, so my suggestion is to add that.

Expected Results

All optimizers are runnable without import errors.

Relevant Code

See above.

teytaud commented 1 year ago

ModuleNotFoundError are on purpose. Users were complaining that we can not easily import / install nevergrad because of the many dependencies. So for some methods, the user has to install them by himself. However, after git clone, pip install -e .[all] will install everything.

For warnings I agree that many of them should just be removed all together... I'll do that. This is done in branch "newsmooth" and will be merged in nevergrad 0.14.0.

I guess the warning for budget < 60 appears with budget >60 when an algorithm distributes the budget among several algorithms (bet and run) and each of them has therefore a low budget.