search

facebookresearch / nevergrad

A Python toolbox for performing gradient-free optimization
https://facebookresearch.github.io/nevergrad/
MIT License
3.89k stars 349 forks source link

Difficulties to follow your development process under Windows #449

Closed brockho closed 4 years ago

brockho commented 4 years ago

I have a few issues with following the suggested work flow here on my Windows machine (Windows 10, Anaconda python 3.7.3).

Steps to reproduce

  1. run pip install -e '.[all]' in the nevergrad folder did not work at all:

    C:\Users\dimo\Desktop\nevergrad>pip install -e `.[all]`
`.[all]` should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+

    running pip install -e . however, installed nevergrad without problems

    C:\Users\dimo\Desktop\nevergrad>pip install -e .
Obtaining file:///C:/Users/dimo/Desktop/nevergrad
Requirement already satisfied: numpy>=1.15.0 in c:\users\dimo\anaconda3\lib\site-packages (from nevergrad==0.3.0) (1.16.2)
Requirement already satisfied: pandas>=0.23.4 in c:\users\dimo\anaconda3\lib\site-packages (from nevergrad==0.3.0) (0.24.2)
Requirement already satisfied: cma>=2.6.0 in c:\users\dimo\anaconda3\lib\site-packages (from nevergrad==0.3.0) (2.7.0)
Requirement already satisfied: bayesian-optimization>=1.0.1 in c:\users\dimo\anaconda3\lib\site-packages (from nevergrad==0.3.0) (1.0.1)
Requirement already satisfied: typing_extensions>=3.6.6 in c:\users\dimo\anaconda3\lib\site-packages (from nevergrad==0.3.0) (3.7.4)
Requirement already satisfied: python-dateutil>=2.5.0 in c:\users\dimo\anaconda3\lib\site-packages (from pandas>=0.23.4->nevergrad==0.3.0) (2.8.0)
Requirement already satisfied: pytz>=2011k in c:\users\dimo\anaconda3\lib\site-packages (from pandas>=0.23.4->nevergrad==0.3.0) (2018.9)
Requirement already satisfied: scikit-learn>=0.18.0 in c:\users\dimo\anaconda3\lib\site-packages (from bayesian-optimization>=1.0.1->nevergrad==0.3.0) (0.20.3)
Requirement already satisfied: scipy>=0.14.0 in c:\users\dimo\anaconda3\lib\site-packages (from bayesian-optimization>=1.0.1->nevergrad==0.3.0) (1.2.1)
Requirement already satisfied: six>=1.5 in c:\users\dimo\anaconda3\lib\site-packages (from python-dateutil>=2.5.0->pandas>=0.23.4->nevergrad==0.3.0) (1.12.0)
Installing collected packages: nevergrad
Found existing installation: nevergrad 0.3.0
Uninstalling nevergrad-0.3.0:
  Successfully uninstalled nevergrad-0.3.0
Running setup.py develop for nevergrad
Successfully installed nevergrad

    So this looks rather a documentation issue than an actual code issue.

  2. However, the next step in the development process is to run pytest nevergrad --cov=nevergrad which, for me, results in quite some problems, mainly related to the operating system (output below from the master branch, cut for length):

    
C:\Users\dimo\Desktop\nevergrad>pytest nevergrad --cov=nevergrad
Test session starts (platform: win32, Python 3.7.3, pytest 4.3.1, pytest-sugar 0.9.2)
rootdir: C:\Users\dimo\Desktop\nevergrad, inifile:
plugins: sugar-0.9.2, remotedata-0.3.1, openfiles-0.3.2, mock-2.0.0, doctestplus-0.3.0, cov-2.8.1, arraydiff-0.3
collecting ...
nevergrad/benchmark/test_core.py ✓                                                                         0%            nevergrad/benchmark/test_core.py ✓✓                                                                        0%            nevergrad/benchmark/test_core.py ✓✓✓                                                                       0%            nevergrad/benchmark/test_core.py ✓✓✓✓                                                                      0% ▏          nevergrad/benchmark/test_core.py ✓✓✓✓✓                                                                     1% ▏

――――――――――――――――――――――――――――――――――――――――――――――― test_commandline_launch ―――――――――――――――――――――――――――――――――――――――――――――――― subprocess.CalledProcessError: Command '['python', '-m', 'nevergrad.benchmark', 'additional_experiment', '--cap_index', '2', '--num_workers', '2', '--output', 'C:\Users\dimo\AppData\Local\Temp\tmpjuk3d1f8\benchmark_launch_test.csv', '--imports', 'C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark\additional\example.py']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

def test_commandline_launch() -> None:
    with tempfile.TemporaryDirectory() as folder:
        output = Path(folder) / "benchmark_launch_test.csv"
        # commandline test
        CommandFunction(command=["python", "-m", "nevergrad.benchmark", "additional_experiment",
                                 "--cap_index", "2", "--num_workers", "2", "--output", str(output),
                               "--imports", str(Path(__file__).parent / "additional" / "example.py")])()

nevergrad\benchmark\test_core.py:46:

self = <nevergrad.instrumentation.utils.CommandFunction object at 0x0000024EFF2C6518>, args = (), kwargs = {} full_command = ['python', '-m', 'nevergrad.benchmark', 'additional_experiment', '--cap_index', '2', ...] outlines = ['Starting repetition 1 / 1'], process = <subprocess.Popen object at 0x0000024EFF2C6D30> line = b'Starting repetition 1 / 1\r\n' stderr = b'concurrent.futures.process._RemoteTraceback: \r\n"""\r\nTraceback (most recent call last):\r\n File "C:\Users\dim...\futures\_base.py", line 384, in __get_result\r\n raise self._exception\r\nKeyError: \'additional_experiment\'\r\n' stdout = 'Starting repetition 1 / 1', retcode = 1 subprocess_error = CalledProcessError(1, ['python', '-m', 'nevergrad.benchmark', 'additional_experiment', '--cap_index', '2', '--num_work...rk_launch_test.csv', '--imports', 'C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark\additional\example.py'])

def __call__(self, *args: Any, **kwargs: Any) -> str:
    """Call the cammand line with addidional arguments
    The keyword arguments will be sent as --{key}={val}
    The logs are bufferized. They will be printed if the job fails, or sent as output of the function
    Errors are provided with the internal stderr
    """
    # TODO make the following command more robust (probably fails in multiple cases)
    full_command = self.command + [str(x) for x in args] + ["--{}={}".format(x, y) for x, y in kwargs.items()]
    if self.verbose:
        print(f"The following command is sent: {full_command}")
    outlines: List[str] = []
    with subprocess.Popen(full_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
                          shell=False, cwd=self.cwd, env=self.env) as process:
        try:
            for line in iter(process.stdout.readline, b''):
                if not line:
                    break
                outlines.append(line.decode().strip())
                if self.verbose:
                    print(outlines[-1], flush=True)
        except Exception:  # pylint: disable=broad-except
            process.kill()
            process.wait()
            raise FailedJobError("Job got killed for an unknown reason.")
        stderr = process.communicate()[1]  # we already got stdout
        stdout = "\n".join(outlines)
        retcode = process.poll()
        if stderr and (retcode or self.verbose):
            print(stderr.decode(), file=sys.stderr)
        if retcode:
            subprocess_error = subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
          raise FailedJobError(stderr.decode()) from subprocess_error

E nevergrad.instrumentation.utils.FailedJobError: concurrent.futures.process._RemoteTraceback: E """ E Traceback (most recent call last): E File "C:\Users\dimo\Anaconda3\lib\concurrent\futures\process.py", line 232, in _process_worker E r = call_item.fn(*call_item.args, **call_item.kwargs) E File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark\core.py", line 173, in compute E for local_ind, (index, xp) in enumerate(self): E File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark\core.py", line 127, in iter E maker = registry[self.name] E File "C:\Users\dimo\Desktop\nevergrad\nevergrad\common\decorators.py", line 61, in getitem E return self.data[key] E KeyError: 'additional_experiment' E """ E E The above exception was the direct cause of the following exception: E E Traceback (most recent call last): E File "C:\Users\dimo\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main E "main", mod_spec) E File "C:\Users\dimo\Anaconda3\lib\runpy.py", line 85, in _run_code E exec(code, run_globals) E File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark__main.py", line 93, in E repetitions=args.repetitions) E File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark__main__.py", line 79, in repeated_launch E seed=None if seed is None else seed + k) E File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark__main__.py", line 28, in launch E df = core.compute(experiment, seed=seed, cap_index=cap_index, executor=executor, num_workers=num_workers) E File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark\core.py", line 279, in compute E dfs = [j.result() for j in jobs] E File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark\core.py", line 279, in E dfs = [j.result() for j in jobs] E File "C:\Users\dimo\Anaconda3\lib\concurrent\futures_base.py", line 432, in result E return self.get_result() E File "C:\Users\dimo\Anaconda3\lib\concurrent\futures_base.py", line 384, in __get_result E raise self._exception E KeyError: 'additional_experiment'

nevergrad\instrumentation\utils.py:114: FailedJobError ------------------------------------------------- Captured stderr call ------------------------------------------------- concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "C:\Users\dimo\Anaconda3\lib\concurrent\futures\process.py", line 232, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark\core.py", line 173, in compute for local_ind, (index, xp) in enumerate(self): File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark\core.py", line 127, in iter maker = registry[self.name] File "C:\Users\dimo\Desktop\nevergrad\nevergrad\common\decorators.py", line 61, in getitem return self.data[key] KeyError: 'additional_experiment' """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Users\dimo\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "C:\Users\dimo\Anaconda3\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark__main.py", line 93, in repetitions=args.repetitions) File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark__main__.py", line 79, in repeated_launch seed=None if seed is None else seed + k) File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark__main__.py", line 28, in launch df = core.compute(experiment, seed=seed, cap_index=cap_index, executor=executor, num_workers=num_workers) File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark\core.py", line 279, in compute dfs = [j.result() for j in jobs] File "C:\Users\dimo\Desktop\nevergrad\nevergrad\benchmark\core.py", line 279, in dfs = [j.result() for j in jobs] File "C:\Users\dimo\Anaconda3\lib\concurrent\futures_base.py", line 432, in result return self.get_result() File "C:\Users\dimo\Anaconda3\lib\concurrent\futures_base.py", line 384, in __get_result raise self._exception KeyError: 'additional_experiment'

nevergrad/benchmark/test_core.py ⨯ 1% ▏ nevergrad/benchmark/test_core.py ⨯✓ 1% ▏
[...] nevergrad/common/test_tools.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 11% █▏ nevergrad/common/test_tools.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 11% █▏ nevergrad/common/test_tools.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 11% █▏

――――――――――――――――――――――――――――――――――――――――――――――――――――― test_sleeper ―――――――――――――――――――――――――――――――――――――――――――――――――――――

def test_sleeper() -> None:
    min_sleep = 1e-5
    sleeper = tools.Sleeper(min_sleep=min_sleep)
    np.testing.assert_almost_equal(sleeper._get_advised_sleep_duration(), min_sleep, decimal=5)
    sleeper.start_timer()
    np.testing.assert_almost_equal(sleeper._get_advised_sleep_duration(), min_sleep, decimal=5)
    sleeper.stop_timer()
  np.testing.assert_almost_equal(sleeper._get_advised_sleep_duration(), min_sleep, decimal=5)

E AssertionError: E Arrays are not almost equal to 5 decimals E ACTUAL: 4.661083221435547e-05 E DESIRED: 1e-05

nevergrad\common\test_tools.py:87: AssertionError

nevergrad/common/test_tools.py ⨯ 11% █▏
nevergrad/functions/test_corefuncs.py ✓ 11% █▏
[...] nevergrad/instrumentation/test_discretization.py ✓✓✓✓✓✓✓✓✓✓✓✓✓ 27% ██▊
nevergrad/instrumentation/test_discretization.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 27% ██▊

――――――――――――――――――――――――――――――――――――――――――――――― test_symlink_folder_tree ―――――――――――――――――――――――――――――――――――――――――――――――

def test_symlink_folder_tree() -> None:
    path = Path(__file__).absolute().parents[1]
    assert path.name == "nevergrad"
    with tempfile.TemporaryDirectory() as folder:
      instantiate.symlink_folder_tree(path, folder)

nevergrad\instrumentation\test_instantiate.py:20:

nevergrad\instrumentation\instantiate.py:94: in symlink_folder_tree symlink_folder_tree(fp, shadow_fp) nevergrad\instrumentation\instantiate.py:94: in symlink_folder_tree symlink_folder_tree(fp, shadow_fp) nevergrad\instrumentation\instantiate.py:96: in symlink_folder_tree shadow_fp.symlink_to(fp)

self = WindowsPath('C:/Users/dimo/AppData/Local/Temp/tmp7le_w28a/benchmark/additional/example.py') target = WindowsPath('C:/Users/dimo/Desktop/nevergrad/nevergrad/benchmark/additional/example.py') target_is_directory = False

def symlink_to(self, target, target_is_directory=False):
    """
    Make this path a symlink pointing to the given path.
    Note the order of arguments (self, target) is the reverse of os.symlink's.
    """
    if self._closed:
        self._raise_closed()
  self._accessor.symlink(target, self, target_is_directory)

E OSError: symbolic link privilege not held

....\Anaconda3\lib\pathlib.py:1330: OSError

nevergrad/instrumentation/test_instantiate.py ⨯ 27% ██▊
nevergrad/instrumentation/test_instantiate.py ⨯✓ 28% ██▊
[...] nevergrad/instrumentation/test_instantiate.py ⨯✓✓✓✓✓✓✓✓ 28% ██▉
nevergrad/instrumentation/test_instantiate.py ⨯✓✓✓✓✓✓✓✓✓ 28% ██▉

―――――――――――――――――――――――――――――――――――――― test_folder_instantiator[with_clean_copy] ―――――――――――――――――――――――――――――――――――――――

clean_copy = True

@testing.parametrized(
    with_clean_copy=(True,),
    without_clean_copy=(False,),
)
def test_folder_instantiator(clean_copy: bool) -> None:
    path = Path(__file__).parent / "examples"
    ifolder = instantiate.FolderInstantiator(path, clean_copy=clean_copy)
    testing.printed_assert_equal(ifolder.placeholders, _EXPECTED)
    np.testing.assert_equal(len(ifolder.file_functions), 1)
  with ifolder.instantiate(value1=12, value2=110., string="") as tmp:

nevergrad\instrumentation\test_instantiate.py:68:

....\Anaconda3\lib\contextlib.py:112: in enter return next(self.gen) nevergrad\instrumentation\instantiate.py:215: in instantiate self.instantiate_to_folder(subtempfolder, kwargs) nevergrad\instrumentation\instantiate.py:204: in instantiate_to_folder symlink_folder_tree(self.folder, outfolder) nevergrad\instrumentation\instantiate.py:96: in symlink_folder_tree shadow_fp.symlink_to(fp)

self = WindowsPath('C:/Users/dimo/AppData/Local/Temp/tmp64j87q_g/examples/script.py') target = WindowsPath('C:/Users/dimo/AppData/Local/Temp/tmp_clean_copy_tmko4nqc/examples/script.py') target_is_directory = False

def symlink_to(self, target, target_is_directory=False):
    """
    Make this path a symlink pointing to the given path.
    Note the order of arguments (self, target) is the reverse of os.symlink's.
    """
    if self._closed:
        self._raise_closed()
  self._accessor.symlink(target, self, target_is_directory)

E OSError: symbolic link privilege not held

....\Anaconda3\lib\pathlib.py:1330: OSError

nevergrad/instrumentation/test_instantiate.py ⨯ 29% ██▉

――――――――――――――――――――――――――――――――――――― test_folder_instantiator[without_clean_copy] ―――――――――――――――――――――――――――――――――――――

clean_copy = False

@testing.parametrized(
    with_clean_copy=(True,),
    without_clean_copy=(False,),
)
def test_folder_instantiator(clean_copy: bool) -> None:
    path = Path(__file__).parent / "examples"
    ifolder = instantiate.FolderInstantiator(path, clean_copy=clean_copy)
    testing.printed_assert_equal(ifolder.placeholders, _EXPECTED)
    np.testing.assert_equal(len(ifolder.file_functions), 1)
  with ifolder.instantiate(value1=12, value2=110., string="") as tmp:

nevergrad\instrumentation\test_instantiate.py:68:

....\Anaconda3\lib\contextlib.py:112: in enter return next(self.gen) nevergrad\instrumentation\instantiate.py:215: in instantiate self.instantiate_to_folder(subtempfolder, kwargs) nevergrad\instrumentation\instantiate.py:204: in instantiate_to_folder symlink_folder_tree(self.folder, outfolder) nevergrad\instrumentation\instantiate.py:96: in symlink_folder_tree shadow_fp.symlink_to(fp)

self = WindowsPath('C:/Users/dimo/AppData/Local/Temp/tmpsteavf_a/examples/script.py') target = WindowsPath('C:/Users/dimo/Desktop/nevergrad/nevergrad/instrumentation/examples/script.py') target_is_directory = False

def symlink_to(self, target, target_is_directory=False):
    """
    Make this path a symlink pointing to the given path.
    Note the order of arguments (self, target) is the reverse of os.symlink's.
    """
    if self._closed:
        self._raise_closed()
  self._accessor.symlink(target, self, target_is_directory)

E OSError: symbolic link privilege not held

....\Anaconda3\lib\pathlib.py:1330: OSError

nevergrad/instrumentation/test_instantiate.py ⨯ 29% ██▉
nevergrad/instrumentation/test_instantiate.py ⨯✓ 29% ██▉
[...] nevergrad/instrumentation/test_instantiate.py ⨯✓✓✓✓✓✓ 29% ██▉
nevergrad/instrumentation/test_instantiate.py ⨯✓✓✓✓✓✓✓ 29% ██▉

――――――――――――――――――――――――――――――――――――――――――――――――― test_folder_function ―――――――――――――――――――――――――――――――――――――――――――――――――

def test_folder_function() -> None:
    folder = Path(__file__).parent / "examples"
    func = instantiate.FolderFunction(str(folder), ["python", "examples/script.py"], clean_copy=True)
  output = func(value1=98, value2=12, string="plop")

nevergrad\instrumentation\test_instantiate.py:114:

nevergrad\instrumentation\instantiate.py:279: in call with self.instantiator.instantiate(**kwargs) as folder: ....\Anaconda3\lib\contextlib.py:112: in enter return next(self.gen) nevergrad\instrumentation\instantiate.py:215: in instantiate self.instantiate_to_folder(subtempfolder, kwargs) nevergrad\instrumentation\instantiate.py:204: in instantiate_to_folder symlink_folder_tree(self.folder, outfolder) nevergrad\instrumentation\instantiate.py:96: in symlink_folder_tree shadow_fp.symlink_to(fp)

self = WindowsPath('C:/Users/dimo/AppData/Local/Temp/tmpf2n3_rpg/examples/script.py') target = WindowsPath('C:/Users/dimo/AppData/Local/Temp/tmp_clean_copy_lsj82by7/examples/script.py') target_is_directory = False

def symlink_to(self, target, target_is_directory=False):
    """
    Make this path a symlink pointing to the given path.
    Note the order of arguments (self, target) is the reverse of os.symlink's.
    """
    if self._closed:
        self._raise_closed()
  self._accessor.symlink(target, self, target_is_directory)

E OSError: symbolic link privilege not held

....\Anaconda3\lib\pathlib.py:1330: OSError

nevergrad/instrumentation/test_instantiate.py ⨯ 30% ██▉
nevergrad/instrumentation/test_multivariables.py ✓ 30% ███
[...] ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 52% █████▎ ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 52% █████▎ ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 52% █████▎ capi_return is NULL Call-back cb_calcfc_incobylauserroutines failed. ✓✓capi_return is NULL█████▎ Call-back cb_calcfc_incobylauser__routines failed. ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 52% █████▎ ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 52% █████▎
[...] ████████▊ ✓✓✓✓✓✓✓✓✓ 88% ████████▊ ✓✓✓✓✓✓✓✓✓✓ 88% ████████▊ ✓✓✓✓✓✓✓✓✓✓✓ 88% ████████▊ ✓✓✓✓✓capi_return is NULL Call-back cb_calcfc_incobylauserroutines failed. ✓✓✓✓✓✓capi_return is NULL Call-back cb_calcfc_incobylauser__routines failed. ✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 89% ████████▉ ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 89% ████████▉ ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓ 89% ████████▉ ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓
[...] 99% ██████████ nevergrad/optimization/test_utils.py ✓✓✓✓✓✓✓✓✓✓ 99% ██████████ nevergrad/optimization/test_utils.py ✓✓✓✓✓✓✓✓✓✓✓ 100% ██████████ =================================================== warnings summary =================================================== C:\Users\dimo\Anaconda3\lib\site-packages\cma\utilities\utils.py:8 C:\Users\dimo\Anaconda3\lib\site-packages\cma\utilities\utils.py:8: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import MutableMapping # since Python 2.4?

nevergrad/benchmark/test_experiments.py::test_experiments_registry[mlda] nevergrad/benchmark/test_experiments.py::test_experiments_registry[mldaas] nevergrad/benchmark/test_experiments.py::test_experiments_registry[realworld] C:\Users\dimo\Desktop\nevergrad\nevergrad\functions\mlda\problems.py:45: RuntimeWarning: invalid value encountered in true_divide self._points /= np.std(self._points, axis=0, keepdims=True)

nevergrad/benchmark/test_experiments.py::test_experiments_registry[mlda] nevergrad/benchmark/test_experiments.py::test_experiments_registry[mldaas] nevergrad/benchmark/test_experiments.py::test_experiments_registry[realworld] C:\Users\dimo\Desktop\nevergrad\nevergrad\functions\mlda\problems.py:181: RuntimeWarning: invalid value encountered in true_divide raw_data /= np.std(raw_data, axis=0, keepdims=True)

nevergrad/optimization/test_optimizerlib.py::test_optimizers_suggest[DEwithLHS] nevergrad/optimization/test_optimizerlib.py::test_optimizers_suggest[DEwithLHSdim] nevergrad/optimization/test_optimizerlib.py::test_optimizers_suggest[DEwithLHS30] nevergrad/optimization/test_optimizerlib.py::test_optimizers_recommendation[DEwithLHS] nevergrad/optimization/test_optimizerlib.py::test_optimizers_recommendation[DEwithLHSdim] nevergrad/optimization/test_optimizerlib.py::test_optimizers_recommendation[DEwithLHS30] C:\Users\dimo\Desktop\nevergrad\nevergrad\optimization\differentialevolution.py:216: InefficientSettingsWarning: DE algorithms are inefficient with budget < 60 warnings.warn("DE algorithms are inefficient with budget < 60", base.InefficientSettingsWarning)

nevergrad/optimization/test_optimizerlib.py::test_optimizers_suggest[PSOwithLHS30] nevergrad/optimization/test_optimizerlib.py::test_optimizers_recommendation[PSOwithLHS30] C:\Users\dimo\Desktop\nevergrad\nevergrad\optimization\optimizerlib.py:633: InefficientSettingsWarning: PSO is inefficient with budget < 60 warnings.warn("PSO is inefficient with budget < 60", base.InefficientSettingsWarning)

-- Docs: https://docs.pytest.org/en/latest/warnings.html

----------- coverage: platform win32, python 3.7.3-final-0 ----------- Name Stmts Miss Cover

nevergrad__init.py 7 0 100% nevergrad\benchmark\init.py 2 0 100% nevergrad\benchmark\main.py 51 5 90% nevergrad\benchmark\additional\example.py 21 1 95% nevergrad\benchmark\core.py 116 9 92% nevergrad\benchmark\execution.py 77 2 97% nevergrad\benchmark\experiments.py 249 0 100% nevergrad\benchmark\frozenexperiments.py 151 0 100% nevergrad\benchmark\plotting.py 293 17 94% nevergrad\benchmark\xpbase.py 127 5 96% nevergrad\common\init.py 0 0 100% nevergrad\common\decorators.py 38 0 100% nevergrad\common\testing.py 65 5 92% nevergrad\common\tools.py 92 6 93% nevergrad\common\typetools.py 16 3 81% nevergrad\functions\init.py 3 0 100% nevergrad\functions\arcoating\init.py 1 0 100% nevergrad\functions\arcoating\core.py 43 1 98% nevergrad\functions\corefuncs.py 200 5 98% nevergrad\functions\functionlib.py 111 2 98% nevergrad\functions\mlda\init.py 1 0 100% nevergrad\functions\mlda\datasets.py 59 11 81% nevergrad\functions\mlda\problems.py 106 2 98% nevergrad\functions\multiobjective\init.py 1 0 100% nevergrad\functions\multiobjective\core.py 47 0 100% nevergrad\functions\multiobjective\pyhv.py 169 89 47% nevergrad\functions\photonics\init.py 1 0 100% nevergrad\functions\photonics\core.py 43 2 95% nevergrad\functions\rl\init.py 3 0 100% nevergrad\functions\rl\agents.py 105 3 97% nevergrad\functions\rl\base.py 130 8 94% nevergrad\functions\rl\envs.py 68 0 100% nevergrad\functions\utils.py 25 2 92% nevergrad\instrumentation\init.py 6 0 100% nevergrad\instrumentation\core.py 87 4 95% nevergrad\instrumentation\discretization.py 43 0 100% nevergrad\instrumentation\instantiate.py 177 34 81% nevergrad\instrumentation\multivariables.py 132 3 98% nevergrad\instrumentation\transforms.py 120 5 96% nevergrad\instrumentation\utils.py 57 8 86% nevergrad\instrumentation\variables.py 177 9 95% nevergrad\optimization\init__.py 4 0 100% nevergrad\optimization\base.py 302 30 90% nevergrad\optimization\callbacks.py 57 2 96% nevergrad\optimization\differentialevolution.py 150 3 98% nevergrad\optimization\families.py 9 0 100% nevergrad\optimization\mutations.py 44 1 98% nevergrad\optimization\oneshot.py 175 2 99% nevergrad\optimization\optimizerlib.py 803 22 97% nevergrad\optimization\recaster.py 133 16 88% nevergrad\optimization\recastlib.py 46 2 96% nevergrad\optimization\requirements_check.py 10 10 0% nevergrad\optimization\rescaledoneshot.py 39 0 100% nevergrad\optimization\sequences.py 134 4 97% nevergrad\optimization\utils.py 204 9 96%

TOTAL 5330 342 94%

Results (157.22s): 900 passed 6 failed

Observed Results

Expected Results

Relevant Code

jrapin commented 4 years ago

Hi, Thank you for raising the issue! We're developing on MacOS and Ubuntu here, so as you point out, we've not been able to check everything worked well on Windows. If you have time to help, we can for sure make it work though.

  1. pip install -e '.[all]' Can you check if pip install -e .[all] works better for you? With MaxOS and Ubuntu, the first one works on both bash and zsh, while the second only works for bash. If the second works on windows, I'll update the documentation and make a specific case for zsh. Note that with pip install -e . you only install the main dependencies so many things are not expected to work (all the benchmark part + all the dev related packages such as pytest, and numpy-stubs since #450)

  2. OSError: symbolic link privilege not held It seems Windows requires admin rights to create a symlink. This test should probably just be deactivated under Windows, not sure we can do much more about it, and not sure it would matter anyway.

  3. KeyError: 'additional_experiment' This is due to a dynamic import that seems to fail on Windows, no idea why. This is a feature that is probably hardly used at all, so I see two options:

    • you're willing to investigate and find a way to make it work on Windows (on my side, I can't iterate fast enough on it without actually working on Windows)
    • we just ignore the test on Windows and if somebody needs the feature, they'll help :)

4.test_sleeper This test is messy because it depends on the current load and resources of the computer... I'll probably just remove most of it.

  1. Adding CI for Windows: indeed. When we are able to solve all this, we can add it. There's an example of how to do it in a related project: https://github.com/facebookresearch/hydra/blob/master/.circleci/config.yml

I think I can provide quick fixes for most of it easily so I'll do it. For 3., I'll just ignore the test for now, if you're willing to investigate just don't hesitate ;) but I'm not sure it's really necessary. I'll probably need help testing this though if you have some time.

jrapin commented 4 years ago

@brockho can you check that #451 (now merged into master) works for you? And as mention above, let me know if pip install -e .[all] also works on Windows

jrapin commented 4 years ago

pip install -e .[all] seems to work on Windows, I've updated the docs accordingly. #488 also improves a bit on the test compatibility.

brockho commented 4 years ago

I agree, with the newest version, the dependency with tools.nnwrap is gone and the pip install -e .[all] works also for me under Windows. Nice :-)

Many thanks!

jrapin commented 4 years ago

No problem ;) I think there are still a few errors in the test on Windows though. I'll try again when I have time to experiment on Windows again, or I'll add a Windows CI if it's straightforward enough.

jrapin commented 4 years ago

CI can be activated for Windows #494 However, it is very flaky and I don't have much time to investigate right now :(

jrapin commented 4 years ago

I've activated the CI since multiple runs seemed to work just fine, I'll close this task now and create a new one for solving flaky tests under Windows when/if we identify some. Also, don't hesitate to create new ones if you encounter any issue!