search

microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14.06k stars 1.82k forks source link

Trials on a Simple Test Function Fail on Colab #4414

Open arsinnius opened 2 years ago

arsinnius commented 2 years ago

Describe the issue: I'm trying to use nni on colab as describe in the book "Time Series Forecasting Using Deep Learning" by Gridin. However all my trials fail and I can't find the logs to troubleshoot. I'm running nni in interactive mode as described in

https://nni.readthedocs.io/en/stable/Tutorial/HowToLaunchFromPython.html?highlight=experiment

Environment:

Configuration:

Log message:

How to reproduce it?: Run the above code.

arsinnius commented 2 years ago

I forgot to include the trial function and search code.

Trial function code:

# trial.py

import nni

params = nni.get_next_parameter()

x = params['x']
y = params['y']
z = params['z']

metric = x + y + z

nni.report_final_result(metric)

Parameter search code:

import time
from pathlib import Path
from nni.experiment import Experiment

# Search Space
search_space = {
    "x": {"_type": "choice", "_value": [-10, -5, 0, 5, 10]},
    "y": {"_type": "choice", "_value": [-10, -5, 0, 5, 10]},
    "z": {"_type": "choice", "_value": [-10, -5, 0, 5, 10]}
}

# Search Configuration
search = Experiment('local')
training_service = 'local'

# Search Name
search.config.experiment_name = 'Hello World Search'
search.config.training_service.use_active_gpu = True
search.config.trial_concurrency = 4
search.config.max_trial_number = 50
search.config.search_space = search_space
search.config.trial_command = 'python3 trial.py'
search.config.trial_code_directory = Path('./hopt/hello_world/trial.py').parent

# Search Tuner Settings
search.config.tuner.name = 'Evolution'
search.config.tuner.class_args['optimize_mode'] = 'minimize'
search.config.tuner.class_args['population_size'] = 8

# Running Search
search.start(5000)
#search.run(port=5000)

# Awaiting Results
while True:
    if search.get_status() == 'DONE':
        trials = search.export_data()
        best_trial = min(trials, key = lambda t: t.value)
        print(f'Best trial params: {best_trial.parameter}')
        input("Experiment is finished. Press any key to exit...")
        break
    time.sleep(10)

Error Report:

ValueError                                Traceback (most recent call last)
<ipython-input-14-702e5cf7ccc4> in <module>()
     37     if search.get_status() == 'DONE':
     38         trials = search.export_data()
---> 39         best_trial = min(trials, key = lambda t: t.value)
     40         print(f'Best trial params: {best_trial.parameter}')
     41         input("Experiment is finished. Press any key to exit...")

ValueError: min() arg is an empty sequence

Any help is greatly appreciated.

arsinnius commented 2 years ago

I also forgot to mention that I'm using ngrok as described in https://nni.readthedocs.io/en/latest/CommunitySharings/NNI_colab_support.html. Thanks again.

ultmaster commented 2 years ago

I tested the your code. The code runs smoothly in colab. My try-out is here: https://colab.research.google.com/drive/1ME7zVP2wdgYTas_ihO9SxlpR0PoOwAss?usp=sharing

I can't reproduce your error.

@arsinnius I didn't use ngrok. Maybe that's the problem? ngrok looks super-complex to me. Would you mind trying again without ngrok?