list index out of range when training model with multi-headed inputs, and Limiters lead to < 1 permutations when fraction limit is set on 4E41 parameter permutations

daniel-corcoran commented 4 years ago

1) Confirm the below

[X ] I have looked for an answer in the Docs
[X ] My Python version is 3.5 or higher
[X ] I have searched through the issues Issues for a duplicate
[X ] I've tested that my Keras model works as a stand-alone

2) Include the output of:

talos.__version__ 0.6.4

3) Explain clearly what you are trying to achieve

A description of your specific use-case and what you hope to achieve with it I have a multi-headed network with multiple inputs. Here is a simple example.

`def train_grid():
    x_c = np.random.rand(7843, 8, 1)
    x_p = np.random.rand(7843, 180, 1)
    x_h = np.random.rand(7843, 9, 1)
    shannon = np.random.rand(7843, 9, 1)
    flank_h = np.random.rand(7843, 20, 1)
    flank_p = np.random.rand(7843, 400, 1)
    flank_s = np.random.rand(7843, 20, 1)
    property = np.random.rand(7843, 8, 1)

    y_train = np.random.rand(7843)

    x_train = [x_c, x_p, x_h, shannon, flank_h, flank_p, flank_s, property]

    scan_object = ta.Scan(x=x_train, y=y_train, x_val=x_train, y_val = y_train, model=get_model, params=grid_params, experiment_name = 'Multiheaded input test')
    ta.Analyze(scan_object)`

I can train the network without using ta.Scan via:

`def train():
    x_c = np.random.rand(7843, 8, 1)
    x_p = np.random.rand(7843, 180, 1)
    x_h = np.random.rand(7843, 9, 1)
    shannon = np.random.rand(7843, 9, 1)
    flank_h = np.random.rand(7843, 20, 1)
    flank_p = np.random.rand(7843, 400, 1)
    flank_s = np.random.rand(7843, 20, 1)
    property = np.random.rand(7843, 8, 1)

    y_train = np.random.rand(7843)

    x_train = [x_c, x_p, x_h, shannon, flank_h, flank_p, flank_s, property]
    for n in x_train: 
        print(n.shape)
    get_model(x_train, y_train, x_train, y_train, test_params)
`

(In the attatched jupyter notebook you will see that I made a second dictionary, 'test_params', with single values to train my network with.)

When I feed information into the model function without using ta.Scan (call the model function normally), it trains correctly. When I call the model with ta.Scan(), I get this error.

    scan_object = ta.Scan(x=x_train, y=y_train, x_val=x_train, y_val = y_train, model=get_model, params=grid_params, experiment_name=filename)
  File "/usr/local/lib/python3.6/dist-packages/talos/scan/Scan.py", line 196, in __init__
    scan_run(self)
  File "/usr/local/lib/python3.6/dist-packages/talos/scan/scan_run.py", line 34, in scan_run
    self = logging_finish(self)
  File "/usr/local/lib/python3.6/dist-packages/talos/logging/logging_finish.py", line 6, in logging_finish
    self = result_todf(self)
  File "/usr/local/lib/python3.6/dist-packages/talos/logging/results.py", line 46, in result_todf
    cols = self.result[0]
IndexError: list index out of range

IndexError: list index out of range

I looked in the traceback and my hypothesis is that the While loop on line 16 of talos/scan/scan_run.py gets skipped over, because self.round_params is false. I don't have a way to verify that this is true.

Here's a jupyter notebook that should explain what my problem is. Run train.py to try a single shot example, and train_grid.py to see what happens when we try it with talos scan. https://drive.google.com/file/d/1_ycfuhxpQvho6Ws6NUp-5dY5wtOxVNFu/view?usp=sharing

Thank you for your time! EDIT: Since my model has a LOT (4E41) permutations, I obviously want to limit this search breadth. So when I set my fraction_limit to any number > 0 and < 1, I get this error

<ipython-input-51-e2a47d3a5ea2> in train_grid()
     15     scan_object = ta.Scan(x=x_train, y=y_train, x_val=x_train,
     16                           y_val = y_train,
---> 17                           fraction_limit = 0.0000001, model=get_model, params=grid_params, experiment_name = 'Multiheaded input test')
     18     ta.Analyze(scan_object)
     19 

/usr/local/lib/python3.6/dist-packages/talos/scan/Scan.py in __init__(self, x, y, params, model, experiment_name, x_val, y_val, val_split, random_method, seed, performance_target, fraction_limit, round_limit, time_limit, boolean_limit, reduction_method, reduction_interval, reduction_window, reduction_threshold, reduction_metric, minimize_loss, disable_progress_bar, print_params, clear_session, save_weights)
    194         # start runtime
    195         from .scan_run import scan_run
--> 196         scan_run(self)

/usr/local/lib/python3.6/dist-packages/talos/scan/scan_run.py in scan_run(self)
      7 
      8     from .scan_prepare import scan_prepare
----> 9     self = scan_prepare(self)
     10 
     11     # initiate the progress bar

/usr/local/lib/python3.6/dist-packages/talos/scan/scan_prepare.py in scan_prepare(self)
     28                                    round_limit=self.round_limit,
     29                                    time_limit=self.time_limit,
---> 30                                    boolean_limit=self.boolean_limit
     31                                    )
     32 

/usr/local/lib/python3.6/dist-packages/talos/parameters/ParamSpace.py in __init__(self, params, param_keys, random_method, fraction_limit, round_limit, time_limit, boolean_limit)
     39 
     40         # apply all the set limits
---> 41         self.param_index = self._param_apply_limits()
     42 
     43         # create the parameter space

/usr/local/lib/python3.6/dist-packages/talos/parameters/ParamSpace.py in _param_apply_limits(self)
     91             return sample_reducer(self.fraction_limit,
     92                                   self.dimensions,
---> 93                                   self.random_method)
     94 
     95         # a round limit is set

/usr/local/lib/python3.6/dist-packages/talos/reducers/sample_reducer.py in sample_reducer(limit, max_value, random_method)
     32     from ..utils.exceptions import TalosDataError
     33     if n < 1:
---> 34         raise TalosDataError("Limiters lead to < 1 permutations.")
     35 
     36     # Initialize Randomizer()

TalosDataError: Limiters lead to < 1 permutations.

github-actions[bot] commented 4 years ago

Welcome to Talos community! Thanks so much for creating your first issue :)

mikkokotila commented 4 years ago

TalosDataError: Limiters lead to < 1 permutations.

The error itself says what the problem is. Once you apply your limiters, you are left with less than 1 permutations so no experiment can be performed (because there is nothing to experiment).

Generally speaking for any permutation space within 10^10, the way to do it is to use round_limit instead of fraction_limit. For any parameter space with more than 10^10 permutations, this falls under "unintended use".

Closing this as resolved.

autonomio / talos