search

keras-team / autokeras

AutoML library for deep learning
http://autokeras.com/
Apache License 2.0
9.14k stars 1.4k forks source link

Error when using TimeSeriesForecaster #1720

Open peter8995 opened 2 years ago

peter8995 commented 2 years ago

Bug Description

Hello, I have issue while using TimeSeriesForecaster to forecast my data.

Bug Reproduction

Code for reproducing the bug:

import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
import autokeras as ak

dataset = pd.read_csv('./Fog.csv')
dataset.drop(dataset.index[dataset[(dataset.values == 'X') | (dataset.values == 'V')].index], inplace=True)
dataset = dataset.drop(['time','Label'],axis=1)
dataset = dataset.astype("float64")
dataset.info()

fig = plt.figure(figsize=(10, 12))

names = ('VIZ', 'temp', 'dpTemp', 'RH', 'WDIR')

for i in range(5):
    data = dataset[names[i]]
    ax = fig.add_subplot(5, 1, i + 1)
    plt.scatter(np.arange(data.size), data, s=1)
    ax.set_xlim([0, data.size])
    ax.set_title(names[i])

plt.tight_layout()
plt.show()

split = 0.125

x = dataset.drop(['VIZ'], axis=1)
y = dataset['VIZ']

slice_index = int(y.size * (1 - split))
x_train, x_test = x[:slice_index], x[slice_index:]
y_train, y_test = y[:slice_index], y[slice_index:]

x_train.info()
x_test.info()
print(x_train.shape)
print(x_test.shape)

lookback = 24
batch_size = 72

fog = ak.TimeseriesForecaster(
    lookback=lookback, max_trials=100, overwrite=True)

fog.fit(
    x_train, y_train, batch_size=batch_size,
    callbacks=[tf.keras.callbacks.EarlyStopping(patience=3)])

Data used by the code: https://github.com/peter8995/Algorithm-and-Data-Processing/blob/master/Final/Fog.csv

Full Error Message


Search: Running Trial #1

Value             |Best Value So Far |Hyperparameter
True              |?                 |timeseries_block_1/rnn_block_1/bidirectional
lstm              |?                 |timeseries_block_1/rnn_block_1/layer_type
2                 |?                 |timeseries_block_1/rnn_block_1/num_layers
0                 |?                 |regression_head_1/dropout
adam              |?                 |optimizer
0.001             |?                 |learning_rate

Epoch 1/1000
    190/Unknown - 11s 22ms/step - loss: 363.5932 - mean_squared_error: 363.5932
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-75-5aa769790503> in <module>
      5     lookback=lookback, max_trials=100, overwrite=True)
      6 
----> 7 fog.fit(
      8     x_train, y_train, batch_size=batch_size,
      9     callbacks=[tf.keras.callbacks.EarlyStopping(patience=3)])

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\tasks\time_series_forecaster.py in fit(self, x, y, validation_split, validation_data, **kwargs)
    266                 [keras.Model.fit](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit).
    267         """
--> 268         super().fit(
    269             x=x,
    270             y=y,

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\tasks\time_series_forecaster.py in fit(self, x, y, epochs, callbacks, validation_split, validation_data, **kwargs)
     86             validation_data = x_val, y_val
     87 
---> 88         history = super().fit(
     89             x=x[: self.train_len],
     90             y=y[self.lookback - 1 :],

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\tasks\structured_data.py in fit(self, x, y, epochs, callbacks, validation_split, validation_data, **kwargs)
    137         self.check_in_fit(x)
    138 
--> 139         history = super().fit(
    140             x=x,
    141             y=y,

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\auto_model.py in fit(self, x, y, batch_size, epochs, callbacks, validation_split, validation_data, verbose, **kwargs)
    290             )
    291 
--> 292         history = self.tuner.search(
    293             x=dataset,
    294             epochs=epochs,

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\engine\tuner.py in search(self, epochs, callbacks, validation_split, verbose, **fit_kwargs)
    191         self._try_build(hp)
    192         self.oracle.update_space(hp)
--> 193         super().search(
    194             epochs=epochs, callbacks=new_callbacks, verbose=verbose, **fit_kwargs
    195         )

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras_tuner\engine\base_tuner.py in search(self, *fit_args, **fit_kwargs)
    177 
    178             self.on_trial_begin(trial)
--> 179             results = self.run_trial(trial, *fit_args, **fit_kwargs)
    180             # `results` is None indicates user updated oracle in `run_trial()`.
    181             if results is None:

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras_tuner\engine\tuner.py in run_trial(self, trial, *args, **kwargs)
    292             callbacks.append(model_checkpoint)
    293             copied_kwargs["callbacks"] = callbacks
--> 294             obj_value = self._build_and_fit_model(trial, *args, **copied_kwargs)
    295 
    296             histories.append(obj_value)

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\engine\tuner.py in _build_and_fit_model(self, trial, *args, **kwargs)
     99         self.adapt(model, kwargs["x"])
    100 
--> 101         _, history = utils.fit_with_adaptive_batch_size(
    102             model, self.hypermodel.batch_size, **kwargs
    103         )

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\utils\utils.py in fit_with_adaptive_batch_size(model, batch_size, **fit_kwargs)
     86 
     87 def fit_with_adaptive_batch_size(model, batch_size, **fit_kwargs):
---> 88     history = run_with_adaptive_batch_size(
     89         batch_size, lambda **kwargs: model.fit(**kwargs), **fit_kwargs
     90     )

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\utils\utils.py in run_with_adaptive_batch_size(batch_size, func, **fit_kwargs)
     99     while batch_size > 0:
    100         try:
--> 101             history = func(x=x, validation_data=validation_data, **fit_kwargs)
    102             break
    103         except tf.errors.ResourceExhaustedError as e:

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\utils\utils.py in <lambda>(**kwargs)
     87 def fit_with_adaptive_batch_size(model, batch_size, **fit_kwargs):
     88     history = run_with_adaptive_batch_size(
---> 89         batch_size, lambda **kwargs: model.fit(**kwargs), **fit_kwargs
     90     )
     91     return model, history

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tensorflow\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     52   try:
     53     ctx.ensure_initialized()
---> 54     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     55                                         inputs, attrs, num_outputs)
     56   except core._NotOkStatusException as e:

InvalidArgumentError: Graph execution error:

Detected at node 'mean_squared_error/SquaredDifference' defined at (most recent call last):
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\runpy.py", line 194, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\ipykernel_launcher.py", line 16, in <module>
      app.launch_new_instance()
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\traitlets\config\application.py", line 664, in launch_instance
      app.start()
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\ipykernel\kernelapp.py", line 612, in start
      self.io_loop.start()
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tornado\platform\asyncio.py", line 149, in start
      self.asyncio_loop.run_forever()
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\asyncio\base_events.py", line 570, in run_forever
      self._run_once()
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\asyncio\base_events.py", line 1859, in _run_once
      handle._run()
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\asyncio\events.py", line 81, in _run
      self._context.run(self._callback, *self._args)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tornado\ioloop.py", line 690, in <lambda>
      lambda f: self._run_callback(functools.partial(callback, future))
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tornado\ioloop.py", line 743, in _run_callback
      ret = callback()
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tornado\gen.py", line 787, in inner
      self.run()
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tornado\gen.py", line 748, in run
      yielded = self.gen.send(value)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\ipykernel\kernelbase.py", line 365, in process_one
      yield gen.maybe_future(dispatch(*args))
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tornado\gen.py", line 209, in wrapper
      yielded = next(result)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\ipykernel\kernelbase.py", line 268, in dispatch_shell
      yield gen.maybe_future(handler(stream, idents, msg))
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tornado\gen.py", line 209, in wrapper
      yielded = next(result)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\ipykernel\kernelbase.py", line 543, in execute_request
      self.do_execute(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\tornado\gen.py", line 209, in wrapper
      yielded = next(result)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\ipykernel\ipkernel.py", line 306, in do_execute
      res = shell.run_cell(code, store_history=store_history, silent=silent)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\ipykernel\zmqshell.py", line 536, in run_cell
      return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\IPython\core\interactiveshell.py", line 2866, in run_cell
      result = self._run_cell(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\IPython\core\interactiveshell.py", line 2895, in _run_cell
      return runner(coro)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\IPython\core\async_helpers.py", line 68, in _pseudo_sync_runner
      coro.send(None)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\IPython\core\interactiveshell.py", line 3071, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\IPython\core\interactiveshell.py", line 3263, in run_ast_nodes
      if (await self.run_code(code, result,  async_=asy)):
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\IPython\core\interactiveshell.py", line 3343, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "<ipython-input-75-5aa769790503>", line 7, in <module>
      fog.fit(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\tasks\time_series_forecaster.py", line 268, in fit
      super().fit(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\tasks\time_series_forecaster.py", line 88, in fit
      history = super().fit(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\tasks\structured_data.py", line 139, in fit
      history = super().fit(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\auto_model.py", line 292, in fit
      history = self.tuner.search(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\engine\tuner.py", line 193, in search
      super().search(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras_tuner\engine\base_tuner.py", line 179, in search
      results = self.run_trial(trial, *fit_args, **fit_kwargs)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras_tuner\engine\tuner.py", line 294, in run_trial
      obj_value = self._build_and_fit_model(trial, *args, **copied_kwargs)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\engine\tuner.py", line 101, in _build_and_fit_model
      _, history = utils.fit_with_adaptive_batch_size(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\utils\utils.py", line 88, in fit_with_adaptive_batch_size
      history = run_with_adaptive_batch_size(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\utils\utils.py", line 101, in run_with_adaptive_batch_size
      history = func(x=x, validation_data=validation_data, **fit_kwargs)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\autokeras\utils\utils.py", line 89, in <lambda>
      batch_size, lambda **kwargs: model.fit(**kwargs), **fit_kwargs
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\engine\training.py", line 1384, in fit
      tmp_logs = self.train_function(iterator)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\engine\training.py", line 1021, in train_function
      return step_function(self, iterator)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\engine\training.py", line 1010, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\engine\training.py", line 1000, in run_step
      outputs = model.train_step(data)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\engine\training.py", line 860, in train_step
      loss = self.compute_loss(x, y, y_pred, sample_weight)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\engine\training.py", line 918, in compute_loss
      return self.compiled_loss(
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\engine\compile_utils.py", line 201, in __call__
      loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\losses.py", line 141, in __call__
      losses = call_fn(y_true, y_pred)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\losses.py", line 245, in call
      return ag_fn(y_true, y_pred, **self._fn_kwargs)
    File "c:\ProgramData\Anaconda3\envs\tfenv\lib\site-packages\keras\losses.py", line 1329, in mean_squared_error
      return backend.mean(tf.math.squared_difference(y_pred, y_true), axis=-1)
Node: 'mean_squared_error/SquaredDifference'
required broadcastable shapes
     [[{{node mean_squared_error/SquaredDifference}}]] [Op:__inference_train_function_61209]

Expected Behavior

Setup Details

Include the details about the versions of:

Additional context

haifeng-jin commented 2 years ago

The TimeSeriesForecaster is still in experimental stage. We will take a look into this as we improve it. Thanks for the report!

eschibli commented 2 years ago

This is Issue 1554. It seems possible to work around it by setting the batch size so that lookback is divisible by batch_size.

peter8995 commented 2 years ago

Thanks, after I set lookback divisible by batch_size it works.