Registering with Keras fails

[martinlie]

I have tried several combinations (Docker and other PC and environment), but the following fails

import deepkit
...
experiment = deepkit.experiment()
experiment.watch_keras_model(model)
deepkit_callback = experiment.create_keras_callback(model)
...
model.fit(generator,epochs=90, callbacks=[deepkit_callback])

with

Traceback (most recent call last):
  File "forecast-with-lstm.py", line 58, in <module>
    model.fit(generator,epochs=90,callbacks=[deepkit_callback])
  File "/usr/local/lib/python3.8/site-packages/deepkit/experiment.py", line 532, in fit
    debugger.set_input(x)
  File "/usr/local/lib/python3.8/site-packages/deepkit/keras_tf.py", line 433, in set_input
    self.model_input = np.array([x[0]] if self.is_batch else x)
ValueError: could not broadcast input array from shape (6,12,1) into shape (6)

Running Python 3.8.3 (on OSX) and these dependencies from requirements.txt:

deepkit
pandas
matplotlib
statsmodels
sklearn
keras
scipy==1.4.1
tensorflow

Any clue of what goes wrong?

[martinlie]

I suspect the network is the problem, but I think it should be supported:

n_input = 12
n_features = 1
generator = TimeseriesGenerator(train, train, length=n_input, batch_size=6)

model = Sequential()
model.add(LSTM(200, activation='relu', input_shape=(n_input, n_features)))
model.add(Dropout(0.15))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

experiment.watch_keras_model(model)
deepkit_callback = experiment.create_keras_callback(model)

model.fit(generator,epochs=90, callbacks=[deepkit_callback])

[marcj]

Mh, yes seems so. It can not automatically infer the input for the debugger. You can try to work around that by providing manually the model input.

experiment.watch_keras_model(model, model_input=x, is_batch=True)

is_batch needs to be True or False, depending on your network structure. model_input is the input (batched or a simple entry) of your network input aka x.

[marcj]

Since you have a generator, you could try

experiment.watch_keras_model(model, model_input=next(iter(generator)), is_batch=True)

[marcj]

Btw, since you already manually called watch_keras_model, you don't have to pass it again to create_keras_callback, so keep this line like that:

deepkit_callback = experiment.create_keras_callback()

[martinlie]

This seems to be an improvement, but when I combine and do:

experiment.watch_keras_model(model, model_input=next(iter(generator)), is_batch=True)
deepkit_callback = experiment.create_keras_callback()
model.fit(generator,epochs=90,callbacks=[deepkit_callback])

the program throws the exception:

  File "/usr/local/lib/python3.8/site-packages/tensorflow/python/keras/callbacks.py", line 601, in on_train_batch_begin
    self.on_batch_begin(batch, logs=logs)
  File "/usr/local/lib/python3.8/site-packages/deepkit/deepkit_keras.py", line 117, in on_batch_begin
    batch_size = logs['size']
KeyError: 'size'
Ended task main # 0 exitCode 1

If I omit the callback, the script runs with the generator, however it would be nice to get real time updates and accuracy reports etc. Any tips?

[marcj]

Which Keras version do you use? It seems it does not report the size log entry in its on_batch_begin hook.

[marcj]

I pushed a change to master which should fix this issue. Please try it (by installing master on your computer or simply copy&paste its file content). When you provide me a reproduction code I can integrate in a unit test.

[martinlie]

This is nice - the fix works! And the GUI is beautiful! Keras version 2.4.3. Btw, how is the Python package release schedule? I would like to run containerised, so the library will be downloaded on each build. The fix would be nice to get pushed to the registry.

[marcj]

I just released 1.0.8 with the fix. Thanks!