ram issue

[talhaanwarch]

My system has 64 GB ram, before starting the training process the ram occupied is 10 gb. but as soon as data passes to scikeras the ram reaches to 64 gb. Though i think the model is trained on gpu, why ram memory is getting increased

[adriangb]

SciKeras does not use memory in a way that scales with the model / dataset size (aside from input transformers, but those scale with number of classes in the target for classifiers or O(1) for regressors). SciKeras definetely does not control where training happens (CPU vs GPU). Can you try the following please?

clf = KerasClassifier(...)
clf.fit(X_batch_1, y_batch_1, epochs=1)  # or some smaller subset of your dataset, this should be very fast
keras_model = clf.model_
keras_model.fit(...)  # fit the Keras model directly

The goal of this is to see if the memory usage / issues you are seeing are related to SciKeras or your model / Keras.

[talhaanwarch]

It say AttributeError: 'KerasClassifier' object has no attribute 'model_'

[adriangb]

That's strange. That attribute should definitely be set after calling KerasClassifier.fit since it is used by Keras Classifier.fit :https://github.com/adriangb/scikeras/blob/57918d67115684250986ffa654e9a09321e9f383/scikeras/wrappers.py#L503

[talhaanwarch]

Please check the code

sk_clf = KerasClassifier(ourmodel,epochs=50,random_state=0,batch_size=1024*4,verbose=False)
pipe_clf=Pipeline([('scaler',StandardScaler3D()),('classifier',sk_clf.model_)])
pipe_clf.fit(data_array, label_array)

[adriangb]

I believe sklearn clones the model before fitting in a pipeline. So if you are accessing sk_clf then yes you'll get an attribute error. You'll need to access the trained estimator inside the pipeline: pipe_clf[-1].model_.

It might also be a good idea to try to debug these things outside of a pipeline. Doing these sorts of things in a pipeline adds another confounding factor.

[talhaanwarch]

I tried to make it simpler by making dummy data, but still it throwing same error. Here is the code

def ourmodel(numberOfLSTMcells=2,fch=64,fc1h=32,opt_lr=1e-3,n_timesteps_in=3000,n_features=61):
    clear_session()
    inp =Input(shape=(n_timesteps_in, n_features))
    lstm= LSTM(numberOfLSTMcells,return_sequences=True, return_state=False) (inp)
    flatten=Flatten()(lstm)
    fc=Dense(fch,activation='relu')(flatten)
    fc=Dense(fc1h,activation='relu')(fc)
    out=Dense(1)(fc)
    out = Activation('sigmoid', dtype='float32')(out)

    model = Model(inputs=inp, outputs=out)
    model.compile(loss='binary_crossentropy', optimizer=Adam(lr=opt_lr), 
                  metrics=['accuracy'])
    return model

import numpy as np
data_array=np.random.rand(10000, 3000, 61).astype('float32')
label_array=np.concatenate((np.zeros(5000),np.ones(5000)))
data_array.shape,label_array.shape,data_array.dtype

sk_clf = KerasClassifier(ourmodel,epochs=50,random_state=0,batch_size=1024*4,verbose=False)
# sk_clf.fit(data_array, label_array)

keras_model = sk_clf.model_
keras_model.fit(data_array, label_array)

[adriangb]

This line is commented out. You are not training or initializing the Keras model: # sk_clf.fit is commented out

[talhaanwarch]

what i think if need to test keras model which is nested in scikeras, i dont need to train a scikeras model. but look like the issue is with my batch size. I was assuming, that my data is around 10 gb and the ram should not go up to 64 gb while model training, the gpu should go to 10 gb or more. But without using scikeras, even with keras, my ram usage is increased upto 64. So, i need to look into may be creating a dataloader.

Thanks for your time

[adriangb]

Glad you could figure out the issue. These are tricky things to debug!