ResNet152 not working properly

[AnabelGRios]

I am trying to use the ResNet152 model from the keras applications module. I have installed the latest version of keras from github, so I have keras applications 1.0.7. I am using tensorflow as backend, version 1.12 (though I have also tried TF 1.9 and the results were similar). Cuda toolkit version: 9.2 Cudnn version: 7.2.1 Python version: 3.6.7

I am using ResNet50 and ResNet152 on two multi-class datasets and in both cases the ResNet50 model works fine, but the ResNet152 model does not converge: accuracy and loss jump around one value (usually more than 0.9 for accuracy). Then, test accuracy is around 0.4 (for 14 classes) . This is weird for me because I was using another ResNet152 model (the one from here) and it worked fine in both datasets. I use transfer learning from imagenet and I only train two newly added layers. I have simplified the code and change the dataset to cifar10 so it can be reproducible. The problem I have found is that it seems to work fine for cifar10.

from keras.applications.resnet import ResNet50, ResNet152, preprocess_input
from keras.datasets import cifar10
from keras.utils import np_utils
from keras.models import Sequential
from keras.layers.core import Dense
from keras.optimizers import SGD
import numpy as np
import cv2

(x_train, y_train), (x_test, y_test) = cifar10.load_data()

x_train = np.array([preprocess_input(cv2.resize(img, (224, 224))) for img in x_train])
x_test = np.array([preprocess_input(cv2.resize(img, (224, 224))) for img in x_test])

y_train = np_utils.to_categorical(y_train, 10)
y_test = np_utils.to_categorical(y_test, 10)

# model = ResNet50(include_top = False, weights='imagenet', pooling = 'avg')
model = ResNet152(include_top = False, weights='imagenet', pooling = 'avg')

train_features = model.predict(x_train, batch_size = 32)
test_features = model.predict(x_test, batch_size = 32)

new_model = Sequential()
new_model.add(Dense(512, activation = 'relu' , input_shape = train_features.shape[1:]))
new_model.add(Dense(10, activation = 'softmax'))

opt = SGD(lr=1e-3, decay=1e-6, momentum=0.9, nesterov=True)
new_model.compile(optimizer=opt, loss='categorical_crossentropy',
                            metrics=['accuracy'])
new_model.fit(train_features, y_train, epochs = 20, batch_size = 32)

predictions = new_model.predict(test_features, batch_size = 32, verbose = 1)

y_test = np.argmax(y_test, axis = 1)
predictions = np.argmax(predictions, axis = 1)
print(sum(y_test == predictions)/len(y_test))

The number of epochs is only 20 so it does not take a long time to run the code. I have increased this number and the results are similar.

Is there anything wrong with the code or is it the ResNet152 model from keras?

Thank you so much.

[taehoonlee]

Thank you for the reports, @AnabelGRios. I checked the followings:

1. The reproducibility of the performance table: PASS
2. The equivalence between include_top=True and include_top=False: PASS

import numpy as np
from keras import backend as K
from keras.applications.resnet import ResNet50, ResNet152, preprocess_input

model1 = ResNet50()
model2 = ResNet152()
model3 = ResNet50(include_top=False, pooling='avg')
model4 = ResNet152(include_top=False, pooling='avg')

x = np.random.random((1, 224, 224, 3)).astype(np.float32)

y1 = K.function([model1.layers[0].input], [model1.layers[-2].output])([x])[0]
y2 = K.function([model2.layers[0].input], [model2.layers[-2].output])([x])[0]

y3 = model3.predict(x)
y4 = model4.predict(x)

print(np.sum((y1 - y3) ** 2))
print(np.sum((y2 - y4) ** 2))

0.0
0.0

3. The reproducibility of your problem: FAIL
   • I ran the shared codes, but both ResNet50 and ResNet152 showed similar results (train/test accuracy 0.4/0.1 for both the models). I think the codes are not perfect to reproduce your problem, because the 8x up-sampling is unrealistic. The 8x up-sampled CIFAR images have totally different distribution from the ImageNet. The convergence rate and the quality of bottle-neck representations will depend on your data.

[taehoonlee]

@AnabelGRios, Do you have any progress?

[AnabelGRios]

@taehoonlee, Yes. I have found that if I read and preprocess the images with the function flow_from_directory() from ImageDataGenerator instead of using opencv (besides the prepocess_input() function from resnet), ResNet152 works fine. However, if I use opencv, I get the same behaviour (accuracy jumping around 98% on training and going to 40% on test).

Do you know what might be going on?

Thank you.

[taehoonlee]

@AnabelGRios, One of the possible scenarios is that you are using preprocess_input twice. The preprocess_input is an in-place operation, thus should be applied only once. Otherwise, could you share your preprocessing codes?

[AnabelGRios]

@taehoonlee, I think I have found the problem. I have a script where I choose from command line which network to use (among ResNet50, ResNet152 and InceptionV3). To do so, I always import the three networks:

from keras.applications.resnet import ResNet50, ResNet152
from keras.applications.inception_v3 import InceptionV3

I was simplifying the script so I could put it here and I have noticed that when I have removed the InceptionV3 import, ResNet152 works fine. This was also a difference between my script reading the images using opencv and using the function flow_from_directory(), but I didn't notice it because I was only using ResNet152.

Do you know why this is happening?

[taehoonlee]

@taehoonlee, from keras.applications.inception_v3 import InceptionV3 can't affect preprocess_input or any namespaces. If you want to use different modules, you should import individual preprocess_inputs as follows:

from keras.applications.resnet import ResNet50, ResNet152
from keras.applications.inception_v3 import InceptionV3

from keras.applications.resnet import preprocess_input as p1
from keras.applications.inception_v3 import preprocess_input as p2

img1 = p1(img.copy())  # you should copy because p1 and p2 are in-place operations.
img2 = p2(img.copy())

[taehoonlee]

@AnabelGRios, I hope you solved your problem. I'll close the issue for now. Please feel free to open it again at any time if you have additional comments.