After saving and loading, the accuracy of the model is incorrect

[tianjiashuo]

Description

I am training a lstm-trec classification model. This is my training code.

import os
#os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2"
os.environ["KERAS_BACKEND"] = 'mxnet'
import pandas as pd
import warnings
import re
import matplotlib.pyplot as plt
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords
import numpy as np
np.random.seed(1234)
import keras.backend as K
print(f"Using {K.backend()} as backend")
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
from keras.layers import Dense, LSTM, Embedding, Dropout, Conv1D, MaxPooling1D, Bidirectional
from keras.models import Sequential

warnings.filterwarnings('ignore')
import keras
import numpy as np

max_features = 55
maxlen = 140
embed_size = 128

model_type = 'lstm'
save_dir = os.path.join(os.getcwd(), 'trec')
model_name = 'trec_%s_model_vgg.{epoch:03d}.h5' % model_type
if not os.path.isdir(save_dir):
    os.makedirs(save_dir)
filepath = os.path.join(save_dir, model_name)
from keras.callbacks import ModelCheckpoint
checkpoint = ModelCheckpoint(filepath=filepath,monitor='val_accuracy',verbose=1,save_best_only=True)
callbacks = [checkpoint]

if os.path.exists('trec_test.h5'):
  print("remove previous model weights")
  os.remove('trec_test.h5')

import numpy as np

x_train=np.load('trec_train_x_char.npy')
y_train=np.load('trec_train_y_char.npy')

x_test=np.load('trec_test_x_char.npy')
y_test=np.load('trec_test_y_char.npy')

x_validation=np.load('trec_validation_x_char.npy')
y_validation=np.load('trec_validation_y_char.npy')

y_train = keras.utils.to_categorical(y_train)
y_validation = keras.utils.to_categorical(y_validation)
y_test = keras.utils.to_categorical(y_test)

def get_lstm_model(max_features, embed_size):
    model = Sequential()
    model.add(Embedding(max_features, embed_size))
    model.add(Bidirectional(LSTM(128, recurrent_dropout=0.1)))
    #model.add(LSTM(128, recurrent_dropout=0.1))
    model.add(Dropout(0.25))
    model.add(Dense(64))
    model.add(Dropout(0.3))
    model.add(Dense(6, activation='softmax'))
    model.summary()

    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['acc'])

    return model

def model_fit(model, x, y, xv, yv):
    return model.fit(x, y, batch_size=100, epochs=3, validation_data=(xv,yv),callbacks=callbacks)

model = get_lstm_model(max_features, embed_size)
model_train = model_fit(model, x_train, y_train, x_validation, y_validation)

def model_predict(model, x):
    return model.predict_classes(x)

score = model.evaluate(x_test,y_test)
model.save('trec_test.h5')
print('acc:',score[1])
print('after load')
load_model = keras.models.load_model('trec_test.h5')
load_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['acc'])
score_1 = load_model.evaluate(x_test,y_test)
print(score_1[1])

Error Message

Using MXNet backend
/usr/local/lib/python3.7/dist-packages/keras/__init__.py:31: DeprecationWarning: MXNet support in Keras is going to be discontinued and v2.2.4.3 is the last release as multi-backend Keras has been discontinued . It is recommended to consider switching to MXNet Gluon. More information can be found here: https://github.com/awslabs/keras-apache-mxnet
  "https://github.com/awslabs/keras-apache-mxnet", DeprecationWarning)
Using mxnet as backend
remove previous model weights
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, None, 128)         7040      
_________________________________________________________________
bidirectional_1 (Bidirection (None, 256)               263168    
_________________________________________________________________
dropout_1 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 64)                16448     
_________________________________________________________________
dropout_2 (Dropout)          (None, 64)                0         
_________________________________________________________________
dense_2 (Dense)              (None, 6)                 390       
=================================================================
Total params: 287,046
Trainable params: 287,046
Non-trainable params: 0
_________________________________________________________________
Train on 5000 samples, validate on 452 samples
Epoch 1/3
5000/5000 [==============================] - 91s 18ms/step - loss: 1.6928 - acc: 0.2158 - val_loss: 1.6423 - val_acc: 0.2478
Epoch 2/3
5000/5000 [==============================] - 88s 18ms/step - loss: 1.6555 - acc: 0.2300 - val_loss: 1.6301 - val_acc: 0.2611
Epoch 3/3
5000/5000 [==============================] - 87s 17ms/step - loss: 1.6420 - acc: 0.2418 - val_loss: 1.6300 - val_acc: 0.2544
500/500 [==============================] - 6s 12ms/step
acc: 0.3080000004768372
after load
500/500 [==============================] - 5s 9ms/step
0.1880000002384186

To Reproduce

https://colab.research.google.com/drive/1J7Kd7cuBe1fnnx6p9TVOZ1_T93vI0kZx?usp=sharing#scrollTo=p_1SQCyjK-29

[github-actions[bot]]

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue. Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly. If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.

[leezu]

FileNotFoundError: [Errno 2] No such file or directory: 'trec_train_x_char.npy'