stared / livelossplot

Live training loss plot in Jupyter Notebook for Keras, PyTorch and others
https://p.migdal.pl/livelossplot
MIT License
1.29k stars 143 forks source link

No plots showing for utils.Sequence Generator model #133

Closed crocokyle closed 3 years ago

crocokyle commented 3 years ago

❓ Questions/Help/Support

Hello, I'm very new to keras and tensorflow, so I may be doing something wrong, but I am unable to get any graphs to display at all. There are no errors or warnings, but I just don't see a plot showing up. I'm currently running in Google Colab and have tried the following imports:

from livelossplot import PlotLossesKerasTF
from livelossplot import PlotLossesKeras

I am using a sequence generator like this:

class MyGenerator(tf.keras.utils.Sequence):
    'Generates data for Keras'
    def __init__(self, ids, train_dir):
        'Initialization'
        self.ids = ids
        self.train_dir = train_dir

    def __len__(self):
        'Denotes the number of batches per epoch'
        return len(self.ids)

    def __getitem__(self, index):
        somebs_id = self.ids[index]
        batch_id = 1
        # load data
        X_train, y_train = [], []
        start_index = seq_len*batch_id
        end_index = start_index + seq_len

        for i in range(start_index, end_index): 
            start_seq = i + start_index
            X_train.append(train_data[i-seq_len:i]) 
            y_train.append(train_data[:, 4][i]) 

        # Save our batch
        X = np.array(X_train)
        y = np.array(y_train)
        batch_id += 1
        return X, y

Here is the code for my model:

callback = [PlotLossesKerasTF(),tf.keras.callbacks.ModelCheckpoint('/content/drive/MyDrive/Code/Machine Learning/models/closing.hdf5', 
                                              monitor='val_loss', 
                                              save_best_only=True, verbose=1),WeightsSaver(1)]

history = model.fit(trainGen, 
                    batch_size=batch_size, # batch_size = 32
                    epochs=1, # Default 35
                    callbacks=[callback],
                    validation_data=validationSet)  

model = tf.keras.models.load_model('/content/drive/MyDrive/Code/Machine Learning/models/closing.hdf5',
                                   custom_objects={'Time2Vector': Time2Vector, 
                                                   'SingleAttention': SingleAttention,
                                                   'MultiAttention': MultiAttention,
                                                   'TransformerEncoder': TransformerEncoder})

I'm not sure if I am improperly sending the callback here or if it is another issue. Thanks for your time and help.

crocokyle commented 3 years ago

I also just realized that it may have something to do with how long it takes me to run a single epoch. I believe it's around 200 hours per epoch with my current model. I was under the impression that callbacks get executed after each batch so I didn't think that was an issue at first, but I'm probably missing some information.

stared commented 3 years ago

@crocokyle Thank you for this submission.

Which version of livelossplot do you use?

And could you link to a Collab that replicates the error?

crocokyle commented 3 years ago

Sorry, this was my first attempt at using livelossplot. I didn't realize it is only updated once per epoch and that was my issue. Thanks for your time!