When load my trained model later, the predictions are messed up

[Jay-IPL]

Hi,

I trained on my own dataset by using the same code as 'cars segmentation (camvid).ipynb'.

By using the same code on my own dataset, I got my trained model and test it on test dataset, the result is good！ However, after several days, I used the same model to test on the same test data, the predictions are messed up. Here is how I run the code: I run code on Jupiter notebook and didn't close it for those days. I just restart the kernel and run all codes again except the training part to do the test.

The training part I didn't run is:

train model for 40 epochs

max_score = 0

for i in range(0, 100):

print('\nEpoch: {}'.format(i))
train_logs = train_epoch.run(train_loader)
valid_logs = valid_epoch.run(valid_loader)

# do something (save model, change lr, etc.)
if max_score < valid_logs['iou_score']:
    max_score = valid_logs['iou_score']
    torch.save(model, './best_model.pth')
    print('Model saved!')

if i == 25:
    optimizer.param_groups[0]['lr'] = 1e-5
    print('Decrease decoder learning rate to 1e-5!')

Do you have any ideas why did this happen? Thanks！

[Progern]

Just make sure that you are using the same data preprocessing, that you used in the training phase. It's an often problem, that the model could not figure out information because of the normalization.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 7 days with no activity.