search

hellochick / PSPNet-tensorflow

TensorFlow-based implementation of "Pyramid Scene Parsing Network".
326 stars 123 forks source link

When the network reaches convergence, The loss has been around 2.0 #47

Open Hukongtao opened 5 years ago

Hukongtao commented 5 years ago

python train.py --is-training --update-mean-var --train-beta-gamma to run the network, but when the network reaches convergence, The loss has been around 2.0. How did you get 0.2? @hellochick Thank you very much for your reply.

Hukongtao commented 5 years ago

And the mIOU is 0.323895

hellochick commented 5 years ago

Because you set the flag --update-mean-var, which means that you will update the moving mean and moving variance for batch normalization layer. Such operations need large batch size to update them well. Can you tell me how large your batch size is?

Hukongtao commented 5 years ago

@hellochick I get it.My batch size is 2 because my GPU is limited. So I will try again. Thanks a lot.

Hukongtao commented 5 years ago

@hellochick So I should run by python train.py --is-training or python train.py --is-training --train-beta-gamma

ningscapr commented 5 years ago

Hi, i have the same situation that the loss is about 2.0. Did you use pretrained model? @Hukongtao

Hukongtao commented 5 years ago

@ningscapr I didn't use pretrained model. When I use the pretrained model, the initial loss is 0.2, but the training is done, the loss is about 2.0. I don't know why

waterputty commented 5 years ago

@Hukongtao Hi, have you solved your issue? I followed hellochick's advice del the --update-mean-var but found the loss was still around 2.0 after 90k iterations. I use cityscapes dataset and the batchsize is 2.

Hukongtao commented 5 years ago

@waterputty No, I didn't

waterputty commented 5 years ago

@Hukongtao My latest finding is the loss went down to 1.7 after 150k iterations. I think for the cityscapes it may needs much more iterations than 60k in the code, or the learning rate must be set much bigger than 1e-3 which is also from the original code. But I'm not sure. @hellochick Could you give more advice. Thx a lot.

Hukongtao commented 5 years ago

@waterputty Have you solved your issue? I am still confused

DenceChen commented 5 years ago

@hellochick can you show your train args? your read.me do not have training introduction

mishaelran commented 5 years ago

@denceChen i am also having the same problem my training args are : @waterputty IMG_MEAN = np.array((103.939, 116.779, 123.68), dtype=np.float32)

BATCH_SIZE = 1 DATA_DIRECTORY = './data/cityscapes_dataset/cityscape/' DATA_LIST_PATH = './list/cityscapes_train_list.txt' IGNORE_LABEL = 255 INPUT_SIZE = '713,713' LEARNING_RATE = 1e-3 MOMENTUM = 0.9 NUM_CLASSES = 19 NUM_STEPS = 60001 POWER = 0.9 RANDOM_SEED = 1234 WEIGHT_DECAY = 0.0001 RESTORE_FROM = './' SNAPSHOT_DIR = './model/' SAVE_NUM_IMAGES = 4 SAVE_PRED_EVERY = 20

i cant set up the Batch for less because it wont run even on a 1080TI gpu with 11gb of memory after 60k steps i got 2.0 loss i tried even 150k steps still 1.9 or so. let me know if you guys succeed

AmeetR commented 5 years ago

I'm having the same issue. The best I'm able to get is 1.7 loss and ~.3 mIOU. @hellochick I want to set this as a baseline, so I'd like to get up to the best possible.

zhulf0804 commented 5 years ago

@AmeetR I think It's hard to reproduce this repo. I ran it 1 month ago, debugged it, and failed. Then I code PSPNet myself, and achieved 0.74 mIoU. Maybe you can try to reproduce PSPNet by yourself. Good luck.