search

NoamRosenberg / autodeeplab

AutoDeeplab / auto-deeplab / AutoML for semantic segmentation, implemented in Pytorch
MIT License
309 stars 73 forks source link

At what time should i stop the searching and begin retraining my model? #88

Open yuncheng97 opened 4 years ago

yuncheng97 commented 4 years ago

thanks lot for your wonderful efforts! but I have some problems when implementing this project I run the code for pascal and cityscapes with the default args, with one gpu RTX 2080Ti, but the result I got at epoch 40 is bad, pascal voc not even reach 20% miou and cityscapes also just around 20%. I'm so confused. I saw your introduction that there should be 3 stages of searching, decoding and re-training. the re-training stage means I should start re-training until the search result reached ~79% or I should start re-training after ~40 epochs and re-train the model for the best result? looking forward to your reply. thanks again.

mrluin commented 4 years ago

Just start to retrain after the ~40 epochs searching. searching for 40 epochs -> retrain from scratch

yuncheng97 commented 4 years ago

Just start to retrain after the ~40 epochs searching. searching for 40 epochs -> retrain from scratch

thanks! I will try this to see if it works

cardwing commented 4 years ago

@Oliver-jiang Can you share your retraining result of the searched architecture with us? Thanks a lot.

cardwing commented 4 years ago

@Oliver-jiang Since RTX 2080Ti only has 11GB memory, you may have banned global pooling in ASPP. That may be the reason for the bad performance.

yuncheng97 commented 4 years ago

I so sorry that i forgot to check my mail for a long time. The problem of the project autodeeplab is still there, but i have not changed any thing in ASPP module. Could you please explain more about this? thanks a lot. In my implement, I only resized the input image to 128 and filter_multiplier to 4 to satisfy the memory. the result is stuck around 25% on pascal now.

------------------ 原始邮件 ------------------ 发件人: "HOU Yuenan"<notifications@github.com>; 发送时间: 2020年1月29日(星期三) 上午10:39 收件人: "NoamRosenberg/AutoML"<AutoML@noreply.github.com>; 抄送: "蒋云丞"<jyc97@qq.com>;"Mention"<mention@noreply.github.com>; 主题: Re: [NoamRosenberg/AutoML] At what time should i stop the searching and begin retraining my model? (#88)

@Oliver-jiang Since RTX 2080Ti only has 11GB memory, you may have banned global pooling in ASPP. That may be the reason for the bad performance.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

yuncheng97 commented 4 years ago

@cardwing can you share some of your search_argparses with us? Thanks a lot.

cardwing commented 4 years ago

Currently, I have reduced the number of layers from 12 to 10 due to the limited memory and kept other hyperparameters fixed. The searching performance on the Cityscapes validation set is around 33% mIoU after 40 epochs.

yuncheng97 commented 4 years ago

@cardwing thanks for your advice, i tried your method but currently the best performance after 7 epochs is around 7% and there was a big fluctuation in some epochs. however, the performance at begin shown in the paper is already around 10% mIoU(1% in my experiment). did you have the same problem? looking forward to your reply

cardwing commented 4 years ago

I have not met that problem. You can just train the model for more epochs and check the result. However, it takes me around 7 days to train the searched architecture and the final performance is only around 65% mIoU. I have to admit that NAS really consumes resources and time.

chendi23 commented 1 year ago

I have not met that problem. You can just train the model for more epochs and check the result. However, it takes me around 7 days to train the searched architecture and the final performance is only around 65% mIoU. I have to admit that NAS really consumes resources and time.

Same. around 65%