bad performance of pytorch model compared to lua

[shuuchen]

Hi,

I compared the model performance of both pytorch and lua. Pytorch model is trained 30 epochs while lua model is provided by the author. The same training data is used.

However, when I test the following image, 33 walls were detected by lua model while only 11 were detected by pytorch model.

The picture is

lua output: (in which 33 walls are detected)

222 121 222 144 door    1   1   

128 161 128 210 door    1   1   

232 77  232 102 door    1   1   

155 96  155 143 door    1   1   

186 68  186 83  door    1   1   

41  67  41  115 door    1   1   

45  48  97  48  door    1   1   

242 48  259 48  door    1   1   

46  227 96  227 door    1   1   

159 48  178 48  door    1   1   

17  145 40  145 door    1   1   

59  145 110 145 door    1   1   

159 147 181 147 door    1   1   

141 227 190 227 door    1   1   

188 50  201 61  washing_basin   1   1   

212 50  229 66  washing_basin   2   1   

203 173 220 226 cooking_counter 1   1   

234 51  273 73  bathtub 1   1   

200 121 220 145 entrance    1   1   

164 52  175 71  toilet  1   1   

2   71.5    52  91.5    closet  1   1   

73  85.5    123 105.5   bedroom 1   1   

145 60  195 80  restroom    1   1   

188.14285714286 82.247619047771 238.14285714286 102.24761904777 washing_room    1   1   

166.52380952371 113.07619047614 216.52380952371 133.07619047614 corridor    1   1   

227.5   65.5    277.5   85.5    bathroom    1   1   

2   120 52  140 closet  1   1   

46  175.5   96  195.5   bedroom 1   1   

148.54046242775 177.54046242775 198.54046242775 197.54046242775 kitchen 1   1   

203 117.666666667   222 117.666666667   wall    1   1   

273.5   47.8666666667   273.5   104 wall    1   1   

232 104 273.5   104 wall    1   1   

232 104 232 117.666666667   wall    1   1   

222 117.666666667   232 117.666666667   wall    1   1   

185.5   47.8666666667   185.5   93  wall    1   1   

155.333333333   93  185.5   93  wall    1   1   

222 156.5   222 227.333333333   wall    1   1   

128 227.333333333   222 227.333333333   wall    1   1   

197.5   156.5   222 156.5   wall    1   1   

197.5   146.733333333   197.5   156.5   wall    1   1   

14  227.333333333   128 227.333333333   wall    1   1   

14  144.6   14  227.333333333   wall    1   1   

13.6666666667   47.6666666667   13.6666666667   116 wall    1   1   

13.6666666667   47.6666666667   41.3333333333   47.6666666667   wall    1   1   

232 47.8666666667   273.5   47.8666666667   wall    1   1   

222 117.666666667   222 146.733333333   wall    1   1   

185.5   47.8666666667   232 47.8666666667   wall    1   1   

155.333333333   47.8666666667   185.5   47.8666666667   wall    1   1   

232 47.8666666667   232 104 wall    1   1   

41.3333333333   47.8666666667   155.333333333   47.8666666667   wall    1   1   

41.3333333333   47.8666666667   41.3333333333   116 wall    1   1   

128 144.6   155.333333333   144.6   wall    1   1   

128 144.6   128 227.333333333   wall    1   1   

41.3333333333   144.6   128 144.6   wall    1   1   

155.333333333   47.8666666667   155.333333333   93  wall    1   1   

197.5   146.733333333   222 146.733333333   wall    1   1   

155.333333333   146.733333333   197.5   146.733333333   wall    1   1   

222 146.733333333   222 156.5   wall    1   1   

41.3333333333   116 41.3333333333   144.6   wall    1   1   

13.6666666667   116 41.3333333333   116 wall    1   1   

155.333333333   93  155.333333333   146.733333333   wall    1   1   

13.6666666667   144.6   41.3333333333   144.6   wall    1   1   

13.6666666667   116 13.6666666667   144.6   wall    1   1   

pytorch output: (in which only 11 walls are detected)

256     256
11
12.449275362318842      208.3768115942029       116.59958071278825      208.3768115942029       3       0
37.315602836879435      107.09478021978023      37.315602836879435      135.16484142914493      0       6
203.3   135.16484142914493      203.3   208.3768115942029       0       2
116.59958071278825      135.16484142914493      203.3   135.16484142914493      0       2
116.59958071278825      135.16484142914493      116.59958071278825      208.3768115942029       2       3
37.315602836879435      135.16484142914493      116.59958071278825      135.16484142914493      0       3
116.59958071278825      208.3768115942029       203.3   208.3768115942029       2       0
12.449275362318842      107.09478021978023      37.315602836879435      107.09478021978023      0       6
12.449275362318842      135.16484142914493      37.315602836879435      135.16484142914493      6       3
12.449275362318842      107.09478021978023      12.449275362318842      135.16484142914493      6       0
12.449275362318842      135.16484142914493      12.449275362318842      208.3768115942029       3       0
142     135.0   166     135.0   door    1       1
15      135.0   36      135.0   door    1       1
53      135.0   102     135.0   door    1       1
116.0   147     116.0   193     door    1       1
12.0    161     12.0    179     door    1       1
42      208.0   88      208.0   door    1       1
130     208.0   173     208.0   door    1       1
213     46      249     67      bathtub 1       1
186     159     201     207     cooking_counter 1       1
149     46      162     64      toilet  1       1
185     109     201     134     entrance        1       1
173     45      186     58      washing_basin   1       1
191     46      209     61      washing_basin   1       1

While the corners are all predicted correctly, the walls are predicted poorly. Since the performance of pytorch model is not checked, is it a bad model or something wrong with IP ? or should I train more epoch ?

[riti1302]

Hello @shuuchen Did you used the same code of pytorch as provided in this repository as it is or you modified it also? I am trying to run the pytorch code but I am having trouble in setting the path to the input files. Looking forward to your reply.

[shuuchen]

@riti1302 My version. https://github.com/shuuchen/FloorplanTransformation/tree/master/pytorch

[riti1302]

@shuuchen Thanks for sharing your version. I am getting the same error as I was getting in @art-programmer code. Do you know how to solve this? keyname=floorplan task=train started Traceback (most recent call last): File "train.py", line 175, in main(args) File "train.py", line 25, in main dataset = FloorplanDataset(options, split='train_1', random=True) File "/home/ritika/Documents/FloorplanTransformation/pytorch/datasets/floorplan_dataset.py", line 303, in init with open(self.dataFolder + split + '.txt') as f: IOError: [Errno 2] No such file or directory: '../data/train_1.txt'

[shuuchen]

@riti1302

IOError: [Errno 2] No such file or directory: '../data/train_1.txt'

It says you don' t have this file. Use '../data/train.txt' instead.

[riti1302]

@shuuchen There is no train.txt file in data directory. Have you used the same data directory? And also can you tell me where have you put your input data?

[shuuchen]

@riti1302 try again

[riti1302]

@shuuchen The same problem persists.

[shuuchen]

@riti1302 clone the project again. I have updated it

[riti1302]

@shuuchen Thanks. It solved my problem. Actually, I am new to pytorch and I am having trouble writing the code for prediction. It will help a lot if you send me your code for prediction.

[shuuchen]

@riti1302 what do you want to predict ?

[riti1302]

@shuuchen If I enter a input floorplan image then a txt file (similar as one used during training) should be given out as an output.

[shuuchen]

@riti1302 I recommend you can train the model for 100 epochs

[riti1302]

@shuuchen I did that. I want to know how you got the output (shown below) that you posted in this issue.

[shuuchen]

@riti1302 which file are you looking ? could you share the url ?

[sanwong15]

@shuuchen Hi, I also face issue training this network. I went into the Pytorch folder and follow the instruction (1: install requirment.txt, 2: python train.py -restore = 0) then it returned No such file or directory: '../data/train.txt' error. How can I solve this? Thanks

[shuuchen]

@sanwong15

you don't have the '../data/train.txt'. just download the file, put it in the right path, and try again.

[sanwong15]

Hi. Thanks for your reply. Where can I download it? How about the test.txt file? I am not sure if I have misread the documents. Thank you for helping me out with this

San

On Sat, Jun 22, 2019, 11:35 PM Shuchen Du notifications@github.com wrote:

@sanwong15 https://github.com/sanwong15

you don't have the '../data/train.txt'. just download the file, put it in the right path, and try again.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/art-programmer/FloorplanTransformation/issues/22?email_source=notifications&email_token=ABWPLCK4ROCSFP5OLRQVWG3P3ZBAFA5CNFSM4HBMG7F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYKLZLY#issuecomment-504675503, or mute the thread https://github.com/notifications/unsubscribe-auth/ABWPLCO6CHB72NSJWQ37UGTP3ZBAFANCNFSM4HBMG7FQ .

[riti1302]

@shuuchen How you test the output for a floorplan?

[shuuchen]

@riti1302 you said the walls are predicted poorly, so what model did you use ? I want details about that

[shuuchen]

@sanwong15 my version https://github.com/shuuchen/FloorplanTransformation/tree/master/pytorch

[riti1302]

@shuuchen As you said in this issue that you tested the image. Can you tell me what command you used to test the image?

[shuuchen]

@riti1302 Ok. That is and old issue. I have already fixed that. Just use the following command

python train.py --task=test

[riti1302]

@shuuchen
Okay. Thanks. And what is the use of function test_batch?

[shuuchen]

@riti1302 test_batch is for test, while test is for validation refer to ../data and you will find the files

[riti1302]

@shuuchen Where is the output text files are stored?

[shuuchen]

@riti1302 it is in test folder. I suggest you reading the source code as it is easy to understand. this discussion page is too long, I recommend open a new issue in my personal page https://github.com/shuuchen/FloorplanTransformation/tree/master/pytorch

[riti1302]

@shuuchen There is no option of creating a issue in your repository.

[shuuchen]

@riti1302 ok, all right

[riti1302]

@shuuchen I have tried training the model with 100 epochs, batch size=5 and number of images = 600. The performance of the model is very poor. Have you tried improving the accuracy? If yes, then can you tell me what changes should I make to improve the accuracy?

[shuuchen]

what train data did you use? you can use the data on my github, the result is good.

2019年6月25日(火) 14:26 Ritika Kumari notifications@github.com:

@shuuchen https://github.com/shuuchen I have tried training the model with 100 epochs, batch size=5 and number of images = 600. The performance of the model is very poor. Have you tried improving the accuracy? If yes, then can you tell me what changes should I make to improve the accuracy?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/art-programmer/FloorplanTransformation/issues/22?email_source=notifications&email_token=ADFCC2LWOS236GHPXNP6273P4GT7FA5CNFSM4HBMG7F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYPBMKY#issuecomment-505288235, or mute the thread https://github.com/notifications/unsubscribe-auth/ADFCC2OTAC6AMHBET7DZDI3P4GT7FANCNFSM4HBMG7FQ .

[riti1302]

@shuuchen I used your data. Can you give me some of your training details like number of epochs, learning rate and batch size?

[shuuchen]

@riti1302 just use python train.py --restore=0 did you modify the code ?

[riti1302]

@shuuchen Yes, a little. I am saving the checkpoint every 5 epochs. Nothing else.

[riti1302]

@shuuchen Also the the default epochs is 1000. Have you trained the model on 1000 epochs?

[shuuchen]

I think 500 epoch is enough. Have you trained that much? I updated my repository. Please reload that

Ritika Kumari notifications@github.com于2019年6月26日 周三下午3:41写道：

@shuuchen https://github.com/shuuchen Also the the default epochs is

1000. Have you trained the model on 1000 epochs?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/art-programmer/FloorplanTransformation/issues/22?email_source=notifications&email_token=ADFCC2PK6KVDGCZ3PPFBH5TP4MFQ5A5CNFSM4HBMG7F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYSQGIA#issuecomment-505742112, or mute the thread https://github.com/notifications/unsubscribe-auth/ADFCC2PA2XQ562TAJ75JTL3P4MFQ5ANCNFSM4HBMG7FQ .

[riti1302]

@shuuchen I trained it at 100 epochs with your dataset and it is showing infeasible.

[shuuchen]

@riti1302 train more please 500 epoch is good

[riti1302]

@shuuchen I figured out my mistake. I was saving the checkpoint after every 5 epochs and the base was set to 0. When I was executing the code for validation it was still set to 0. For validation I have to change the base to 95 so that it will load the latest checkpoint. Now it's giving good accuracy. Thanks a lot for your help.

[riti1302]

@shuuchen What is the meaning of last two columns of the text file? Also can you tell me how to detect different type of doors?

[shuuchen]

@riti1302 what do you mean by last two columns of the text file?

[riti1302]

The last two columns. For example: 1 1 after kitchen

[shuuchen]

@riti1302 Well, the last two columns are not used yet. If you dive into the code, you can realize that.

[riti1302]

@shuuchen Okay, i got it. I am trying to train the model on my dataset but it is giving error. I posted that error above.

[shuuchen]

@riti1302 it seems you use bool object as dataloader just check the variables according to the error messege

[atacand1920]

@shuuchen @riti1302 hello guys, can i use the task=test_batch or task=test without drn? i have a model_floorplan.t7 that i plan to use it instead of the drn dataset because i don't have an account(no access). If you can help me with that, it will be appreciated.

[liunan123456]

@riti1302 it seems you use bool object as dataloader just check the variables according to the error messege

@sanwong15 my version https://github.com/shuuchen/FloorplanTransformation/tree/master/pytorch

hello @shuuchen i have traind the model for 200 epochs,but the result of the predict is very poor,can you give me a trained model to have a test? thanks,591939876@qq.com

[1021928487]

when I run “python train.py --task=test”，output: It's not what you run it How can I solve this problem?@shuuchen

[joaocmd]

@shuuchen I figured out my mistake. I was saving the checkpoint after every 5 epochs and the base was set to 0. When I was executing the code for validation it was still set to 0. For validation I have to change the base to 95 so that it will load the latest checkpoint. Now it's giving good accuracy. Thanks a lot for your help.

Hello, could you explain what the change was? I do not understand what base you were referring to in your comment.

[lilu-REA]

how many GPU memory you need to train? it is huge on my side.

[BobbyZ04]

Heyyy! I wonder how did you output these combined output images?

I could only get separate output for door/line/icon/room but no combined output. and I checked IP.py and could not find anywhere they combine them together. Thank you!!