Train problem

[Sero8139]

Hi, @MichalBusta

I try to train the model use my data. and I output the log , find out that gt_labels is a float type .

I1120 05:50:09.142750 3234 solver.cpp:397] Test net output #7500: gt_labels = 50.2 I1120 05:50:09.142755 3234 solver.cpp:397] Test net output #7501: gt_labels = 52.3 I1120 05:50:09.142760 3234 solver.cpp:397] Test net output #7502: gt_labels = 58.1 I1120 05:50:09.142769 3234 solver.cpp:397] Test net output #7503: loss_RegionLayer_0_split_0 = 0 ( 1 = 0 loss) I1120 05:50:09.142776 3234 solver.cpp:397] Test net output #7504: out = 0 ( 1 = 0 loss)

and it always occur at first test sample.

Could you give me some tips ? I don't know what mistake I do.

thx

[MichalBusta]

Hi, no mistake, just ugly behavior, see: https://github.com/MichalBusta/DeepTextSpotter/issues/67#issuecomment-430993331

[Sero8139]

@MichalBusta , thanks for your reply . I have seen the #67 , and I have tried to change the loglevel. but I still have some problem .

the following is some information.

I train the model use train.py command.

python train.py -data_dir / -train_list data/train.txt -valid_list data/test.txt -batch_size 1

and listfile of my data like below /opt/DeepTextSpotter/data/test/_i_0_j_0_00401.jpg /opt/DeepTextSpotter/data/test/_i_0_j_0_00403.jpg /opt/DeepTextSpotter/data/test/_i_0_j_0_00407.jpg /opt/DeepTextSpotter/data/test/_i_0_j_0_00409.jpg /opt/DeepTextSpotter/data/test/_i_0_j_0_00411.jpg /opt/DeepTextSpotter/data/test/_i_0_j_0_00413.jpg

and annotation file of my data is like below cls_id , cx, cy, width, height, angle -1 0.17445838815789472 0.08311944901315789 0.0685370767088 0.0485656452315 0.0379568530092 30A I have seen #53 , and use the method to convert my data.

and I always encountered libgomp: Out of memory allocating 328111969088 bytes

and I observe the phenomenon which is always occur when the recognition phase start to run.

and I struggle with the problem for long time.