run so slow

[shangfenghuang]

Thank your work. When I run the work，the speed of running one epoch is very slow. it is about one hour. But I see in the log file that the running one epoch just 20 minutes. So I can't understand. Can you help me? this is the log file in one epoch 2020-12-03 20:29:16.650905 ---- batch: 050 ---- mean loss: 184.534502 accuracy: 0.414029 ---- batch: 100 ---- mean loss: 135.335496 accuracy: 0.530120 ---- batch: 150 ---- mean loss: 129.747210 accuracy: 0.535523 ---- batch: 200 ---- mean loss: 122.139866 accuracy: 0.558028 ---- batch: 250 ---- mean loss: 119.617929 accuracy: 0.560829 ---- batch: 300 ---- mean loss: 113.885079 accuracy: 0.584236 ---- batch: 350 ---- mean loss: 116.340719 accuracy: 0.569333 ---- batch: 400 ---- mean loss: 113.555371 accuracy: 0.580139 ---- batch: 450 ---- mean loss: 110.236181 accuracy: 0.584645 ---- batch: 500 ---- mean loss: 110.504616 accuracy: 0.594045 ---- batch: 550 ---- mean loss: 104.735284 accuracy: 0.608489 ---- batch: 600 ---- mean loss: 104.923058 accuracy: 0.601770 ---- batch: 650 ---- mean loss: 106.618274 accuracy: 0.598362 ---- batch: 700 ---- mean loss: 104.379048 accuracy: 0.612791 ---- batch: 750 ---- mean loss: 102.553243 accuracy: 0.618261 ---- batch: 800 ---- mean loss: 102.361754 accuracy: 0.611996 ---- batch: 850 ---- mean loss: 102.780255 accuracy: 0.615874 ---- batch: 900 ---- mean loss: 100.150999 accuracy: 0.626256 ---- batch: 950 ---- mean loss: 104.170692 accuracy: 0.608262 ---- batch: 1000 ---- mean loss: 102.701527 accuracy: 0.615849 ---- batch: 1050 ---- mean loss: 101.427789 accuracy: 0.613831 ---- batch: 1100 ---- mean loss: 104.226453 accuracy: 0.599298 ---- batch: 1150 ---- mean loss: 97.109982 accuracy: 0.631632 ---- batch: 1200 ---- mean loss: 99.082409 accuracy: 0.623211 ---- batch: 1250 ---- mean loss: 98.161291 accuracy: 0.618052 ---- batch: 1300 ---- mean loss: 93.044155 accuracy: 0.639523 ---- batch: 1350 ---- mean loss: 90.239651 accuracy: 0.652435 ---- batch: 1400 ---- mean loss: 90.905718 accuracy: 0.650580 ---- batch: 1450 ---- mean loss: 90.796373 accuracy: 0.650573 ---- batch: 1500 ---- mean loss: 88.180042 accuracy: 0.663508 ---- batch: 1550 ---- mean loss: 90.931050 accuracy: 0.647735 ---- batch: 1600 ---- mean loss: 90.572594 accuracy: 0.647632 ---- batch: 1650 ---- mean loss: 83.347111 accuracy: 0.676880 ---- batch: 1700 ---- mean loss: 88.313284 accuracy: 0.657798 ---- batch: 1750 ---- mean loss: 82.861588 accuracy: 0.681264 ---- batch: 1800 ---- mean loss: 89.783586 accuracy: 0.651877 ---- batch: 1850 ---- mean loss: 84.404577 accuracy: 0.673862 ---- batch: 1900 ---- mean loss: 87.348818 accuracy: 0.658631 ---- batch: 1950 ---- mean loss: 83.427303 accuracy: 0.670092 ---- batch: 2000 ---- mean loss: 88.491244 accuracy: 0.654787 ---- batch: 2050 ---- mean loss: 84.942625 accuracy: 0.661988 ---- batch: 2100 ---- mean loss: 84.637836 accuracy: 0.667242 ---- batch: 2150 ---- mean loss: 86.843850 accuracy: 0.660539 ---- batch: 2200 ---- mean loss: 85.992690 accuracy: 0.670484 ---- batch: 2250 ---- mean loss: 86.092916 accuracy: 0.659830 ---- batch: 2300 ---- mean loss: 82.865510 accuracy: 0.679619 ---- batch: 2350 ---- mean loss: 82.640754 accuracy: 0.674528 ---- batch: 2400 ---- mean loss: 81.347898 accuracy: 0.683257 ---- batch: 2450 ---- mean loss: 83.726160 accuracy: 0.670507 ---- batch: 2500 ---- mean loss: 82.711281 accuracy: 0.667460 ---- batch: 2550 ---- mean loss: 85.248889 accuracy: 0.664610 ---- batch: 2600 ---- mean loss: 79.271644 accuracy: 0.684864 ---- batch: 2650 ---- mean loss: 82.488315 accuracy: 0.672837 ---- batch: 2700 ---- mean loss: 81.616334 accuracy: 0.676569 ---- batch: 2750 ---- mean loss: 83.177334 accuracy: 0.668547 ---- batch: 2800 ---- mean loss: 81.139334 accuracy: 0.684465 ---- batch: 2850 ---- mean loss: 80.436449 accuracy: 0.679211 ---- batch: 2900 ---- mean loss: 80.295713 accuracy: 0.678259 ---- batch: 2950 ---- mean loss: 80.749244 accuracy: 0.671857 ---- batch: 3000 ---- mean loss: 80.518642 accuracy: 0.677207 ---- batch: 3050 ---- mean loss: 77.829687 accuracy: 0.685728 ---- batch: 3100 ---- mean loss: 81.392671 accuracy: 0.671245 ---- batch: 3150 ---- mean loss: 76.950525 accuracy: 0.691033 ---- batch: 3200 ---- mean loss: 79.833296 accuracy: 0.682424 ---- batch: 3250 ---- mean loss: 81.639724 accuracy: 0.670625 ---- batch: 3300 ---- mean loss: 77.314783 accuracy: 0.688428 ---- batch: 3350 ---- mean loss: 76.034729 accuracy: 0.694535 ---- batch: 3400 ---- mean loss: 78.178265 accuracy: 0.684664 ---- batch: 3450 ---- mean loss: 75.660341 accuracy: 0.692333 ---- batch: 3500 ---- mean loss: 74.944008 accuracy: 0.687972 ---- batch: 3550 ---- mean loss: 77.615459 accuracy: 0.687553 ---- batch: 3600 ---- mean loss: 77.393342 accuracy: 0.685459 ---- batch: 3650 ---- mean loss: 80.323210 accuracy: 0.676606 ---- batch: 3700 ---- mean loss: 77.831140 accuracy: 0.678844 ---- batch: 3750 ---- mean loss: 73.645795 accuracy: 0.701312 ---- batch: 3800 ---- mean loss: 73.109120 accuracy: 0.698930 ---- batch: 3850 ---- mean loss: 72.719140 accuracy: 0.707183 ---- batch: 3900 ---- mean loss: 76.973215 accuracy: 0.686412 ---- batch: 3950 ---- mean loss: 72.995662 accuracy: 0.698651 ---- batch: 4000 ---- mean loss: 74.334438 accuracy: 0.692604 ---- batch: 4050 ---- mean loss: 71.758526 accuracy: 0.710445 ---- batch: 4100 ---- mean loss: 73.972742 accuracy: 0.695303 ---- batch: 4150 ---- mean loss: 70.600237 accuracy: 0.705352 ---- batch: 4200 ---- mean loss: 71.107945 accuracy: 0.703613 training one batch require 791.24 milliseconds 2020-12-03 21:49:02.473758 ---- EPOCH 000 EVALUATION ---- eval mean loss: 12.938971 eval overall accuracy: 0.732570 eval avg class acc: 0.566511 eval mIoU of other20: 0.432905 eval mIoU of wall: 0.602921 eval mIoU of floor: 0.916897 eval mIoU of cabinet: 0.323481 eval mIoU of bed: 0.547132 eval mIoU of chair: 0.734238 eval mIoU of sofa: 0.625162 eval mIoU of table: 0.548581 eval mIoU of door: 0.265242 eval mIoU of window: 0.233336 eval mIoU of bookshelf: 0.480115 eval mIoU of picture: 0.001725 eval mIoU of counter: 0.326607 eval mIoU of desk: 0.312228 eval mIoU of curtain: 0.325191 eval mIoU of refridgerator: 0.171487 eval mIoU of shower curtain: 0.188239 eval mIoU of toilet: 0.371836 eval mIoU of sink: 0.325391 eval mIoU of bathtub: 0.414665 eval mIoU of otherfurniture: 0.162511 eval mIoU of all classes: 0.395709 testing one batch require 334.10 milliseconds Model saved in file: /home/disk1/hsf/SPH3D-GCN/log_scannet/model.ckpt-0

[shangfenghuang]

my GPU just is used about 4000 memory.

[EnyaHermite]

You can use less buffer_size in the input_fn. Shuffling tfrecord dataset with 10000 buffer_size setting indeed burdens the CPU memory. Reduce it to 1000 should also work well.