search

yuehuang2023 / cryoNeFEN

GNU General Public License v3.0
22 stars 0 forks source link

Unsatisfactory result #2

Open emberslee opened 4 months ago

emberslee commented 4 months ago

I was using cryoNeFEN to do reconstruction on experimental particles. The output logs is

2024-07-25 17:47:15 Use cuda True 2024-07-25 17:47:15 train.py /data/cryosparc/demo/CS-cryosparc-tutorial/J15/J15_006_particles.cs --mask /data/cryosparc/demo/CS-cryosparc-tutorial/J15/J15_006_volume_mask_refine.mrc --lazy --outdir demo/sp1 --split 1 --batch-size 1 2024-07-25 17:47:15 Loading dataset from /data/cryosparc/demo/CS-cryosparc-tutorial/J15/J15_006_particles.cs 2024-07-25 17:47:15 Split dataset randomly for GSFSC 2024-07-25 17:47:16 Loaded 6034 440x440 images 2024-07-25 17:47:16.075097 Image size (pix) : 440 2024-07-25 17:47:16.075156 A/pix : 0.6575000286102295 2024-07-25 17:47:16.075174 DefocusU (A) : 12478.5703125 2024-07-25 17:47:16.075192 DefocusV (A) : 12365.890625 2024-07-25 17:47:16.075205 Dfang (deg) : 268.94610595703125 2024-07-25 17:47:16.075216 voltage (kV) : 300.0 2024-07-25 17:47:16.075227 cs (mm) : 2.700000047683716 2024-07-25 17:47:16.075239 w : 0.10000000149011612 2024-07-25 17:47:16.075250 Phase shift (deg) : 0.0 2024-07-25 17:47:21.547067 Extracting rotations from alignments3D/pose 2024-07-25 17:47:21.567661 Transposing rotation matrix 2024-07-25 17:47:21.578344 Extracting translations from alignments3D/shift 2024-07-25 17:47:22 Training with C1 symmetry 2024-07-25 17:47:26 PositionalDecoder( (decoder): MLP( (main): Sequential( (0): MyLinear(in_features=192, out_features=256, bias=True) (1): ReLU(inplace=True) (2): MyLinear(in_features=256, out_features=256, bias=True) (3): ReLU(inplace=True) (4): MyLinear(in_features=256, out_features=256, bias=True) (5): ReLU(inplace=True) (6): MyLinear(in_features=256, out_features=1, bias=True) ) ) ) 2024-07-25 17:47:26 181249 parameters in model 2024-07-25 17:47:55.591014 # [Train Epoch: 1/20] [100/6034 images] loss=0.627213 2024-07-25 17:48:19.169251 # [Train Epoch: 1/20] [200/6034 images] loss=0.641797 2024-07-25 17:48:42.798538 # [Train Epoch: 1/20] [300/6034 images] loss=0.650648 2024-07-25 17:49:06.497974 # [Train Epoch: 1/20] [400/6034 images] loss=0.641573 2024-07-25 17:49:30.237806 # [Train Epoch: 1/20] [500/6034 images] loss=0.629527 2024-07-25 17:49:53.927724 # [Train Epoch: 1/20] [600/6034 images] loss=0.634932 2024-07-25 17:50:17.609029 # [Train Epoch: 1/20] [700/6034 images] loss=0.623167 2024-07-25 17:50:41.292956 # [Train Epoch: 1/20] [800/6034 images] loss=0.634316 2024-07-25 17:51:05.011809 # [Train Epoch: 1/20] [900/6034 images] loss=0.620026 2024-07-25 17:51:28.741274 # [Train Epoch: 1/20] [1000/6034 images] loss=0.615340 2024-07-25 17:51:52.451034 # [Train Epoch: 1/20] [1100/6034 images] loss=0.633602 2024-07-25 17:52:16.198808 # [Train Epoch: 1/20] [1200/6034 images] loss=0.604127 2024-07-25 17:52:39.892095 # [Train Epoch: 1/20] [1300/6034 images] loss=0.616086 2024-07-25 17:53:03.576296 # [Train Epoch: 1/20] [1400/6034 images] loss=0.617445 2024-07-25 17:53:27.261979 # [Train Epoch: 1/20] [1500/6034 images] loss=0.616798 2024-07-25 17:53:50.968840 # [Train Epoch: 1/20] [1600/6034 images] loss=0.616812 2024-07-25 17:54:14.668438 # [Train Epoch: 1/20] [1700/6034 images] loss=0.648810 2024-07-25 17:54:38.358986 # [Train Epoch: 1/20] [1800/6034 images] loss=0.653584 2024-07-25 17:55:02.005502 # [Train Epoch: 1/20] [1900/6034 images] loss=0.620596 2024-07-25 17:55:25.667524 # [Train Epoch: 1/20] [2000/6034 images] loss=0.641084 2024-07-25 17:55:49.342192 # [Train Epoch: 1/20] [2100/6034 images] loss=0.627600 2024-07-25 17:56:13.039045 # [Train Epoch: 1/20] [2200/6034 images] loss=0.637172 2024-07-25 17:56:36.743038 # [Train Epoch: 1/20] [2300/6034 images] loss=0.627206 2024-07-25 17:57:00.459332 # [Train Epoch: 1/20] [2400/6034 images] loss=0.645853 2024-07-25 17:57:24.188580 # [Train Epoch: 1/20] [2500/6034 images] loss=0.619509 2024-07-25 17:57:47.875377 # [Train Epoch: 1/20] [2600/6034 images] loss=0.624987 2024-07-25 17:58:11.552231 # [Train Epoch: 1/20] [2700/6034 images] loss=0.617783 2024-07-25 17:58:35.224357 # [Train Epoch: 1/20] [2800/6034 images] loss=0.645013 2024-07-25 17:58:58.900323 # [Train Epoch: 1/20] [2900/6034 images] loss=0.624054 2024-07-25 17:59:22.561648 # [Train Epoch: 1/20] [3000/6034 images] loss=0.624242 2024-07-25 17:59:46.217728 # [Train Epoch: 1/20] [3100/6034 images] loss=0.627480 2024-07-25 18:00:09.881972 # [Train Epoch: 1/20] [3200/6034 images] loss=0.624482 2024-07-25 18:00:33.544642 # [Train Epoch: 1/20] [3300/6034 images] loss=0.622299 2024-07-25 18:00:57.233591 # [Train Epoch: 1/20] [3400/6034 images] loss=0.625317 2024-07-25 18:01:20.903191 # [Train Epoch: 1/20] [3500/6034 images] loss=0.666523 2024-07-25 18:01:44.590440 # [Train Epoch: 1/20] [3600/6034 images] loss=0.628195 2024-07-25 18:02:08.250259 # [Train Epoch: 1/20] [3700/6034 images] loss=0.615803 2024-07-25 18:02:31.956224 # [Train Epoch: 1/20] [3800/6034 images] loss=0.612390 2024-07-25 18:02:55.589841 # [Train Epoch: 1/20] [3900/6034 images] loss=0.604301 2024-07-25 18:03:19.261250 # [Train Epoch: 1/20] [4000/6034 images] loss=0.620211 2024-07-25 18:03:42.915172 # [Train Epoch: 1/20] [4100/6034 images] loss=0.632283 2024-07-25 18:04:06.588694 # [Train Epoch: 1/20] [4200/6034 images] loss=0.633816 2024-07-25 18:04:30.283447 # [Train Epoch: 1/20] [4300/6034 images] loss=0.616493 2024-07-25 18:04:53.923550 # [Train Epoch: 1/20] [4400/6034 images] loss=0.622755 2024-07-25 18:05:17.526363 # [Train Epoch: 1/20] [4500/6034 images] loss=0.631106 2024-07-25 18:05:41.165608 # [Train Epoch: 1/20] [4600/6034 images] loss=0.628087 2024-07-25 18:06:04.809578 # [Train Epoch: 1/20] [4700/6034 images] loss=0.611841 2024-07-25 18:06:28.464131 # [Train Epoch: 1/20] [4800/6034 images] loss=0.639557 2024-07-25 18:06:52.101400 # [Train Epoch: 1/20] [4900/6034 images] loss=0.614648 2024-07-25 18:07:15.765804 # [Train Epoch: 1/20] [5000/6034 images] loss=0.617827 2024-07-25 18:07:39.407157 # [Train Epoch: 1/20] [5100/6034 images] loss=0.650619 2024-07-25 18:08:03.020079 # [Train Epoch: 1/20] [5200/6034 images] loss=0.650210 2024-07-25 18:08:26.671058 # [Train Epoch: 1/20] [5300/6034 images] loss=0.611166 2024-07-25 18:08:50.372102 # [Train Epoch: 1/20] [5400/6034 images] loss=0.647055 2024-07-25 18:09:13.997244 # [Train Epoch: 1/20] [5500/6034 images] loss=0.639192 2024-07-25 18:09:37.609761 # [Train Epoch: 1/20] [5600/6034 images] loss=0.624940 2024-07-25 18:10:01.277943 # [Train Epoch: 1/20] [5700/6034 images] loss=0.627872 2024-07-25 18:10:24.915843 # [Train Epoch: 1/20] [5800/6034 images] loss=0.648153 2024-07-25 18:10:48.598096 # [Train Epoch: 1/20] [5900/6034 images] loss=0.664348 2024-07-25 18:11:12.267249 # [Train Epoch: 1/20] [6000/6034 images] loss=0.653563 2024-07-25 18:11:20 # =====> Epoch: 1 Average loss = 0.629609; Finished in 0:23:54.122198 2024-07-25 18:11:49.394473 # [Train Epoch: 2/20] [100/6034 images] loss=0.598672 2024-07-25 18:12:13.054234 # [Train Epoch: 2/20] [200/6034 images] loss=0.639132 2024-07-25 18:12:36.747917 # [Train Epoch: 2/20] [300/6034 images] loss=0.631875 2024-07-25 18:13:00.444016 # [Train Epoch: 2/20] [400/6034 images] loss=0.617447 2024-07-25 18:13:24.130461 # [Train Epoch: 2/20] [500/6034 images] loss=0.613297 2024-07-25 18:13:47.816655 # [Train Epoch: 2/20] [600/6034 images] loss=0.627942 2024-07-25 18:14:11.495528 # [Train Epoch: 2/20] [700/6034 images] loss=0.631822 2024-07-25 18:14:35.200845 # [Train Epoch: 2/20] [800/6034 images] loss=0.637396 2024-07-25 18:14:58.869296 # [Train Epoch: 2/20] [900/6034 images] loss=0.609530 2024-07-25 18:15:22.546383 # [Train Epoch: 2/20] [1000/6034 images] loss=0.624590 2024-07-25 18:15:46.279113 # [Train Epoch: 2/20] [1100/6034 images] loss=0.636707 2024-07-25 18:16:09.966267 # [Train Epoch: 2/20] [1200/6034 images] loss=0.630167 2024-07-25 18:16:33.635864 # [Train Epoch: 2/20] [1300/6034 images] loss=0.635837 2024-07-25 18:16:57.334801 # [Train Epoch: 2/20] [1400/6034 images] loss=0.646231 2024-07-25 18:17:20.995959 # [Train Epoch: 2/20] [1500/6034 images] loss=0.625542 2024-07-25 18:17:44.663095 # [Train Epoch: 2/20] [1600/6034 images] loss=0.651992 2024-07-25 18:18:08.335106 # [Train Epoch: 2/20] [1700/6034 images] loss=0.629318 2024-07-25 18:18:32.013620 # [Train Epoch: 2/20] [1800/6034 images] loss=0.617220 2024-07-25 18:18:55.689990 # [Train Epoch: 2/20] [1900/6034 images] loss=0.637089 2024-07-25 18:19:19.339278 # [Train Epoch: 2/20] [2000/6034 images] loss=0.617712 2024-07-25 18:19:43.054029 # [Train Epoch: 2/20] [2100/6034 images] loss=0.640502 2024-07-25 18:20:06.764591 # [Train Epoch: 2/20] [2200/6034 images] loss=0.637101 2024-07-25 18:20:30.458574 # [Train Epoch: 2/20] [2300/6034 images] loss=0.646547 2024-07-25 18:20:54.199797 # [Train Epoch: 2/20] [2400/6034 images] loss=0.633386 2024-07-25 18:21:17.876349 # [Train Epoch: 2/20] [2500/6034 images] loss=0.641081 2024-07-25 18:21:41.604636 # [Train Epoch: 2/20] [2600/6034 images] loss=0.627648 2024-07-25 18:22:05.317286 # [Train Epoch: 2/20] [2700/6034 images] loss=0.607032 2024-07-25 18:22:29.064555 # [Train Epoch: 2/20] [2800/6034 images] loss=0.630947 2024-07-25 18:22:52.757243 # [Train Epoch: 2/20] [2900/6034 images] loss=0.647319 2024-07-25 18:23:16.467392 # [Train Epoch: 2/20] [3000/6034 images] loss=0.645594 2024-07-25 18:23:40.173148 # [Train Epoch: 2/20] [3100/6034 images] loss=0.622823 2024-07-25 18:24:03.909861 # [Train Epoch: 2/20] [3200/6034 images] loss=0.605803 2024-07-25 18:24:27.560150 # [Train Epoch: 2/20] [3300/6034 images] loss=0.586648 2024-07-25 18:24:51.232344 # [Train Epoch: 2/20] [3400/6034 images] loss=0.665559 2024-07-25 18:25:14.869906 # [Train Epoch: 2/20] [3500/6034 images] loss=0.611366 2024-07-25 18:25:38.543976 # [Train Epoch: 2/20] [3600/6034 images] loss=0.645908 2024-07-25 18:26:02.229942 # [Train Epoch: 2/20] [3700/6034 images] loss=0.621035 2024-07-25 18:26:25.900423 # [Train Epoch: 2/20] [3800/6034 images] loss=0.633054 2024-07-25 18:26:49.612985 # [Train Epoch: 2/20] [3900/6034 images] loss=0.647956 2024-07-25 18:27:13.286458 # [Train Epoch: 2/20] [4000/6034 images] loss=0.621115 2024-07-25 18:27:36.954691 # [Train Epoch: 2/20] [4100/6034 images] loss=0.629311 2024-07-25 18:28:00.610413 # [Train Epoch: 2/20] [4200/6034 images] loss=0.653298 2024-07-25 18:28:24.336579 # [Train Epoch: 2/20] [4300/6034 images] loss=0.638991 2024-07-25 18:28:47.994084 # [Train Epoch: 2/20] [4400/6034 images] loss=0.625249 2024-07-25 18:29:11.755912 # [Train Epoch: 2/20] [4500/6034 images] loss=0.618482 2024-07-25 18:29:35.467544 # [Train Epoch: 2/20] [4600/6034 images] loss=0.643689 2024-07-25 18:29:59.101120 # [Train Epoch: 2/20] [4700/6034 images] loss=0.625866 2024-07-25 18:30:22.780699 # [Train Epoch: 2/20] [4800/6034 images] loss=0.632836 2024-07-25 18:30:46.459171 # [Train Epoch: 2/20] [4900/6034 images] loss=0.619271 2024-07-25 18:31:10.187477 # [Train Epoch: 2/20] [5000/6034 images] loss=0.647901 2024-07-25 18:31:33.875075 # [Train Epoch: 2/20] [5100/6034 images] loss=0.632365 2024-07-25 18:31:57.551622 # [Train Epoch: 2/20] [5200/6034 images] loss=0.613220 2024-07-25 18:32:21.240382 # [Train Epoch: 2/20] [5300/6034 images] loss=0.633799 2024-07-25 18:32:44.904743 # [Train Epoch: 2/20] [5400/6034 images] loss=0.635370 2024-07-25 18:33:08.585999 # [Train Epoch: 2/20] [5500/6034 images] loss=0.647640 2024-07-25 18:33:32.266068 # [Train Epoch: 2/20] [5600/6034 images] loss=0.618259 2024-07-25 18:33:55.964274 # [Train Epoch: 2/20] [5700/6034 images] loss=0.637486 2024-07-25 18:34:19.658368 # [Train Epoch: 2/20] [5800/6034 images] loss=0.659924 2024-07-25 18:34:43.333461 # [Train Epoch: 2/20] [5900/6034 images] loss=0.618750 2024-07-25 18:35:07.037457 # [Train Epoch: 2/20] [6000/6034 images] loss=0.638715 2024-07-25 18:35:15 # =====> Epoch: 2 Average loss = 0.629586; Finished in 0:23:51.187412 2024-07-25 18:35:43.177155 # [Train Epoch: 3/20] [100/6034 images] loss=0.620500 2024-07-25 18:36:06.820825 # [Train Epoch: 3/20] [200/6034 images] loss=0.624781 2024-07-25 18:36:30.496724 # [Train Epoch: 3/20] [300/6034 images] loss=0.619964 2024-07-25 18:36:54.155854 # [Train Epoch: 3/20] [400/6034 images] loss=0.642200 2024-07-25 18:37:17.874300 # [Train Epoch: 3/20] [500/6034 images] loss=0.625857 2024-07-25 18:37:41.571346 # [Train Epoch: 3/20] [600/6034 images] loss=0.645447 2024-07-25 18:38:05.259206 # [Train Epoch: 3/20] [700/6034 images] loss=0.636654 2024-07-25 18:38:28.941949 # [Train Epoch: 3/20] [800/6034 images] loss=0.622363 2024-07-25 18:38:52.668436 # [Train Epoch: 3/20] [900/6034 images] loss=0.629939 2024-07-25 18:39:16.335130 # [Train Epoch: 3/20] [1000/6034 images] loss=0.622533 2024-07-25 18:39:40.011661 # [Train Epoch: 3/20] [1100/6034 images] loss=0.642721 2024-07-25 18:40:03.724665 # [Train Epoch: 3/20] [1200/6034 images] loss=0.682286 2024-07-25 18:40:27.457923 # [Train Epoch: 3/20] [1300/6034 images] loss=0.631012 2024-07-25 18:40:51.172376 # [Train Epoch: 3/20] [1400/6034 images] loss=0.613824 2024-07-25 18:41:14.853944 # [Train Epoch: 3/20] [1500/6034 images] loss=0.617932 2024-07-25 18:41:38.532768 # [Train Epoch: 3/20] [1600/6034 images] loss=0.607033 2024-07-25 18:42:02.211782 # [Train Epoch: 3/20] [1700/6034 images] loss=0.619915 2024-07-25 18:42:25.883500 # [Train Epoch: 3/20] [1800/6034 images] loss=0.614146 2024-07-25 18:42:49.546079 # [Train Epoch: 3/20] [1900/6034 images] loss=0.642173 2024-07-25 18:43:13.202760 # [Train Epoch: 3/20] [2000/6034 images] loss=0.624101 2024-07-25 18:43:36.863999 # [Train Epoch: 3/20] [2100/6034 images] loss=0.675971 2024-07-25 18:44:00.517821 # [Train Epoch: 3/20] [2200/6034 images] loss=0.630669 2024-07-25 18:44:24.186898 # [Train Epoch: 3/20] [2300/6034 images] loss=0.631562 2024-07-25 18:44:47.808417 # [Train Epoch: 3/20] [2400/6034 images] loss=0.634404 2024-07-25 18:45:11.471386 # [Train Epoch: 3/20] [2500/6034 images] loss=0.619937 2024-07-25 18:45:35.126230 # [Train Epoch: 3/20] [2600/6034 images] loss=0.623403 2024-07-25 18:45:58.779552 # [Train Epoch: 3/20] [2700/6034 images] loss=0.614716 2024-07-25 18:46:22.402342 # [Train Epoch: 3/20] [2800/6034 images] loss=0.622856 2024-07-25 18:46:46.079516 # [Train Epoch: 3/20] [2900/6034 images] loss=0.632229 2024-07-25 18:47:09.749736 # [Train Epoch: 3/20] [3000/6034 images] loss=0.627359 2024-07-25 18:47:33.427444 # [Train Epoch: 3/20] [3100/6034 images] loss=0.651026 2024-07-25 18:47:57.075739 # [Train Epoch: 3/20] [3200/6034 images] loss=0.651461 2024-07-25 18:48:20.761287 # [Train Epoch: 3/20] [3300/6034 images] loss=0.608389 2024-07-25 18:48:44.424021 # [Train Epoch: 3/20] [3400/6034 images] loss=0.627393 2024-07-25 18:49:08.091354 # [Train Epoch: 3/20] [3500/6034 images] loss=0.634575 2024-07-25 18:49:31.771808 # [Train Epoch: 3/20] [3600/6034 images] loss=0.643496 2024-07-25 18:49:55.482504 # [Train Epoch: 3/20] [3700/6034 images] loss=0.588214 2024-07-25 18:50:19.114201 # [Train Epoch: 3/20] [3800/6034 images] loss=0.621381 2024-07-25 18:50:42.764886 # [Train Epoch: 3/20] [3900/6034 images] loss=0.619028 2024-07-25 18:51:06.439347 # [Train Epoch: 3/20] [4000/6034 images] loss=0.651700 2024-07-25 18:51:30.108461 # [Train Epoch: 3/20] [4100/6034 images] loss=0.614737 2024-07-25 18:51:53.739204 # [Train Epoch: 3/20] [4200/6034 images] loss=0.649796 2024-07-25 18:52:17.295399 # [Train Epoch: 3/20] [4300/6034 images] loss=0.635947 2024-07-25 18:52:40.960896 # [Train Epoch: 3/20] [4400/6034 images] loss=0.654194 2024-07-25 18:53:04.609979 # [Train Epoch: 3/20] [4500/6034 images] loss=0.639273 2024-07-25 18:53:28.269746 # [Train Epoch: 3/20] [4600/6034 images] loss=0.654999 2024-07-25 18:53:51.884493 # [Train Epoch: 3/20] [4700/6034 images] loss=0.655280 2024-07-25 18:54:15.526173 # [Train Epoch: 3/20] [4800/6034 images] loss=0.641862 2024-07-25 18:54:39.159012 # [Train Epoch: 3/20] [4900/6034 images] loss=0.614687 2024-07-25 18:55:02.781659 # [Train Epoch: 3/20] [5000/6034 images] loss=0.633429 2024-07-25 18:55:26.411523 # [Train Epoch: 3/20] [5100/6034 images] loss=0.648705 2024-07-25 18:55:50.055020 # [Train Epoch: 3/20] [5200/6034 images] loss=0.636508 2024-07-25 18:56:13.683950 # [Train Epoch: 3/20] [5300/6034 images] loss=0.647893 2024-07-25 18:56:37.320095 # [Train Epoch: 3/20] [5400/6034 images] loss=0.619667 2024-07-25 18:57:00.977351 # [Train Epoch: 3/20] [5500/6034 images] loss=0.675648 2024-07-25 18:57:24.637773 # [Train Epoch: 3/20] [5600/6034 images] loss=0.634531 2024-07-25 18:57:48.305456 # [Train Epoch: 3/20] [5700/6034 images] loss=0.642847 2024-07-25 18:58:11.964970 # [Train Epoch: 3/20] [5800/6034 images] loss=0.639108 2024-07-25 18:58:35.621150 # [Train Epoch: 3/20] [5900/6034 images] loss=0.626094 2024-07-25 18:58:59.271147 # [Train Epoch: 3/20] [6000/6034 images] loss=0.641058 2024-07-25 18:59:07 # =====> Epoch: 3 Average loss = 0.629574; Finished in 0:23:49.473001 ........ cropped ...... 2024-07-26 00:09:45.933129 # [Train Epoch: 17/20] [100/6034 images] loss=0.649460 2024-07-26 00:10:09.486752 # [Train Epoch: 17/20] [200/6034 images] loss=0.651335 2024-07-26 00:10:33.072006 # [Train Epoch: 17/20] [300/6034 images] loss=0.619919 2024-07-26 00:10:56.704462 # [Train Epoch: 17/20] [400/6034 images] loss=0.626946 2024-07-26 00:11:20.334247 # [Train Epoch: 17/20] [500/6034 images] loss=0.634391 2024-07-26 00:11:43.946871 # [Train Epoch: 17/20] [600/6034 images] loss=0.651951 2024-07-26 00:12:07.531427 # [Train Epoch: 17/20] [700/6034 images] loss=0.618359 2024-07-26 00:12:31.123325 # [Train Epoch: 17/20] [800/6034 images] loss=0.629494 2024-07-26 00:12:54.745543 # [Train Epoch: 17/20] [900/6034 images] loss=0.628982 2024-07-26 00:13:18.370906 # [Train Epoch: 17/20] [1000/6034 images] loss=0.616766 2024-07-26 00:13:42.008894 # [Train Epoch: 17/20] [1100/6034 images] loss=0.606552 2024-07-26 00:14:05.670470 # [Train Epoch: 17/20] [1200/6034 images] loss=0.654685 2024-07-26 00:14:29.320057 # [Train Epoch: 17/20] [1300/6034 images] loss=0.622554 2024-07-26 00:14:52.900597 # [Train Epoch: 17/20] [1400/6034 images] loss=0.616841 2024-07-26 00:15:16.528619 # [Train Epoch: 17/20] [1500/6034 images] loss=0.649561 2024-07-26 00:15:40.154109 # [Train Epoch: 17/20] [1600/6034 images] loss=0.638512 2024-07-26 00:16:03.798537 # [Train Epoch: 17/20] [1700/6034 images] loss=0.602204 2024-07-26 00:16:27.402983 # [Train Epoch: 17/20] [1800/6034 images] loss=0.630032 2024-07-26 00:16:51.018643 # [Train Epoch: 17/20] [1900/6034 images] loss=0.635756 2024-07-26 00:17:14.653993 # [Train Epoch: 17/20] [2000/6034 images] loss=0.617845 2024-07-26 00:17:38.275822 # [Train Epoch: 17/20] [2100/6034 images] loss=0.627701 2024-07-26 00:18:01.891403 # [Train Epoch: 17/20] [2200/6034 images] loss=0.643037 2024-07-26 00:18:25.495065 # [Train Epoch: 17/20] [2300/6034 images] loss=0.613045 2024-07-26 00:18:49.092944 # [Train Epoch: 17/20] [2400/6034 images] loss=0.626189 2024-07-26 00:19:12.699963 # [Train Epoch: 17/20] [2500/6034 images] loss=0.608703 2024-07-26 00:19:36.319057 # [Train Epoch: 17/20] [2600/6034 images] loss=0.616476 2024-07-26 00:19:59.930769 # [Train Epoch: 17/20] [2700/6034 images] loss=0.629137 2024-07-26 00:20:23.554368 # [Train Epoch: 17/20] [2800/6034 images] loss=0.627101 2024-07-26 00:20:47.145897 # [Train Epoch: 17/20] [2900/6034 images] loss=0.619877 2024-07-26 00:21:10.767867 # [Train Epoch: 17/20] [3000/6034 images] loss=0.652526 2024-07-26 00:21:34.430811 # [Train Epoch: 17/20] [3100/6034 images] loss=0.629003 2024-07-26 00:21:58.090013 # [Train Epoch: 17/20] [3200/6034 images] loss=0.648064 2024-07-26 00:22:21.715827 # [Train Epoch: 17/20] [3300/6034 images] loss=0.622036 2024-07-26 00:22:45.356224 # [Train Epoch: 17/20] [3400/6034 images] loss=0.616467 2024-07-26 00:23:08.959534 # [Train Epoch: 17/20] [3500/6034 images] loss=0.621483 2024-07-26 00:23:32.600060 # [Train Epoch: 17/20] [3600/6034 images] loss=0.620327 2024-07-26 00:23:56.189131 # [Train Epoch: 17/20] [3700/6034 images] loss=0.664479 2024-07-26 00:24:19.852704 # [Train Epoch: 17/20] [3800/6034 images] loss=0.631689 2024-07-26 00:24:43.462617 # [Train Epoch: 17/20] [3900/6034 images] loss=0.626994 2024-07-26 00:25:07.026596 # [Train Epoch: 17/20] [4000/6034 images] loss=0.658661 2024-07-26 00:25:30.667724 # [Train Epoch: 17/20] [4100/6034 images] loss=0.648747 2024-07-26 00:25:54.304973 # [Train Epoch: 17/20] [4200/6034 images] loss=0.652464 2024-07-26 00:26:17.936094 # [Train Epoch: 17/20] [4300/6034 images] loss=0.614258 2024-07-26 00:26:41.614994 # [Train Epoch: 17/20] [4400/6034 images] loss=0.613460 2024-07-26 00:27:05.249536 # [Train Epoch: 17/20] [4500/6034 images] loss=0.614606 2024-07-26 00:27:28.906419 # [Train Epoch: 17/20] [4600/6034 images] loss=0.618334 2024-07-26 00:27:52.529242 # [Train Epoch: 17/20] [4700/6034 images] loss=0.628526 2024-07-26 00:28:16.153451 # [Train Epoch: 17/20] [4800/6034 images] loss=0.632095 2024-07-26 00:28:39.819720 # [Train Epoch: 17/20] [4900/6034 images] loss=0.653661 2024-07-26 00:29:03.450328 # [Train Epoch: 17/20] [5000/6034 images] loss=0.623318 2024-07-26 00:29:27.057009 # [Train Epoch: 17/20] [5100/6034 images] loss=0.603281 2024-07-26 00:29:50.695353 # [Train Epoch: 17/20] [5200/6034 images] loss=0.615789 2024-07-26 00:30:14.336584 # [Train Epoch: 17/20] [5300/6034 images] loss=0.624600 2024-07-26 00:30:37.998227 # [Train Epoch: 17/20] [5400/6034 images] loss=0.616637 2024-07-26 00:31:01.654474 # [Train Epoch: 17/20] [5500/6034 images] loss=0.623651 2024-07-26 00:31:25.256624 # [Train Epoch: 17/20] [5600/6034 images] loss=0.627580 2024-07-26 00:31:48.914071 # [Train Epoch: 17/20] [5700/6034 images] loss=0.620687 2024-07-26 00:32:12.543853 # [Train Epoch: 17/20] [5800/6034 images] loss=0.628819 2024-07-26 00:32:36.172217 # [Train Epoch: 17/20] [5900/6034 images] loss=0.644310 2024-07-26 00:32:59.792592 # [Train Epoch: 17/20] [6000/6034 images] loss=0.625462 2024-07-26 00:33:07 # =====> Epoch: 17 Average loss = 0.629521; Finished in 0:23:46.969443 2024-07-26 00:33:36.921873 # [Train Epoch: 18/20] [100/6034 images] loss=0.620798 2024-07-26 00:34:00.874143 # [Train Epoch: 18/20] [200/6034 images] loss=0.646348 2024-07-26 00:34:24.482922 # [Train Epoch: 18/20] [300/6034 images] loss=0.639068 2024-07-26 00:34:48.104565 # [Train Epoch: 18/20] [400/6034 images] loss=0.601596 2024-07-26 00:35:11.761465 # [Train Epoch: 18/20] [500/6034 images] loss=0.659014 2024-07-26 00:35:35.351197 # [Train Epoch: 18/20] [600/6034 images] loss=0.671283 2024-07-26 00:35:58.935346 # [Train Epoch: 18/20] [700/6034 images] loss=0.631405 2024-07-26 00:36:22.531860 # [Train Epoch: 18/20] [800/6034 images] loss=0.627936 2024-07-26 00:36:46.173542 # [Train Epoch: 18/20] [900/6034 images] loss=0.611682 2024-07-26 00:37:09.802457 # [Train Epoch: 18/20] [1000/6034 images] loss=0.624903 2024-07-26 00:37:33.446514 # [Train Epoch: 18/20] [1100/6034 images] loss=0.629394 2024-07-26 00:37:57.095573 # [Train Epoch: 18/20] [1200/6034 images] loss=0.629295 2024-07-26 00:38:20.734945 # [Train Epoch: 18/20] [1300/6034 images] loss=0.631517 2024-07-26 00:38:44.356659 # [Train Epoch: 18/20] [1400/6034 images] loss=0.634080 2024-07-26 00:39:07.961599 # [Train Epoch: 18/20] [1500/6034 images] loss=0.630342 2024-07-26 00:39:31.572524 # [Train Epoch: 18/20] [1600/6034 images] loss=0.643356 2024-07-26 00:39:55.182159 # [Train Epoch: 18/20] [1700/6034 images] loss=0.659491 2024-07-26 00:40:18.813316 # [Train Epoch: 18/20] [1800/6034 images] loss=0.618579 2024-07-26 00:40:42.429128 # [Train Epoch: 18/20] [1900/6034 images] loss=0.637358 2024-07-26 00:41:06.026522 # [Train Epoch: 18/20] [2000/6034 images] loss=0.613876 2024-07-26 00:41:29.525152 # [Train Epoch: 18/20] [2100/6034 images] loss=0.633756 2024-07-26 00:41:53.090509 # [Train Epoch: 18/20] [2200/6034 images] loss=0.617752 2024-07-26 00:42:16.712277 # [Train Epoch: 18/20] [2300/6034 images] loss=0.653677 2024-07-26 00:42:40.356168 # [Train Epoch: 18/20] [2400/6034 images] loss=0.634993 2024-07-26 00:43:03.983435 # [Train Epoch: 18/20] [2500/6034 images] loss=0.619814 2024-07-26 00:43:27.571647 # [Train Epoch: 18/20] [2600/6034 images] loss=0.652546 2024-07-26 00:43:51.215808 # [Train Epoch: 18/20] [2700/6034 images] loss=0.608754 2024-07-26 00:44:14.845155 # [Train Epoch: 18/20] [2800/6034 images] loss=0.632830 2024-07-26 00:44:38.461778 # [Train Epoch: 18/20] [2900/6034 images] loss=0.615669 2024-07-26 00:45:02.120508 # [Train Epoch: 18/20] [3000/6034 images] loss=0.614793 2024-07-26 00:45:25.734798 # [Train Epoch: 18/20] [3100/6034 images] loss=0.619950 2024-07-26 00:45:49.366558 # [Train Epoch: 18/20] [3200/6034 images] loss=0.620418 2024-07-26 00:46:13.004358 # [Train Epoch: 18/20] [3300/6034 images] loss=0.639479 2024-07-26 00:46:36.632504 # [Train Epoch: 18/20] [3400/6034 images] loss=0.613440 2024-07-26 00:47:00.261298 # [Train Epoch: 18/20] [3500/6034 images] loss=0.616993 2024-07-26 00:47:23.877149 # [Train Epoch: 18/20] [3600/6034 images] loss=0.625539 2024-07-26 00:47:47.496917 # [Train Epoch: 18/20] [3700/6034 images] loss=0.619979 2024-07-26 00:48:11.132100 # [Train Epoch: 18/20] [3800/6034 images] loss=0.624160 2024-07-26 00:48:34.758594 # [Train Epoch: 18/20] [3900/6034 images] loss=0.640755 2024-07-26 00:48:58.402793 # [Train Epoch: 18/20] [4000/6034 images] loss=0.622173 2024-07-26 00:49:22.031767 # [Train Epoch: 18/20] [4100/6034 images] loss=0.620584 2024-07-26 00:49:45.673256 # [Train Epoch: 18/20] [4200/6034 images] loss=0.620318 2024-07-26 00:50:09.259340 # [Train Epoch: 18/20] [4300/6034 images] loss=0.633424 2024-07-26 00:50:32.912356 # [Train Epoch: 18/20] [4400/6034 images] loss=0.628185 2024-07-26 00:50:56.509715 # [Train Epoch: 18/20] [4500/6034 images] loss=0.613278 2024-07-26 00:51:20.144007 # [Train Epoch: 18/20] [4600/6034 images] loss=0.601685 2024-07-26 00:51:43.780355 # [Train Epoch: 18/20] [4700/6034 images] loss=0.636729 2024-07-26 00:52:07.452838 # [Train Epoch: 18/20] [4800/6034 images] loss=0.634034 2024-07-26 00:52:31.072780 # [Train Epoch: 18/20] [4900/6034 images] loss=0.608853 2024-07-26 00:52:54.694036 # [Train Epoch: 18/20] [5000/6034 images] loss=0.616070 2024-07-26 00:53:18.314826 # [Train Epoch: 18/20] [5100/6034 images] loss=0.621158 2024-07-26 00:53:41.942048 # [Train Epoch: 18/20] [5200/6034 images] loss=0.621091 2024-07-26 00:54:05.585018 # [Train Epoch: 18/20] [5300/6034 images] loss=0.628304 2024-07-26 00:54:29.192689 # [Train Epoch: 18/20] [5400/6034 images] loss=0.663418 2024-07-26 00:54:52.831969 # [Train Epoch: 18/20] [5500/6034 images] loss=0.630408 2024-07-26 00:55:16.444613 # [Train Epoch: 18/20] [5600/6034 images] loss=0.623841 2024-07-26 00:55:40.044561 # [Train Epoch: 18/20] [5700/6034 images] loss=0.642830 2024-07-26 00:56:03.680978 # [Train Epoch: 18/20] [5800/6034 images] loss=0.627235 2024-07-26 00:56:27.306307 # [Train Epoch: 18/20] [5900/6034 images] loss=0.639585 2024-07-26 00:56:50.929587 # [Train Epoch: 18/20] [6000/6034 images] loss=0.648527 2024-07-26 00:56:59 # =====> Epoch: 18 Average loss = 0.629519; Finished in 0:23:47.485114 2024-07-26 00:57:27.605743 # [Train Epoch: 19/20] [100/6034 images] loss=0.631597 2024-07-26 00:57:51.240327 # [Train Epoch: 19/20] [200/6034 images] loss=0.659867 2024-07-26 00:58:14.831835 # [Train Epoch: 19/20] [300/6034 images] loss=0.654502 2024-07-26 00:58:38.433941 # [Train Epoch: 19/20] [400/6034 images] loss=0.631415 2024-07-26 00:59:02.068105 # [Train Epoch: 19/20] [500/6034 images] loss=0.639996 2024-07-26 00:59:25.740845 # [Train Epoch: 19/20] [600/6034 images] loss=0.616986 2024-07-26 00:59:49.368912 # [Train Epoch: 19/20] [700/6034 images] loss=0.627096 2024-07-26 01:00:12.981956 # [Train Epoch: 19/20] [800/6034 images] loss=0.643580 2024-07-26 01:00:36.618846 # [Train Epoch: 19/20] [900/6034 images] loss=0.630161 2024-07-26 01:01:00.258123 # [Train Epoch: 19/20] [1000/6034 images] loss=0.639868 2024-07-26 01:01:23.881787 # [Train Epoch: 19/20] [1100/6034 images] loss=0.612895 2024-07-26 01:01:47.518671 # [Train Epoch: 19/20] [1200/6034 images] loss=0.637419 2024-07-26 01:02:11.173190 # [Train Epoch: 19/20] [1300/6034 images] loss=0.641939 2024-07-26 01:02:34.787760 # [Train Epoch: 19/20] [1400/6034 images] loss=0.635245 2024-07-26 01:02:58.403516 # [Train Epoch: 19/20] [1500/6034 images] loss=0.649670 2024-07-26 01:03:21.991821 # [Train Epoch: 19/20] [1600/6034 images] loss=0.637819 2024-07-26 01:03:45.580884 # [Train Epoch: 19/20] [1700/6034 images] loss=0.646003 2024-07-26 01:04:09.158326 # [Train Epoch: 19/20] [1800/6034 images] loss=0.635325 2024-07-26 01:04:32.835027 # [Train Epoch: 19/20] [1900/6034 images] loss=0.633313 2024-07-26 01:04:56.434100 # [Train Epoch: 19/20] [2000/6034 images] loss=0.620792 2024-07-26 01:05:20.077409 # [Train Epoch: 19/20] [2100/6034 images] loss=0.638890 2024-07-26 01:05:43.725753 # [Train Epoch: 19/20] [2200/6034 images] loss=0.618378 2024-07-26 01:06:07.380716 # [Train Epoch: 19/20] [2300/6034 images] loss=0.607985 2024-07-26 01:06:30.997458 # [Train Epoch: 19/20] [2400/6034 images] loss=0.640571 2024-07-26 01:06:54.615546 # [Train Epoch: 19/20] [2500/6034 images] loss=0.626613 2024-07-26 01:07:18.225730 # [Train Epoch: 19/20] [2600/6034 images] loss=0.620959 2024-07-26 01:07:41.842856 # [Train Epoch: 19/20] [2700/6034 images] loss=0.633053 2024-07-26 01:08:05.475777 # [Train Epoch: 19/20] [2800/6034 images] loss=0.626503 2024-07-26 01:08:29.105424 # [Train Epoch: 19/20] [2900/6034 images] loss=0.608082 2024-07-26 01:08:52.736062 # [Train Epoch: 19/20] [3000/6034 images] loss=0.602367 2024-07-26 01:09:16.367029 # [Train Epoch: 19/20] [3100/6034 images] loss=0.658936 2024-07-26 01:09:39.983209 # [Train Epoch: 19/20] [3200/6034 images] loss=0.646814 2024-07-26 01:10:03.596396 # [Train Epoch: 19/20] [3300/6034 images] loss=0.629390 2024-07-26 01:10:27.213383 # [Train Epoch: 19/20] [3400/6034 images] loss=0.651514 2024-07-26 01:10:50.828501 # [Train Epoch: 19/20] [3500/6034 images] loss=0.617186 2024-07-26 01:11:14.428160 # [Train Epoch: 19/20] [3600/6034 images] loss=0.634988 2024-07-26 01:11:38.030994 # [Train Epoch: 19/20] [3700/6034 images] loss=0.629126 2024-07-26 01:12:01.651411 # [Train Epoch: 19/20] [3800/6034 images] loss=0.655509 2024-07-26 01:12:25.290786 # [Train Epoch: 19/20] [3900/6034 images] loss=0.619497 2024-07-26 01:12:48.907463 # [Train Epoch: 19/20] [4000/6034 images] loss=0.618183 2024-07-26 01:13:12.549606 # [Train Epoch: 19/20] [4100/6034 images] loss=0.617856 2024-07-26 01:13:36.191961 # [Train Epoch: 19/20] [4200/6034 images] loss=0.648037 2024-07-26 01:13:59.798549 # [Train Epoch: 19/20] [4300/6034 images] loss=0.637058 2024-07-26 01:14:23.410501 # [Train Epoch: 19/20] [4400/6034 images] loss=0.616414 2024-07-26 01:14:47.059869 # [Train Epoch: 19/20] [4500/6034 images] loss=0.605897 2024-07-26 01:15:10.628014 # [Train Epoch: 19/20] [4600/6034 images] loss=0.642864 2024-07-26 01:15:34.240226 # [Train Epoch: 19/20] [4700/6034 images] loss=0.640116 2024-07-26 01:15:57.868828 # [Train Epoch: 19/20] [4800/6034 images] loss=0.649293 2024-07-26 01:16:21.449323 # [Train Epoch: 19/20] [4900/6034 images] loss=0.637883 2024-07-26 01:16:45.080930 # [Train Epoch: 19/20] [5000/6034 images] loss=0.622234 2024-07-26 01:17:08.691844 # [Train Epoch: 19/20] [5100/6034 images] loss=0.622069 2024-07-26 01:17:32.314839 # [Train Epoch: 19/20] [5200/6034 images] loss=0.626515 2024-07-26 01:17:55.903366 # [Train Epoch: 19/20] [5300/6034 images] loss=0.638065 2024-07-26 01:18:19.519865 # [Train Epoch: 19/20] [5400/6034 images] loss=0.636863 2024-07-26 01:18:43.128898 # [Train Epoch: 19/20] [5500/6034 images] loss=0.619021 2024-07-26 01:19:06.774005 # [Train Epoch: 19/20] [5600/6034 images] loss=0.643172 2024-07-26 01:19:30.385691 # [Train Epoch: 19/20] [5700/6034 images] loss=0.638648 2024-07-26 01:19:53.988645 # [Train Epoch: 19/20] [5800/6034 images] loss=0.641203 2024-07-26 01:20:17.620568 # [Train Epoch: 19/20] [5900/6034 images] loss=0.620258 2024-07-26 01:20:41.209606 # [Train Epoch: 19/20] [6000/6034 images] loss=0.632708 2024-07-26 01:20:49 # =====> Epoch: 19 Average loss = 0.629517; Finished in 0:23:46.782116 2024-07-26 01:21:17.755587 # [Train Epoch: 20/20] [100/6034 images] loss=0.608755 2024-07-26 01:21:41.373699 # [Train Epoch: 20/20] [200/6034 images] loss=0.643354 2024-07-26 01:22:04.980259 # [Train Epoch: 20/20] [300/6034 images] loss=0.623220 2024-07-26 01:22:28.940454 # [Train Epoch: 20/20] [400/6034 images] loss=0.636334 2024-07-26 01:22:52.550103 # [Train Epoch: 20/20] [500/6034 images] loss=0.635392 2024-07-26 01:23:16.134203 # [Train Epoch: 20/20] [600/6034 images] loss=0.644271 2024-07-26 01:23:39.757319 # [Train Epoch: 20/20] [700/6034 images] loss=0.641854 2024-07-26 01:24:03.368683 # [Train Epoch: 20/20] [800/6034 images] loss=0.636212 2024-07-26 01:24:27.000352 # [Train Epoch: 20/20] [900/6034 images] loss=0.651255 2024-07-26 01:24:50.632542 # [Train Epoch: 20/20] [1000/6034 images] loss=0.634010 2024-07-26 01:25:14.259277 # [Train Epoch: 20/20] [1100/6034 images] loss=0.621232 2024-07-26 01:25:37.877868 # [Train Epoch: 20/20] [1200/6034 images] loss=0.657213 2024-07-26 01:26:01.513508 # [Train Epoch: 20/20] [1300/6034 images] loss=0.606820 2024-07-26 01:26:25.146506 # [Train Epoch: 20/20] [1400/6034 images] loss=0.622389 2024-07-26 01:26:48.770674 # [Train Epoch: 20/20] [1500/6034 images] loss=0.624019 2024-07-26 01:27:12.376359 # [Train Epoch: 20/20] [1600/6034 images] loss=0.644730 2024-07-26 01:27:36.020166 # [Train Epoch: 20/20] [1700/6034 images] loss=0.617613 2024-07-26 01:27:59.653474 # [Train Epoch: 20/20] [1800/6034 images] loss=0.628958 2024-07-26 01:28:23.289704 # [Train Epoch: 20/20] [1900/6034 images] loss=0.622762 2024-07-26 01:28:46.929439 # [Train Epoch: 20/20] [2000/6034 images] loss=0.627004 2024-07-26 01:29:10.559859 # [Train Epoch: 20/20] [2100/6034 images] loss=0.621360 2024-07-26 01:29:34.176819 # [Train Epoch: 20/20] [2200/6034 images] loss=0.637050 2024-07-26 01:29:57.793628 # [Train Epoch: 20/20] [2300/6034 images] loss=0.636165 2024-07-26 01:30:21.398130 # [Train Epoch: 20/20] [2400/6034 images] loss=0.621422 2024-07-26 01:30:45.022617 # [Train Epoch: 20/20] [2500/6034 images] loss=0.625328 2024-07-26 01:31:08.644301 # [Train Epoch: 20/20] [2600/6034 images] loss=0.604179 2024-07-26 01:31:32.336907 # [Train Epoch: 20/20] [2700/6034 images] loss=0.643436 2024-07-26 01:31:55.962321 # [Train Epoch: 20/20] [2800/6034 images] loss=0.646632 2024-07-26 01:32:19.588386 # [Train Epoch: 20/20] [2900/6034 images] loss=0.635331 2024-07-26 01:32:43.177028 # [Train Epoch: 20/20] [3000/6034 images] loss=0.626736 2024-07-26 01:33:06.743779 # [Train Epoch: 20/20] [3100/6034 images] loss=0.658776 2024-07-26 01:33:30.307755 # [Train Epoch: 20/20] [3200/6034 images] loss=0.606905 2024-07-26 01:33:53.876578 # [Train Epoch: 20/20] [3300/6034 images] loss=0.661898 2024-07-26 01:34:17.511898 # [Train Epoch: 20/20] [3400/6034 images] loss=0.619430 2024-07-26 01:34:40.984945 # [Train Epoch: 20/20] [3500/6034 images] loss=0.620472 2024-07-26 01:35:04.605250 # [Train Epoch: 20/20] [3600/6034 images] loss=0.617133 2024-07-26 01:35:28.224874 # [Train Epoch: 20/20] [3700/6034 images] loss=0.665840 2024-07-26 01:35:51.848627 # [Train Epoch: 20/20] [3800/6034 images] loss=0.615736 2024-07-26 01:36:15.479521 # [Train Epoch: 20/20] [3900/6034 images] loss=0.610490 2024-07-26 01:36:39.127957 # [Train Epoch: 20/20] [4000/6034 images] loss=0.653534 2024-07-26 01:37:02.772571 # [Train Epoch: 20/20] [4100/6034 images] loss=0.612495 2024-07-26 01:37:26.387718 # [Train Epoch: 20/20] [4200/6034 images] loss=0.642350 2024-07-26 01:37:50.006801 # [Train Epoch: 20/20] [4300/6034 images] loss=0.637203 2024-07-26 01:38:13.609510 # [Train Epoch: 20/20] [4400/6034 images] loss=0.617057 2024-07-26 01:38:37.242028 # [Train Epoch: 20/20] [4500/6034 images] loss=0.618193 2024-07-26 01:39:00.878907 # [Train Epoch: 20/20] [4600/6034 images] loss=0.620886 2024-07-26 01:39:24.497406 # [Train Epoch: 20/20] [4700/6034 images] loss=0.624699 2024-07-26 01:39:48.113845 # [Train Epoch: 20/20] [4800/6034 images] loss=0.635145 2024-07-26 01:40:11.723307 # [Train Epoch: 20/20] [4900/6034 images] loss=0.627381 2024-07-26 01:40:35.316550 # [Train Epoch: 20/20] [5000/6034 images] loss=0.607748 2024-07-26 01:40:58.927295 # [Train Epoch: 20/20] [5100/6034 images] loss=0.615319 2024-07-26 01:41:22.550314 # [Train Epoch: 20/20] [5200/6034 images] loss=0.612360 2024-07-26 01:41:46.181740 # [Train Epoch: 20/20] [5300/6034 images] loss=0.601920 2024-07-26 01:42:09.843588 # [Train Epoch: 20/20] [5400/6034 images] loss=0.622127 2024-07-26 01:42:33.435552 # [Train Epoch: 20/20] [5500/6034 images] loss=0.606135 2024-07-26 01:42:57.058923 # [Train Epoch: 20/20] [5600/6034 images] loss=0.625183 2024-07-26 01:43:20.669368 # [Train Epoch: 20/20] [5700/6034 images] loss=0.593739 2024-07-26 01:43:44.281128 # [Train Epoch: 20/20] [5800/6034 images] loss=0.648652 2024-07-26 01:44:07.903076 # [Train Epoch: 20/20] [5900/6034 images] loss=0.667879 2024-07-26 01:44:31.506014 # [Train Epoch: 20/20] [6000/6034 images] loss=0.615976 2024-07-26 01:44:39 # =====> Epoch: 20 Average loss = 0.629515; Finished in 0:23:47.075703 2024-07-26 01:44:43 Finished in 7:57:20.402279 (0:23:52.020114 per epoch)

It's wired that the loss hardrly drops. Am I using the wrong command? Or the batchsize influences?

The cryoNeFEN result looks unsatisfactory image

The one with cryosparc resonstruction looks like image which is really well with a resolution ~3A.

Do you know what causes the result? Meanwhile, the running consumes ~39G memory in my A100 GPU. Why does it consume so much memory?

yuehuang2023 commented 4 months ago

The unsatisfied result may be caused by the applied symmetry. The image size 440 pix increases the consumption of GPU memory. Downsampling the particles may solve the consumption problem.

emberslee commented 4 months ago

The unsatisfied result may be caused by the applied symmetry. The image size 440 pix increases the consumption of GPU memory. Downsampling the particles may solve the consumption problem.

Thank you, I will try a new setting.

yuehuang2023 commented 4 months ago

The number of images is small ~6000, which may require more epochs to train as well as enforcing symmetry.