search

lfranke / TRIPS

https://lfranke.github.io/trips/
MIT License
513 stars 30 forks source link

Training time? #12

Open antithing opened 7 months ago

antithing commented 7 months ago

Hi, I have the code built and have run the colmap conversation and the point cloud pre process. (I used render scale = 0.25)

I am using the mipnerf360 garden data set .

Training has now been running for 12 hours and is up to epoch 106.

I see the config is set to 600 epochs, so I assume another few days of training time.

Is this the expected time to train? I am on a rtx 3090 desktop.

Thanks!

lfranke commented 7 months ago

Hi, that seems unusually long. How many points are in your point cloud? I have been working on an RTX4090 and training took more like 4h, I don't think the 3090 should be that much slower.

antithing commented 7 months ago

It's the mipnerf 360 garden, and I ran the point cloud pre process with a factor of 2.

antithing commented 7 months ago

...I am trying again without the pre process (points = 136412) so I will report back!

antithing commented 7 months ago

Running the standard garden dataset (without pre-processing) runs great until around epoch 105, then it slows considerably (0 -to 7% in 30 mins).

I am trying with batch size reductions as:

train --config configs/train_normalnet.ini --TrainParams.scene_names garden_test --TrainParams.batch_size 1 --TrainParams.inner_batch_size 1 --TrainParams.train_crop_size 256

I will see if that helps... Any other tips appreciated!

lfranke commented 7 months ago

I was not able to figure this out directly, sorry! I notice you seem to take the sparse output of COLMAP, our method requires the dense output btw. However this should not impact speed.

I will upload the mipnerf360 datasets and checkpoints we used soon (at least the 7 that are openly sharable), so you can try out the result at least :)

antithing commented 7 months ago

Thank you, that would be great.

Using the train --config configs/train_normalnet.ini --TrainParams.scene_names garden_test --TrainParams.batch_size 1 --TrainParams.inner_batch_size 1 --TrainParams.train_crop_size 256 with a render scale of 0.25 I was able to train, but he result is pretty patchy due to lowering the res! I will try a colmap dense cloud as well. Thanks again!

lfranke commented 7 months ago

I have added the 7 scenes from the mipnerf dataset (outdoor and indoor half resolution as in our evaluation) as well as the boat to the supplemental: https://zenodo.org/records/10641253

antithing commented 7 months ago

Thank you! I have now computed the dense cloud with colmap, and running with:

train --config configs/train_normalnet.ini --TrainParams.scene_names garden_dense --TrainParams.batch_size 1 --TrainParams.inner_batch_size 1 --TrainParams.train_crop_size 256

and a render_scale of 0.25 fails after the 'test' run.

train --config configs/train_normalnet.ini --TrainParams.scene_names garden_dense --TrainParams.batch_size 1 --TrainParams.inner_batch_size 1 --TrainParams.train_crop_size 256
register neural render info
register TnnInfo
PyTorch version: 2.1.0
The cuDNN version is 8801
cuDNN avail? 1
The CUDA runtime version is 12010
The driver version is 12020
Loading Config File configs/train_normalnet.ini
Using Random Seed: 2147483647
POINT GRADIENTS ARE COMPUTED.
torch::cuda::cudnn_is_available() 1
Render Mode epochs: DT -2 - Fullblend -2 - Fuzzyblend -2 - BilinearBlend -2 - FastBlend 0
Use NeAT recogarden_dense
====================================
Scene Loaded
  Name       garden_dense
  Path       D:\NERF\TRIPS\scenes\garden_dense
  Image Size 5187x3361
  Aspect     1.54329
  K          3844.9 3852.36 2593.5 1680.5 0
  ocam       5187x3361 affine(1, 0, 0, 0, 0) cam2world() world2cam()
  ocam cut   1
  normalized center 0 0
  dist       0 0 0 0 0 0 0 0
CAM model: CameraModel::PINHOLE_DISTORTION
  Points     7877153
  Colors     1
  Normals    1
  Avg. EV  0
  Num Images 185
  Num Cameras 1
Compute scene importance bounding box as 95% of points interval around center of mass
Starting Compute center of mass...center of mass:0.308283
2.15885
1.12378
 Done in 74.9361ms.
Starting Build range vec... Done in 46.5421ms.
Starting Sort range vec... Done in 889.808ms.
Starting Extend box... Done in 161ms.
Box: AABB: [-12.1599 -5.48379 -11.3698 ] [12.8644 6.71304 12.093 ]
====================================
Modulo stepsize: 8
Train(161): 1 2 3 4 5 6 7 9 10 11 12 13 14 15 17 18 19 20 21 22 23 25 26 27 28 29 30 31 33 34 35 36 37 38 39 41 42 43 44 45 46 47 49 50 51 52 53 54 55 57 58 59 60 61 62 63 65 66 67 68 69 70 71 73 74 75 76 77 78 79 81 82 83 84 85 86 87 89 90 91 92 93 94 95 97 98 99 100 101 102 103 105 106 107 108 109 110 111 113 114 115 116 117 118 119 121 122 123 124 125 126 127 129 130 131 132 133 134 135 137 138 139 140 141 142 143 145 146 147 148 149 150 151 153 154 155 156 157 158 159 161 162 163 164 165 166 167 169 170 171 172 173 174 175 177 178 179 180 181 182 183
Test(24): 0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184
cropped sampler 5187x3361 to 256 x 256 render scale 0.25
cropped sampler 5187x3361 to 256 x 256 render scale 0.25
full sampler 1296x840 render scale 0.25
full sampler 1296x840 render scale 0.25
POINT GRADIENTS ARE COMPUTED.
GPU memory - Point Cloud: 283.578MB
Pinhole Intrinsics: 3844.9 3852.36 2593.5 1680.5 0  -- 0 0 0 0 0 0 0 0
Tensor [1, 13] float cuda:0 Min/Max 0 3852.36 Mean 920.866 Sum 11971.3 sdev 1531.62 req-grad 0 Min-Coords: [0, 4, ]
GPU memory - Texture [4, 7877153] : 189.052MB
Using Mm Adam texture optimizer
optimizing texture with lr 0.1/0.004
optimizing response with lr 0.0001
optimizing exposure with lr 0.0005
Optimizing with my adam implementation.
optimizing 3D points with lr 0.0001
optimizing point size with lr 0.01
optimizing poses with lr 0.0001
no intrinsics optimizer
POINT GRADIENTS ARE COMPUTED.
Using MultiScaleUnet2dDecOnlySmall with filters:
32 32 32 32 32 32 32 32
RENDER NETWORK: NUM PARAMETERS: 22
Total Model Params: 59675
Loading VGG from: loss/traced_caffe_vgg_optim.pt

=== Epoch 0 ===
Scene Log - Texture: Tensor [4, 7877153] float cpu Min/Max 0 1 Mean 0.499994 Sum 1.57541e+07 sdev 0.288652 req-grad 1 Min-Coords: [0, 2355555, ]
  Background Desc:
  Confidence per point: Tensor [1, 7877153] float cpu Min/Max 0.5 0.5 Mean 0.5 Sum 3.93858e+06 sdev 0 req-grad 0 Min-Coords: [0, 0, ]
  Confidences under 0.5: 7877153
  LayerBuf per point: Tensor [7877153, 1] float cuda:0 Min/Max -8.03805 17.4942 Mean -5.37804 Sum -4.23637e+07 sdev 0.689841 req-grad 1 Min-Coords: [6001664, 0, ]
      softplus: Tensor [7877153, 1] float cuda:0 Min/Max 0.000322886 17.4942 Mean 0.00614902 Sum 48436.8 sdev 0.0173115 req-grad 1 Min-Coords: [6001664, 0, ]
  Poses: Tensor [185, 8] double cuda:0 Min/Max -1.98294 4.1659 Mean 0.33086 Sum 489.672 sdev 1.33716 req-grad 0 Min-Coords: [117, 5, ]
  Point Position: Tensor [7877153, 4] float cuda:0 Min/Max -81.9788 96.2361 Mean 0.903868 Sum 2.84796e+07 sdev 3.4264 req-grad 1 Min-Coords: [4570866, 2, ]
Eval  0 |   0% |                              |   0/161 [00:00:0000] [0.00 e/s] USING NEW IMPL
Eval  0 | 100% |##############################| 161/161 [00:58:0019] [2.77 e/s]  Cur=1510.437378 Avg=1545.123047
Best - Worst (Eval) [1461.39, 1650.78]
Loss:         garden_dense Param 0 VGG 1544.31 L1 0.363617 MSE 0.183437 PSNR 7.38558 SSIM 0.447156 LPIPS 0.966876 count 161
Test  0 | 100% |##############################| 24/24 [00:14:0136] [1.70 e/s]  Cur=1526.503418 Avg=1546.691284
Best - Worst (Eval) [1493.11, 1638.31]
Loss:         garden_dense Param 0 VGG 1545.88 L1 0.362421 MSE 0.182466 PSNR 7.40616 SSIM 0.447356 LPIPS 0.966075 count 24
Saving checkpoint...

I skipped the point pre-process step here as well. Is it simply a hardware issue? Can I downsample anything further?

...the colmaptoadop step outputs:

colmap2adop --sparse_dir D:\NERF\data\garden\sparse\0\  --image_dir D:\NERF\data\garden\images  --point_cloud_file D:\NERF\data\GardenDense\dense\pointCLoudDense.ply --output_path D:\NERF\TRIPS\scenes/garden_dense --scale_intrinsics 1 --render_scale 1
register neural render info
register TnnInfo
Preprocessing Colmap scene D:\NERF\data\garden\sparse\0\ -> D:\NERF\TRIPS\scenes/garden_dense
Num cameras 1
id: 0 camera model: 1  K: 3844.9 3852.36 2593.5 1680.5 0  Dis: 0 0 0 0 0 0 0 0
Num images 185
DSC07956.JPG 1 -> 0 0  Position: -0.328303 -1.92596 3.95806
DSC07957.JPG 2 -> 1 0  Position: -0.339415 -1.93374 3.83564
DSC07958.JPG 3 -> 2 0  Position: -0.195422 -1.8746 3.78042
DSC07959.JPG 4 -> 3 0  Position: -0.101426 -1.85149 3.62929
DSC07960.JPG 5 -> 4 0  Position: -0.166749 -1.7962 3.61052
DSC07961.JPG 6 -> 5 0  Position: 0.103536 -1.74608 3.51963
DSC07962.JPG 7 -> 6 0  Position: 0.139471 -1.7349 3.56568
DSC07963.JPG 8 -> 7 0  Position: 0.237843 -1.66885 3.5316
DSC07964.JPG 9 -> 8 0  Position: 0.322959 -1.73085 3.49269
DSC07965.JPG 10 -> 9 0  Position: 0.411638 -1.68849 3.54808
DSC07966.JPG 11 -> 10 0  Position: 0.481593 -1.65 3.51928
DSC07967.JPG 12 -> 11 0  Position: 0.319985 -1.70972 3.69101
DSC07968.JPG 13 -> 12 0  Position: 0.303116 -1.72452 3.82456
DSC07969.JPG 14 -> 13 0  Position: 0.33985 -1.80291 3.69163
DSC07970.JPG 15 -> 14 0  Position: 0.302801 -1.86831 3.56633
DSC07971.JPG 16 -> 15 0  Position: 0.264994 -1.68942 3.617
DSC07972.JPG 17 -> 16 0  Position: 0.481547 -1.75262 3.65521
DSC07973.JPG 18 -> 17 0  Position: 0.392175 -1.81316 3.78582
DSC07974.JPG 19 -> 18 0  Position: 0.29677 -1.78424 3.87035
DSC07975.JPG 20 -> 19 0  Position: 0.2139 -1.66939 3.89093
DSC07976.JPG 21 -> 20 0  Position: 0.236366 -1.79084 3.67281
DSC07977.JPG 22 -> 21 0  Position: 0.209117 -1.86742 3.80086
DSC07978.JPG 23 -> 22 0  Position: -0.0204816 -1.86802 3.68511
DSC07979.JPG 24 -> 23 0  Position: 0.0360777 -1.8221 3.64822
DSC07980.JPG 25 -> 24 0  Position: -0.0700306 -1.91849 3.44788
DSC07981.JPG 26 -> 25 0  Position: -0.0519024 -1.89668 3.21038
DSC07982.JPG 27 -> 26 0  Position: -0.282299 -1.91702 3.16808
DSC07983.JPG 28 -> 27 0  Position: -0.337168 -1.79231 3.305
DSC07984.JPG 29 -> 28 0  Position: -0.390926 -1.76908 3.45993
DSC07985.JPG 30 -> 29 0  Position: -0.410164 -1.71716 3.62934
DSC07986.JPG 31 -> 30 0  Position: -0.429303 -1.68239 3.90802
DSC07987.JPG 32 -> 31 0  Position: -0.381233 -1.57431 4.16293
DSC07988.JPG 33 -> 32 0  Position: -0.442618 -1.62928 4.1659
DSC07989.JPG 34 -> 33 0  Position: -0.262865 -1.82867 4.00413
DSC07990.JPG 35 -> 34 0  Position: -0.375365 -1.8418 3.84879
DSC07991.JPG 36 -> 35 0  Position: -0.256106 -1.75244 3.81778
DSC07992.JPG 37 -> 36 0  Position: -0.154271 -1.66268 3.66817
DSC07993.JPG 38 -> 37 0  Position: -0.0716001 -1.68759 3.50432
DSC07994.JPG 39 -> 38 0  Position: -0.0333679 -1.61363 3.42395
DSC07995.JPG 40 -> 39 0  Position: 0.00734142 -1.57905 3.37089
DSC07996.JPG 41 -> 40 0  Position: 0.0522469 -1.65532 3.18136
DSC07997.JPG 42 -> 41 0  Position: -0.174225 -1.57989 3.2987
DSC07998.JPG 43 -> 42 0  Position: 0.210844 -1.72202 3.15155
DSC07999.JPG 44 -> 43 0  Position: 0.255498 -1.71542 3.21001
DSC08000.JPG 45 -> 44 0  Position: 0.164915 -1.80741 3.13076
DSC08001.JPG 46 -> 45 0  Position: 0.234389 -1.78789 3.16399
DSC08002.JPG 47 -> 46 0  Position: 0.33887 -1.79376 3.09271
DSC08003.JPG 48 -> 47 0  Position: 0.442813 -1.79206 3.07863
DSC08004.JPG 49 -> 48 0  Position: 0.428228 -1.81625 3.10165
DSC08005.JPG 50 -> 49 0  Position: 0.49012 -1.81144 3.18419
DSC08006.JPG 51 -> 50 0  Position: 0.458612 -1.8135 3.21591
DSC08007.JPG 52 -> 51 0  Position: 0.460178 -1.87712 3.24413
DSC08008.JPG 53 -> 52 0  Position: 0.422927 -1.92417 3.3632
DSC08009.JPG 54 -> 53 0  Position: 0.362306 -1.9162 3.5076
DSC08010.JPG 55 -> 54 0  Position: 0.436953 -1.98225 3.56853
DSC08011.JPG 56 -> 55 0  Position: 0.322273 -1.92608 3.55255
DSC08012.JPG 57 -> 56 0  Position: 0.166861 -1.89006 3.55628
DSC08013.JPG 58 -> 57 0  Position: 0.174541 -1.95644 3.64198
DSC08014.JPG 59 -> 58 0  Position: 0.0339834 -1.9527 3.48767
DSC08015.JPG 60 -> 59 0  Position: -0.190494 -1.97407 3.36099
DSC08016.JPG 61 -> 60 0  Position: -0.157105 -1.94758 3.01321
DSC08017.JPG 62 -> 61 0  Position: -0.319465 -1.95852 2.89761
DSC08018.JPG 63 -> 62 0  Position: -0.339131 -1.88627 2.94776
DSC08019.JPG 64 -> 63 0  Position: -0.435145 -1.82774 3.02637
DSC08020.JPG 65 -> 64 0  Position: -0.469142 -1.79883 3.03505
DSC08021.JPG 66 -> 65 0  Position: -0.37875 -1.81588 3.21317
DSC08022.JPG 67 -> 66 0  Position: -0.418187 -1.83548 3.29344
DSC08023.JPG 68 -> 67 0  Position: -0.401813 -1.74309 3.6098
DSC08024.JPG 69 -> 68 0  Position: -0.330303 -1.75318 3.6155
DSC08025.JPG 70 -> 69 0  Position: -0.232523 -1.35283 3.86754
DSC08026.JPG 71 -> 70 0  Position: -0.283491 -1.34617 3.69206
DSC08027.JPG 72 -> 71 0  Position: -0.23422 -1.34787 3.51592
DSC08028.JPG 73 -> 72 0  Position: -0.112677 -1.46191 3.36843
DSC08029.JPG 74 -> 73 0  Position: -0.0814558 -1.50297 3.20059
DSC08030.JPG 75 -> 74 0  Position: 0.0311563 -1.55018 3.10799
DSC08031.JPG 76 -> 75 0  Position: 0.137555 -1.5542 3.06744
DSC08032.JPG 77 -> 76 0  Position: 0.24062 -1.6063 3.00469
DSC08033.JPG 78 -> 77 0  Position: 0.455599 -1.60725 3.01644
DSC08034.JPG 79 -> 78 0  Position: 0.375231 -1.65729 3.00104
DSC08035.JPG 80 -> 79 0  Position: 0.374385 -1.73958 3.04413
DSC08036.JPG 81 -> 80 0  Position: 0.310069 -1.69982 3.04833
DSC08037.JPG 82 -> 81 0  Position: 0.325087 -1.72593 3.07855
DSC08038.JPG 83 -> 82 0  Position: 0.494657 -1.74965 3.04628
DSC08039.JPG 84 -> 83 0  Position: 0.527438 -1.7984 3.07755
DSC08040.JPG 85 -> 84 0  Position: 0.663991 -1.82155 3.20621
DSC08041.JPG 86 -> 85 0  Position: 0.484877 -1.88004 3.25265
DSC08042.JPG 87 -> 86 0  Position: 0.465191 -1.8906 3.37447
DSC08043.JPG 88 -> 87 0  Position: 0.318649 -1.94291 3.30323
DSC08044.JPG 89 -> 88 0  Position: 0.298463 -1.93945 3.43248
DSC08045.JPG 90 -> 89 0  Position: 0.0966744 -1.86706 3.44138
DSC08046.JPG 91 -> 90 0  Position: 0.0852767 -1.94443 3.29111
DSC08047.JPG 92 -> 91 0  Position: -0.147439 -1.8713 2.73255
DSC08048.JPG 93 -> 92 0  Position: -0.213857 -1.84745 2.89573
DSC08049.JPG 94 -> 93 0  Position: -0.24506 -1.85226 3.1156
DSC08050.JPG 95 -> 94 0  Position: -0.452214 -1.8025 3.14637
DSC08051.JPG 96 -> 95 0  Position: -0.468396 -1.73913 3.12803
DSC08052.JPG 97 -> 96 0  Position: -0.362975 -1.78204 2.98397
DSC08053.JPG 98 -> 97 0  Position: -0.332308 -1.80049 2.98527
DSC08054.JPG 99 -> 98 0  Position: -0.213014 -1.73471 3.24814
DSC08055.JPG 100 -> 99 0  Position: -0.385408 -1.47091 3.45605
DSC08056.JPG 101 -> 100 0  Position: -0.57804 -1.39854 3.36971
DSC08057.JPG 102 -> 101 0  Position: -0.26028 -1.41635 3.35374
DSC08058.JPG 103 -> 102 0  Position: -0.119723 -1.3399 3.28544
DSC08059.JPG 104 -> 103 0  Position: -0.146615 -1.33603 3.40775
DSC08060.JPG 105 -> 104 0  Position: -0.00694941 -1.30105 3.27462
DSC08061.JPG 106 -> 105 0  Position: 0.111974 -1.33533 3.32091
DSC08062.JPG 107 -> 106 0  Position: 0.207543 -1.36459 3.34961
DSC08063.JPG 108 -> 107 0  Position: 0.316874 -1.36987 3.30271
DSC08064.JPG 109 -> 108 0  Position: 0.276097 -1.38934 3.41758
DSC08065.JPG 110 -> 109 0  Position: 0.37282 -1.47768 3.35035
DSC08066.JPG 111 -> 110 0  Position: 0.42564 -1.54508 3.38425
DSC08067.JPG 112 -> 111 0  Position: 0.487805 -1.68635 3.17881
DSC08068.JPG 113 -> 112 0  Position: 0.509142 -1.79065 3.31939
DSC08069.JPG 114 -> 113 0  Position: 0.459552 -1.81121 3.44258
DSC08070.JPG 115 -> 114 0  Position: 0.367526 -1.83496 3.58794
DSC08071.JPG 116 -> 115 0  Position: 0.330328 -1.94092 3.70551
DSC08072.JPG 117 -> 116 0  Position: 0.291786 -1.92346 3.85881
DSC08073.JPG 118 -> 117 0  Position: 0.297219 -1.98294 3.54675
DSC08074.JPG 119 -> 118 0  Position: 0.1044 -1.93843 3.6135
DSC08075.JPG 120 -> 119 0  Position: 0.0849007 -1.77155 3.69038
DSC08076.JPG 121 -> 120 0  Position: -0.158823 -1.79276 3.38485
DSC08077.JPG 122 -> 121 0  Position: -0.0978079 -1.68023 3.3131
DSC08078.JPG 123 -> 122 0  Position: -0.311782 -1.6379 2.92122
DSC08079.JPG 124 -> 123 0  Position: -0.388204 -1.61957 2.97079
DSC08080.JPG 125 -> 124 0  Position: -0.411853 -1.5716 2.82786
DSC08081.JPG 126 -> 125 0  Position: -0.357626 -1.54196 2.91663
DSC08082.JPG 127 -> 126 0  Position: -0.341887 -1.58438 3.08313
DSC08083.JPG 128 -> 127 0  Position: -0.298701 -1.57322 3.13813
DSC08084.JPG 129 -> 128 0  Position: -0.232759 -1.27713 3.18764
DSC08085.JPG 130 -> 129 0  Position: -0.320486 -1.32108 3.12531
DSC08086.JPG 131 -> 130 0  Position: -0.230607 -1.03703 3.16802
DSC08087.JPG 132 -> 131 0  Position: -0.165147 -1.19958 3.09174
DSC08088.JPG 133 -> 132 0  Position: -0.124163 -1.24252 2.94369
DSC08089.JPG 134 -> 133 0  Position: 0.0726911 -1.20688 2.92207
DSC08090.JPG 135 -> 134 0  Position: 0.115787 -1.2315 2.93086
DSC08091.JPG 136 -> 135 0  Position: 0.192219 -1.21575 2.90137
DSC08092.JPG 137 -> 136 0  Position: 0.336076 -1.24159 2.94439
DSC08093.JPG 138 -> 137 0  Position: 0.479569 -1.19402 2.99701
DSC08094.JPG 139 -> 138 0  Position: 0.433556 -1.37479 3.01975
DSC08095.JPG 140 -> 139 0  Position: 0.473003 -1.44898 3.07226
DSC08096.JPG 141 -> 140 0  Position: 0.487487 -1.43973 3.05791
DSC08097.JPG 142 -> 141 0  Position: 0.527668 -1.65442 3.02645
DSC08098.JPG 143 -> 142 0  Position: 0.397056 -1.77868 2.89683
DSC08099.JPG 144 -> 143 0  Position: 0.477238 -1.90962 3.0736
DSC08100.JPG 145 -> 144 0  Position: 0.0701699 -1.89448 3.25384
DSC08101.JPG 146 -> 145 0  Position: 0.173458 -1.89492 3.12595
DSC08102.JPG 147 -> 146 0  Position: 0.0554461 -1.69799 3.37318
DSC08103.JPG 148 -> 147 0  Position: 0.0168238 -1.64063 3.34841
DSC08104.JPG 149 -> 148 0  Position: -0.127735 -1.69305 3.34099
DSC08105.JPG 150 -> 149 0  Position: -0.342714 -1.86512 3.09336
DSC08106.JPG 151 -> 150 0  Position: -0.284415 -1.80548 2.86986
DSC08107.JPG 152 -> 151 0  Position: -0.277267 -1.6395 2.85679
DSC08108.JPG 153 -> 152 0  Position: -0.425163 -1.58235 2.81091
DSC08109.JPG 154 -> 153 0  Position: -0.507894 -1.55113 2.89469
DSC08110.JPG 155 -> 154 0  Position: -0.52058 -1.60501 2.82636
DSC08111.JPG 156 -> 155 0  Position: -0.381229 -1.55823 2.83395
DSC08112.JPG 157 -> 156 0  Position: -0.57999 -1.49777 2.86486
DSC08113.JPG 158 -> 157 0  Position: -0.38627 -1.48188 2.93633
DSC08114.JPG 159 -> 158 0  Position: -0.258888 -1.2035 3.74586
DSC08115.JPG 160 -> 159 0  Position: -0.175381 -1.16148 3.75502
DSC08116.JPG 161 -> 160 0  Position: -0.268922 -1.17119 3.57105
DSC08117.JPG 162 -> 161 0  Position: -0.246106 -1.1431 3.68389
DSC08118.JPG 163 -> 162 0  Position: 0.118409 -1.06575 3.44983
DSC08119.JPG 164 -> 163 0  Position: 0.100933 -1.25282 3.38293
DSC08120.JPG 165 -> 164 0  Position: 0.097242 -1.22811 3.5222
DSC08121.JPG 166 -> 165 0  Position: 0.273065 -1.23686 3.73239
DSC08122.JPG 167 -> 166 0  Position: 0.277591 -1.10776 3.57431
DSC08123.JPG 168 -> 167 0  Position: 0.330968 -1.15043 3.24851
DSC08124.JPG 169 -> 168 0  Position: 0.450116 -1.28972 3.39463
DSC08125.JPG 170 -> 169 0  Position: 0.262321 -1.24024 3.50949
DSC08126.JPG 171 -> 170 0  Position: 0.518222 -1.44805 3.45459
DSC08127.JPG 172 -> 171 0  Position: 0.388204 -1.65362 3.50501
DSC08128.JPG 173 -> 172 0  Position: 0.375651 -1.58983 3.51948
DSC08129.JPG 174 -> 173 0  Position: 0.317699 -1.63949 3.71252
DSC08130.JPG 175 -> 174 0  Position: 0.182846 -1.63953 3.46624
DSC08131.JPG 176 -> 175 0  Position: -0.0128081 -1.62026 3.66195
DSC08132.JPG 177 -> 176 0  Position: -0.205225 -1.58577 3.69358
DSC08133.JPG 178 -> 177 0  Position: -0.219897 -1.55109 3.13919
DSC08134.JPG 179 -> 178 0  Position: -0.349114 -1.59975 2.92843
DSC08135.JPG 180 -> 179 0  Position: -0.417455 -1.39586 3.33858
DSC08136.JPG 181 -> 180 0  Position: -0.229331 -1.55632 3.23761
DSC08137.JPG 182 -> 181 0  Position: -0.483666 -1.38332 3.65131
DSC08138.JPG 183 -> 182 0  Position: -0.581307 -1.40212 3.7283
DSC08139.JPG 184 -> 183 0  Position: -0.366571 -1.43995 3.5898
DSC08140.JPG 185 -> 184 0  Position: -0.34099 -1.15868 3.95895
Num Point3D 138766
No EXIF exposure value found for image DSC07956.JPG
No EXIF exposure value found for image DSC07957.JPG
No EXIF exposure value found for image DSC07958.JPG
No EXIF exposure value found for image DSC07959.JPG
No EXIF exposure value found for image DSC07960.JPG
No EXIF exposure value found for image DSC07961.JPG
No EXIF exposure value found for image DSC07962.JPG
No EXIF exposure value found for image DSC07963.JPG
No EXIF exposure value found for image DSC07964.JPG
No EXIF exposure value found for image DSC07965.JPG
No EXIF exposure value found for image DSC07966.JPG
No EXIF exposure value found for image DSC07967.JPG
No EXIF exposure value found for image DSC07968.JPG
No EXIF exposure value found for image DSC07969.JPG
No EXIF exposure value found for image DSC07970.JPG
No EXIF exposure value found for image DSC07971.JPG
No EXIF exposure value found for image DSC07972.JPG
No EXIF exposure value found for image DSC07973.JPG
No EXIF exposure value found for image DSC07974.JPG
No EXIF exposure value found for image DSC07975.JPG
No EXIF exposure value found for image DSC07976.JPG
No EXIF exposure value found for image DSC07977.JPG
No EXIF exposure value found for image DSC07978.JPG
No EXIF exposure value found for image DSC07979.JPG
No EXIF exposure value found for image DSC07980.JPG
No EXIF exposure value found for image DSC07981.JPG
No EXIF exposure value found for image DSC07982.JPG
No EXIF exposure value found for image DSC07983.JPG
No EXIF exposure value found for image DSC07984.JPG
No EXIF exposure value found for image DSC07985.JPG
No EXIF exposure value found for image DSC07986.JPG
No EXIF exposure value found for image DSC07987.JPG
No EXIF exposure value found for image DSC07988.JPG
No EXIF exposure value found for image DSC07989.JPG
No EXIF exposure value found for image DSC07990.JPG
No EXIF exposure value found for image DSC07991.JPG
No EXIF exposure value found for image DSC07992.JPG
No EXIF exposure value found for image DSC07993.JPG
No EXIF exposure value found for image DSC07994.JPG
No EXIF exposure value found for image DSC07995.JPG
No EXIF exposure value found for image DSC07996.JPG
No EXIF exposure value found for image DSC07997.JPG
No EXIF exposure value found for image DSC07998.JPG
No EXIF exposure value found for image DSC07999.JPG
No EXIF exposure value found for image DSC08000.JPG
No EXIF exposure value found for image DSC08001.JPG
No EXIF exposure value found for image DSC08002.JPG
No EXIF exposure value found for image DSC08003.JPG
No EXIF exposure value found for image DSC08004.JPG
No EXIF exposure value found for image DSC08005.JPG
No EXIF exposure value found for image DSC08006.JPG
No EXIF exposure value found for image DSC08007.JPG
No EXIF exposure value found for image DSC08008.JPG
No EXIF exposure value found for image DSC08009.JPG
No EXIF exposure value found for image DSC08010.JPG
No EXIF exposure value found for image DSC08011.JPG
No EXIF exposure value found for image DSC08012.JPG
No EXIF exposure value found for image DSC08013.JPG
No EXIF exposure value found for image DSC08014.JPG
No EXIF exposure value found for image DSC08015.JPG
No EXIF exposure value found for image DSC08016.JPG
No EXIF exposure value found for image DSC08017.JPG
No EXIF exposure value found for image DSC08018.JPG
No EXIF exposure value found for image DSC08019.JPG
No EXIF exposure value found for image DSC08020.JPG
No EXIF exposure value found for image DSC08021.JPG
No EXIF exposure value found for image DSC08022.JPG
No EXIF exposure value found for image DSC08023.JPG
No EXIF exposure value found for image DSC08024.JPG
No EXIF exposure value found for image DSC08025.JPG
No EXIF exposure value found for image DSC08026.JPG
No EXIF exposure value found for image DSC08027.JPG
No EXIF exposure value found for image DSC08028.JPG
No EXIF exposure value found for image DSC08029.JPG
No EXIF exposure value found for image DSC08030.JPG
No EXIF exposure value found for image DSC08031.JPG
No EXIF exposure value found for image DSC08032.JPG
No EXIF exposure value found for image DSC08033.JPG
No EXIF exposure value found for image DSC08034.JPG
No EXIF exposure value found for image DSC08035.JPG
No EXIF exposure value found for image DSC08036.JPG
No EXIF exposure value found for image DSC08037.JPG
No EXIF exposure value found for image DSC08038.JPG
No EXIF exposure value found for image DSC08039.JPG
No EXIF exposure value found for image DSC08040.JPG
No EXIF exposure value found for image DSC08041.JPG
No EXIF exposure value found for image DSC08042.JPG
No EXIF exposure value found for image DSC08043.JPG
No EXIF exposure value found for image DSC08044.JPG
No EXIF exposure value found for image DSC08045.JPG
No EXIF exposure value found for image DSC08046.JPG
No EXIF exposure value found for image DSC08047.JPG
No EXIF exposure value found for image DSC08048.JPG
No EXIF exposure value found for image DSC08049.JPG
No EXIF exposure value found for image DSC08050.JPG
No EXIF exposure value found for image DSC08051.JPG
No EXIF exposure value found for image DSC08052.JPG
No EXIF exposure value found for image DSC08053.JPG
No EXIF exposure value found for image DSC08054.JPG
No EXIF exposure value found for image DSC08055.JPG
No EXIF exposure value found for image DSC08056.JPG
No EXIF exposure value found for image DSC08057.JPG
No EXIF exposure value found for image DSC08058.JPG
No EXIF exposure value found for image DSC08059.JPG
No EXIF exposure value found for image DSC08060.JPG
No EXIF exposure value found for image DSC08061.JPG
No EXIF exposure value found for image DSC08062.JPG
No EXIF exposure value found for image DSC08063.JPG
No EXIF exposure value found for image DSC08064.JPG
No EXIF exposure value found for image DSC08065.JPG
No EXIF exposure value found for image DSC08066.JPG
No EXIF exposure value found for image DSC08067.JPG
No EXIF exposure value found for image DSC08068.JPG
No EXIF exposure value found for image DSC08069.JPG
No EXIF exposure value found for image DSC08070.JPG
No EXIF exposure value found for image DSC08071.JPG
No EXIF exposure value found for image DSC08072.JPG
No EXIF exposure value found for image DSC08073.JPG
No EXIF exposure value found for image DSC08074.JPG
No EXIF exposure value found for image DSC08075.JPG
No EXIF exposure value found for image DSC08076.JPG
No EXIF exposure value found for image DSC08077.JPG
No EXIF exposure value found for image DSC08078.JPG
No EXIF exposure value found for image DSC08079.JPG
No EXIF exposure value found for image DSC08080.JPG
No EXIF exposure value found for image DSC08081.JPG
No EXIF exposure value found for image DSC08082.JPG
No EXIF exposure value found for image DSC08083.JPG
No EXIF exposure value found for image DSC08084.JPG
No EXIF exposure value found for image DSC08085.JPG
No EXIF exposure value found for image DSC08086.JPG
No EXIF exposure value found for image DSC08087.JPG
No EXIF exposure value found for image DSC08088.JPG
No EXIF exposure value found for image DSC08089.JPG
No EXIF exposure value found for image DSC08090.JPG
No EXIF exposure value found for image DSC08091.JPG
No EXIF exposure value found for image DSC08092.JPG
No EXIF exposure value found for image DSC08093.JPG
No EXIF exposure value found for image DSC08094.JPG
No EXIF exposure value found for image DSC08095.JPG
No EXIF exposure value found for image DSC08096.JPG
No EXIF exposure value found for image DSC08097.JPG
No EXIF exposure value found for image DSC08098.JPG
No EXIF exposure value found for image DSC08099.JPG
No EXIF exposure value found for image DSC08100.JPG
No EXIF exposure value found for image DSC08101.JPG
No EXIF exposure value found for image DSC08102.JPG
No EXIF exposure value found for image DSC08103.JPG
No EXIF exposure value found for image DSC08104.JPG
No EXIF exposure value found for image DSC08105.JPG
No EXIF exposure value found for image DSC08106.JPG
No EXIF exposure value found for image DSC08107.JPG
No EXIF exposure value found for image DSC08108.JPG
No EXIF exposure value found for image DSC08109.JPG
No EXIF exposure value found for image DSC08110.JPG
No EXIF exposure value found for image DSC08111.JPG
No EXIF exposure value found for image DSC08112.JPG
No EXIF exposure value found for image DSC08113.JPG
No EXIF exposure value found for image DSC08114.JPG
No EXIF exposure value found for image DSC08115.JPG
No EXIF exposure value found for image DSC08116.JPG
No EXIF exposure value found for image DSC08117.JPG
No EXIF exposure value found for image DSC08118.JPG
No EXIF exposure value found for image DSC08119.JPG
No EXIF exposure value found for image DSC08120.JPG
No EXIF exposure value found for image DSC08121.JPG
No EXIF exposure value found for image DSC08122.JPG
No EXIF exposure value found for image DSC08123.JPG
No EXIF exposure value found for image DSC08124.JPG
No EXIF exposure value found for image DSC08125.JPG
No EXIF exposure value found for image DSC08126.JPG
No EXIF exposure value found for image DSC08127.JPG
No EXIF exposure value found for image DSC08128.JPG
No EXIF exposure value found for image DSC08129.JPG
No EXIF exposure value found for image DSC08130.JPG
No EXIF exposure value found for image DSC08131.JPG
No EXIF exposure value found for image DSC08132.JPG
No EXIF exposure value found for image DSC08133.JPG
No EXIF exposure value found for image DSC08134.JPG
No EXIF exposure value found for image DSC08135.JPG
No EXIF exposure value found for image DSC08136.JPG
No EXIF exposure value found for image DSC08137.JPG
No EXIF exposure value found for image DSC08138.JPG
No EXIF exposure value found for image DSC08139.JPG
No EXIF exposure value found for image DSC08140.JPG
EV Statistic:
Num         = [185]
Min,Max     = [0,0]
Mean,Median,Rms = [0,0,0]
sdev,var    = [0,0]
dynamic range: 1
  Image Size 5187x3361
  Aspect     1.54329
  K          3844.9 3852.36 2593.5 1680.5 0
  ocam       5187x3361 affine(1, 0, 0, 0, 0) cam2world() world2cam()
  ocam cut   1
  normalized center 0 0
  dist       0 0 0 0 0 0 0 0
====================================
Scene Loaded
  Name       garden_dense
  Path       D:\NERF\TRIPS\scenes\garden_dense
  Image Size 5187x3361
  Aspect     1.54329
  K          3844.9 3852.36 2593.5 1680.5 0
  ocam       5187x3361 affine(1, 0, 0, 0, 0) cam2world() world2cam()
  ocam cut   1
  normalized center 0 0
  dist       0 0 0 0 0 0 0 0
CAM model: CameraModel::PINHOLE_DISTORTION
>> Loading initial PLY point cloud and preprocessing it (done only once)
>> This can take a while...
Node  Trans: 0 0 0 1
Bounding Box point cloud:
AABB: [-77.568 -36.0105 -81.9788 ] [64.2399 7.03698 63.3852 ]
Starting Build KDTree... Done in 8510.88ms.
Starting RemoveLonelyPoints... Done in 8716.28ms.
remove lonely dis 0.02 Points 7877153 -> 7877153
Starting Build KDTree... Done in 8507.71ms.
Starting RemovePointsInCloseArea... Done in 7036.05ms.
remove close z dis 0.0001 Points 7877153 -> 7877153
Starting Build KDTree... Done in 8424.85ms.
Starting Compute radius... Done in 4239.98ms.
average_dis = 0.012298
Starting Computing random range sample... Done in 114.673ms.
Save point cloud D:\NERF\TRIPS\scenes\garden_dense/point_cloud.bin
  No normals: fill with data
  Points     7877153
  Colors     1
  Normals    1
  Num Images 185
  Num Cameras 1
Compute scene importance bounding box as 95% of points interval around center of mass
Starting Compute center of mass...center of mass:0.308283
2.15885
1.12378
 Done in 73.8001ms.
Starting Build range vec... Done in 48.6036ms.
Starting Sort range vec... Done in 898.124ms.
Starting Extend box... Done in 160.954ms.
Box: AABB: [-12.1599 -5.48379 -11.3698 ] [12.8644 6.71304 12.093 ]
====================================
Starting SceneData::Save... Done in 19897.9ms.
lfranke commented 7 months ago

Hi,

from your outputs, the only thing that seems wierd to me are your torch and cuda versions. Running train on my machine, the versions look like this:

PyTorch version: 1.13.1
The cuDNN version is 8302
cuDNN avail? 1
The CUDA runtime version is 11070

while you have PyTorch 2.1.0, cuDNN 8801 and CUDA runtime 12010.

Maybe that is the issue there?

HungNgoCT commented 6 months ago

Hi.,

I have a similar problem with training time. I used tt_playground, and train with default parameters: build\bin\RelWithDebInfo\train.exe --config configs\train_normalnet.ini

It took 18 hours for 330(/600) epoch, as shown in the image below.

image

My GPU: RTX 3090 Windows: 11 Cuda 11.8

Do you have any recommendations so that I can reduce training time for this data?

Thank you in advance