search

ashawkey / torch-ngp

A pytorch CUDA extension implementation of instant-ngp (sdf and nerf), with a GUI.
MIT License
2.09k stars 273 forks source link

psnr drop gradually #79

Open menglongyue opened 2 years ago

menglongyue commented 2 years ago

Thanks for your excellent work. I meet with a problem. As training progresses, PSNR decreases，from 20.82 to 20.00. how to solve it ?

ashawkey commented 2 years ago

@menglongyue Hi, could you provide more information? e.g., the dataset you use, the command line, the full output log.

menglongyue commented 2 years ago

Thanks for your reply. the training images is 3000x 2000, I set both bound and bg_radius to 16. the command line is:python main_nerf.py mydata --workspace trial_nerf -O --bound 16 --bg_radius 16，and use llff2nerf.py to get the json file, the aabb_scale not changed, default to 2. here is my training log: [INFO] Trainer: ngp | 2022-07-07_15-48-19 | cuda | fp16 | /hy-tmp/trail_nerf2 [INFO] #parameters: 15081712 [INFO] Loading latest checkpoint ... [WARN] No checkpoint found, model randomly initialized. ==> Start Training Epoch 1, lr=0.010000 ... ==> Finished Epoch 1. ==> Start Training Epoch 2, lr=0.009994 ... ==> Finished Epoch 2. ==> Start Training Epoch 3, lr=0.009988 ... ==> Finished Epoch 3. ==> Start Training Epoch 4, lr=0.009983 ... ==> Finished Epoch 4. ==> Start Training Epoch 5, lr=0.009977 ... ==> Finished Epoch 5. ==> Start Training Epoch 6, lr=0.009971 ... ==> Finished Epoch 6. ==> Start Training Epoch 7, lr=0.009966 ... ==> Finished Epoch 7. ==> Start Training Epoch 8, lr=0.009960 ... ==> Finished Epoch 8. ==> Start Training Epoch 9, lr=0.009954 ... ==> Finished Epoch 9. ==> Start Training Epoch 10, lr=0.009948 ... ==> Finished Epoch 10. ==> Start Training Epoch 11, lr=0.009943 ... ==> Finished Epoch 11. ==> Start Training Epoch 12, lr=0.009937 ... ==> Finished Epoch 12. ==> Start Training Epoch 13, lr=0.009931 ... ==> Finished Epoch 13. ==> Start Training Epoch 14, lr=0.009925 ... ==> Finished Epoch 14. ==> Start Training Epoch 15, lr=0.009920 ... ==> Finished Epoch 15. ==> Start Training Epoch 16, lr=0.009914 ... ==> Finished Epoch 16. ==> Start Training Epoch 17, lr=0.009908 ... ==> Finished Epoch 17. ==> Start Training Epoch 18, lr=0.009903 ... ==> Finished Epoch 18. ==> Start Training Epoch 19, lr=0.009897 ... ==> Finished Epoch 19. ==> Start Training Epoch 20, lr=0.009891 ... ==> Finished Epoch 20. ==> Start Training Epoch 21, lr=0.009886 ... ==> Finished Epoch 21. ==> Start Training Epoch 22, lr=0.009880 ... ==> Finished Epoch 22. ==> Start Training Epoch 23, lr=0.009874 ... ==> Finished Epoch 23. ==> Start Training Epoch 24, lr=0.009868 ... ==> Finished Epoch 24. ==> Start Training Epoch 25, lr=0.009863 ... ==> Finished Epoch 25. ==> Start Training Epoch 26, lr=0.009857 ... ==> Finished Epoch 26. ==> Start Training Epoch 27, lr=0.009851 ... ==> Finished Epoch 27. ==> Start Training Epoch 28, lr=0.009846 ... ==> Finished Epoch 28. ==> Start Training Epoch 29, lr=0.009840 ... ==> Finished Epoch 29. ==> Start Training Epoch 30, lr=0.009834 ... ==> Finished Epoch 30. ==> Start Training Epoch 31, lr=0.009829 ... ==> Finished Epoch 31. ==> Start Training Epoch 32, lr=0.009823 ... ==> Finished Epoch 32. ==> Start Training Epoch 33, lr=0.009817 ... ==> Finished Epoch 33. ==> Start Training Epoch 34, lr=0.009812 ... ==> Finished Epoch 34. ==> Start Training Epoch 35, lr=0.009806 ... ==> Finished Epoch 35. ==> Start Training Epoch 36, lr=0.009801 ... ==> Finished Epoch 36. ==> Start Training Epoch 37, lr=0.009795 ... ==> Finished Epoch 37. ==> Start Training Epoch 38, lr=0.009789 ... ==> Finished Epoch 38. ==> Start Training Epoch 39, lr=0.009784 ... ==> Finished Epoch 39. ==> Start Training Epoch 40, lr=0.009778 ... ==> Finished Epoch 40. ==> Start Training Epoch 41, lr=0.009772 ... ==> Finished Epoch 41. ==> Start Training Epoch 42, lr=0.009767 ... ==> Finished Epoch 42. ==> Start Training Epoch 43, lr=0.009761 ... ==> Finished Epoch 43. ==> Start Training Epoch 44, lr=0.009756 ... ==> Finished Epoch 44. ==> Start Training Epoch 45, lr=0.009750 ... ==> Finished Epoch 45. ==> Start Training Epoch 46, lr=0.009744 ... ==> Finished Epoch 46. ==> Start Training Epoch 47, lr=0.009739 ... ==> Finished Epoch 47. ==> Start Training Epoch 48, lr=0.009733 ... ==> Finished Epoch 48. ==> Start Training Epoch 49, lr=0.009727 ... ==> Finished Epoch 49. ==> Start Training Epoch 50, lr=0.009722 ... ==> Finished Epoch 50. ++> Evaluate at epoch 50 ... PSNR = 20.739291 ++> Evaluate epoch 50 Finished. [INFO] New best result: None --> 0.008512702537700534 ==> Start Training Epoch 51, lr=0.009716 ... ==> Finished Epoch 51. ==> Start Training Epoch 52, lr=0.009711 ... ==> Finished Epoch 52. ==> Start Training Epoch 53, lr=0.009705 ... ==> Finished Epoch 53. ==> Start Training Epoch 54, lr=0.009700 ... ==> Finished Epoch 54. ==> Start Training Epoch 55, lr=0.009694 ... ==> Finished Epoch 55. ==> Start Training Epoch 56, lr=0.009688 ... ==> Finished Epoch 56. ==> Start Training Epoch 57, lr=0.009683 ... ==> Finished Epoch 57. ==> Start Training Epoch 58, lr=0.009677 ... ==> Finished Epoch 58. ==> Start Training Epoch 59, lr=0.009672 ... ==> Finished Epoch 59. ==> Start Training Epoch 60, lr=0.009666 ... ==> Finished Epoch 60. ==> Start Training Epoch 61, lr=0.009661 ... ==> Finished Epoch 61. ==> Start Training Epoch 62, lr=0.009655 ... ==> Finished Epoch 62. ==> Start Training Epoch 63, lr=0.009649 ... ==> Finished Epoch 63. ==> Start Training Epoch 64, lr=0.009644 ... ==> Finished Epoch 64. ==> Start Training Epoch 65, lr=0.009638 ... ==> Finished Epoch 65. ==> Start Training Epoch 66, lr=0.009633 ... ==> Finished Epoch 66. ==> Start Training Epoch 67, lr=0.009627 ... ==> Finished Epoch 67. ==> Start Training Epoch 68, lr=0.009622 ... ==> Finished Epoch 68. ==> Start Training Epoch 69, lr=0.009616 ... ==> Finished Epoch 69. ==> Start Training Epoch 70, lr=0.009611 ... ==> Finished Epoch 70. ==> Start Training Epoch 71, lr=0.009605 ... ==> Finished Epoch 71. ==> Start Training Epoch 72, lr=0.009600 ... ==> Finished Epoch 72. ==> Start Training Epoch 73, lr=0.009594 ... ==> Finished Epoch 73. ==> Start Training Epoch 74, lr=0.009588 ... ==> Finished Epoch 74. ==> Start Training Epoch 75, lr=0.009583 ... ==> Finished Epoch 75. ==> Start Training Epoch 76, lr=0.009577 ... ==> Finished Epoch 76. ==> Start Training Epoch 77, lr=0.009572 ... ==> Finished Epoch 77. ==> Start Training Epoch 78, lr=0.009566 ... ==> Finished Epoch 78. ==> Start Training Epoch 79, lr=0.009561 ... ==> Finished Epoch 79. ==> Start Training Epoch 80, lr=0.009555 ... ==> Finished Epoch 80. ==> Start Training Epoch 81, lr=0.009550 ... ==> Finished Epoch 81. ==> Start Training Epoch 82, lr=0.009544 ... ==> Finished Epoch 82. ==> Start Training Epoch 83, lr=0.009539 ... ==> Finished Epoch 83. ==> Start Training Epoch 84, lr=0.009533 ... ==> Finished Epoch 84. ==> Start Training Epoch 85, lr=0.009528 ... ==> Finished Epoch 85. ==> Start Training Epoch 86, lr=0.009522 ... ==> Finished Epoch 86. ==> Start Training Epoch 87, lr=0.009517 ... ==> Finished Epoch 87. ==> Start Training Epoch 88, lr=0.009512 ... ==> Finished Epoch 88. ==> Start Training Epoch 89, lr=0.009506 ... ==> Finished Epoch 89. ==> Start Training Epoch 90, lr=0.009501 ... ==> Finished Epoch 90. ==> Start Training Epoch 91, lr=0.009495 ... ==> Finished Epoch 91. ==> Start Training Epoch 92, lr=0.009490 ... ==> Finished Epoch 92. ==> Start Training Epoch 93, lr=0.009484 ... ==> Finished Epoch 93. ==> Start Training Epoch 94, lr=0.009479 ... ==> Finished Epoch 94. ==> Start Training Epoch 95, lr=0.009473 ... ==> Finished Epoch 95. ==> Start Training Epoch 96, lr=0.009468 ... ==> Finished Epoch 96. ==> Start Training Epoch 97, lr=0.009462 ... ==> Finished Epoch 97. ==> Start Training Epoch 98, lr=0.009457 ... ==> Finished Epoch 98. ==> Start Training Epoch 99, lr=0.009451 ... ==> Finished Epoch 99. ==> Start Training Epoch 100, lr=0.009446 ... ==> Finished Epoch 100. ++> Evaluate at epoch 100 ... PSNR = 20.808737 ++> Evaluate epoch 100 Finished. [INFO] New best result: 0.008512702537700534 --> 0.008385548368096352 ==> Start Training Epoch 101, lr=0.009441 ... ==> Finished Epoch 101. ==> Start Training Epoch 102, lr=0.009435 ... ==> Finished Epoch 102. ==> Start Training Epoch 103, lr=0.009430 ... ==> Finished Epoch 103. ==> Start Training Epoch 104, lr=0.009424 ... ==> Finished Epoch 104. ==> Start Training Epoch 105, lr=0.009419 ... ==> Finished Epoch 105. ==> Start Training Epoch 106, lr=0.009413 ... ==> Finished Epoch 106. ==> Start Training Epoch 107, lr=0.009408 ... ==> Finished Epoch 107. ==> Start Training Epoch 108, lr=0.009403 ... ==> Finished Epoch 108. ==> Start Training Epoch 109, lr=0.009397 ... ==> Finished Epoch 109. ==> Start Training Epoch 110, lr=0.009392 ... ==> Finished Epoch 110. ==> Start Training Epoch 111, lr=0.009386 ... ==> Finished Epoch 111. ==> Start Training Epoch 112, lr=0.009381 ... ==> Finished Epoch 112. ==> Start Training Epoch 113, lr=0.009376 ... ==> Finished Epoch 113. ==> Start Training Epoch 114, lr=0.009370 ... ==> Finished Epoch 114. ==> Start Training Epoch 115, lr=0.009365 ... ==> Finished Epoch 115. ==> Start Training Epoch 116, lr=0.009359 ... ==> Finished Epoch 116. ==> Start Training Epoch 117, lr=0.009354 ... ==> Finished Epoch 117. ==> Start Training Epoch 118, lr=0.009349 ... ==> Finished Epoch 118. ==> Start Training Epoch 119, lr=0.009343 ... ==> Finished Epoch 119. ==> Start Training Epoch 120, lr=0.009338 ... ==> Finished Epoch 120. ==> Start Training Epoch 121, lr=0.009333 ... ==> Finished Epoch 121. ==> Start Training Epoch 122, lr=0.009327 ... ==> Finished Epoch 122. ==> Start Training Epoch 123, lr=0.009322 ... ==> Finished Epoch 123. ==> Start Training Epoch 124, lr=0.009316 ... ==> Finished Epoch 124. ==> Start Training Epoch 125, lr=0.009311 ... ==> Finished Epoch 125. ==> Start Training Epoch 126, lr=0.009306 ... ==> Finished Epoch 126. ==> Start Training Epoch 127, lr=0.009300 ... ==> Finished Epoch 127. ==> Start Training Epoch 128, lr=0.009295 ... ==> Finished Epoch 128. ==> Start Training Epoch 129, lr=0.009290 ... ==> Finished Epoch 129. ==> Start Training Epoch 130, lr=0.009284 ... ==> Finished Epoch 130. ==> Start Training Epoch 131, lr=0.009279 ... ==> Finished Epoch 131. ==> Start Training Epoch 132, lr=0.009274 ... ==> Finished Epoch 132. ==> Start Training Epoch 133, lr=0.009268 ... ==> Finished Epoch 133. ==> Start Training Epoch 134, lr=0.009263 ... ==> Finished Epoch 134. ==> Start Training Epoch 135, lr=0.009258 ... ==> Finished Epoch 135. ==> Start Training Epoch 136, lr=0.009252 ... ==> Finished Epoch 136. ==> Start Training Epoch 137, lr=0.009247 ... ==> Finished Epoch 137. ==> Start Training Epoch 138, lr=0.009242 ... ==> Finished Epoch 138. ==> Start Training Epoch 139, lr=0.009236 ... ==> Finished Epoch 139. ==> Start Training Epoch 140, lr=0.009231 ... ==> Finished Epoch 140. ==> Start Training Epoch 141, lr=0.009226 ... ==> Finished Epoch 141. ==> Start Training Epoch 142, lr=0.009220 ... ==> Finished Epoch 142. ==> Start Training Epoch 143, lr=0.009215 ... ==> Finished Epoch 143. ==> Start Training Epoch 144, lr=0.009210 ... ==> Finished Epoch 144. ==> Start Training Epoch 145, lr=0.009204 ... ==> Finished Epoch 145. ==> Start Training Epoch 146, lr=0.009199 ... ==> Finished Epoch 146. ==> Start Training Epoch 147, lr=0.009194 ... ==> Finished Epoch 147. ==> Start Training Epoch 148, lr=0.009189 ... ==> Finished Epoch 148. ==> Start Training Epoch 149, lr=0.009183 ... ==> Finished Epoch 149. ==> Start Training Epoch 150, lr=0.009178 ... ==> Finished Epoch 150. ++> Evaluate at epoch 150 ... PSNR = 20.734601 ++> Evaluate epoch 150 Finished. ==> Start Training Epoch 151, lr=0.009173 ... ==> Finished Epoch 151. ==> Start Training Epoch 152, lr=0.009167 ... ==> Finished Epoch 152. ==> Start Training Epoch 153, lr=0.009162 ... ==> Finished Epoch 153. ==> Start Training Epoch 154, lr=0.009157 ... ==> Finished Epoch 154. ==> Start Training Epoch 155, lr=0.009152 ... ==> Finished Epoch 155. ==> Start Training Epoch 156, lr=0.009146 ... ==> Finished Epoch 156. ==> Start Training Epoch 157, lr=0.009141 ... ==> Finished Epoch 157. ==> Start Training Epoch 158, lr=0.009136 ... ==> Finished Epoch 158. ==> Start Training Epoch 159, lr=0.009131 ... ==> Finished Epoch 159. ==> Start Training Epoch 160, lr=0.009125 ... ==> Finished Epoch 160. ==> Start Training Epoch 161, lr=0.009120 ... ==> Finished Epoch 161. ==> Start Training Epoch 162, lr=0.009115 ... ==> Finished Epoch 162. ==> Start Training Epoch 163, lr=0.009110 ... ==> Finished Epoch 163. ==> Start Training Epoch 164, lr=0.009104 ... ==> Finished Epoch 164. ==> Start Training Epoch 165, lr=0.009099 ... ==> Finished Epoch 165. ==> Start Training Epoch 166, lr=0.009094 ... ==> Finished Epoch 166. ==> Start Training Epoch 167, lr=0.009089 ... ==> Finished Epoch 167. ==> Start Training Epoch 168, lr=0.009083 ... ==> Finished Epoch 168. ==> Start Training Epoch 169, lr=0.009078 ... ==> Finished Epoch 169. ==> Start Training Epoch 170, lr=0.009073 ... ==> Finished Epoch 170. ==> Start Training Epoch 171, lr=0.009068 ... ==> Finished Epoch 171. ==> Start Training Epoch 172, lr=0.009063 ... ==> Finished Epoch 172. ==> Start Training Epoch 173, lr=0.009057 ... ==> Finished Epoch 173. ==> Start Training Epoch 174, lr=0.009052 ... ==> Finished Epoch 174. ==> Start Training Epoch 175, lr=0.009047 ... ==> Finished Epoch 175. ==> Start Training Epoch 176, lr=0.009042 ... ==> Finished Epoch 176. ==> Start Training Epoch 177, lr=0.009036 ... ==> Finished Epoch 177. ==> Start Training Epoch 178, lr=0.009031 ... ==> Finished Epoch 178. ==> Start Training Epoch 179, lr=0.009026 ... ==> Finished Epoch 179. ==> Start Training Epoch 180, lr=0.009021 ... ==> Finished Epoch 180. ==> Start Training Epoch 181, lr=0.009016 ... ==> Finished Epoch 181. ==> Start Training Epoch 182, lr=0.009011 ... ==> Finished Epoch 182. ==> Start Training Epoch 183, lr=0.009005 ... ==> Finished Epoch 183. ==> Start Training Epoch 184, lr=0.009000 ... ==> Finished Epoch 184. ==> Start Training Epoch 185, lr=0.008995 ... ==> Finished Epoch 185. ==> Start Training Epoch 186, lr=0.008990 ... ==> Finished Epoch 186. ==> Start Training Epoch 187, lr=0.008985 ... ==> Finished Epoch 187. ==> Start Training Epoch 188, lr=0.008979 ... ==> Finished Epoch 188. ==> Start Training Epoch 189, lr=0.008974 ... ==> Finished Epoch 189. ==> Start Training Epoch 190, lr=0.008969 ... ==> Finished Epoch 190. ==> Start Training Epoch 191, lr=0.008964 ... ==> Finished Epoch 191. ==> Start Training Epoch 192, lr=0.008959 ... ==> Finished Epoch 192. ==> Start Training Epoch 193, lr=0.008954 ... ==> Finished Epoch 193. ==> Start Training Epoch 194, lr=0.008948 ... ==> Finished Epoch 194. ==> Start Training Epoch 195, lr=0.008943 ... ==> Finished Epoch 195. ==> Start Training Epoch 196, lr=0.008938 ... ==> Finished Epoch 196. ==> Start Training Epoch 197, lr=0.008933 ... ==> Finished Epoch 197. ==> Start Training Epoch 198, lr=0.008928 ... ==> Finished Epoch 198. ==> Start Training Epoch 199, lr=0.008923 ... ==> Finished Epoch 199. ==> Start Training Epoch 200, lr=0.008918 ... ==> Finished Epoch 200. ++> Evaluate at epoch 200 ... PSNR = 20.714772 ++> Evaluate epoch 200 Finished. ==> Start Training Epoch 201, lr=0.008913 ... ==> Finished Epoch 201. ==> Start Training Epoch 202, lr=0.008907 ... ==> Finished Epoch 202. ==> Start Training Epoch 203, lr=0.008902 ... ==> Finished Epoch 203. ==> Start Training Epoch 204, lr=0.008897 ... ==> Finished Epoch 204. ==> Start Training Epoch 205, lr=0.008892 ... ==> Finished Epoch 205. ==> Start Training Epoch 206, lr=0.008887 ... ==> Finished Epoch 206. ==> Start Training Epoch 207, lr=0.008882 ... ==> Finished Epoch 207. ==> Start Training Epoch 208, lr=0.008877 ... ==> Finished Epoch 208. ==> Start Training Epoch 209, lr=0.008872 ... ==> Finished Epoch 209. ==> Start Training Epoch 210, lr=0.008866 ... ==> Finished Epoch 210. ==> Start Training Epoch 211, lr=0.008861 ... ==> Finished Epoch 211. ==> Start Training Epoch 212, lr=0.008856 ... ==> Finished Epoch 212. ==> Start Training Epoch 213, lr=0.008851 ... ==> Finished Epoch 213. ==> Start Training Epoch 214, lr=0.008846 ... ==> Finished Epoch 214. ==> Start Training Epoch 215, lr=0.008841 ... ==> Finished Epoch 215. ==> Start Training Epoch 216, lr=0.008836 ... ==> Finished Epoch 216. ==> Start Training Epoch 217, lr=0.008831 ... ==> Finished Epoch 217. ==> Start Training Epoch 218, lr=0.008826 ... ==> Finished Epoch 218. ==> Start Training Epoch 219, lr=0.008821 ... ==> Finished Epoch 219. ==> Start Training Epoch 220, lr=0.008816 ... ==> Finished Epoch 220. ==> Start Training Epoch 221, lr=0.008810 ... ==> Finished Epoch 221. ==> Start Training Epoch 222, lr=0.008805 ... ==> Finished Epoch 222. ==> Start Training Epoch 223, lr=0.008800 ... ==> Finished Epoch 223. ==> Start Training Epoch 224, lr=0.008795 ... ==> Finished Epoch 224. ==> Start Training Epoch 225, lr=0.008790 ... ==> Finished Epoch 225. ==> Start Training Epoch 226, lr=0.008785 ... ==> Finished Epoch 226. ==> Start Training Epoch 227, lr=0.008780 ... ==> Finished Epoch 227. ==> Start Training Epoch 228, lr=0.008775 ... ==> Finished Epoch 228. ==> Start Training Epoch 229, lr=0.008770 ... ==> Finished Epoch 229. ==> Start Training Epoch 230, lr=0.008765 ... ==> Finished Epoch 230. ==> Start Training Epoch 231, lr=0.008760 ... ==> Finished Epoch 231. ==> Start Training Epoch 232, lr=0.008755 ... ==> Finished Epoch 232. ==> Start Training Epoch 233, lr=0.008750 ... ==> Finished Epoch 233. ==> Start Training Epoch 234, lr=0.008745 ... ==> Finished Epoch 234. ==> Start Training Epoch 235, lr=0.008740 ... ==> Finished Epoch 235. ==> Start Training Epoch 236, lr=0.008735 ... ==> Finished Epoch 236. ==> Start Training Epoch 237, lr=0.008730 ... ==> Finished Epoch 237. ==> Start Training Epoch 238, lr=0.008725 ... ==> Finished Epoch 238. ==> Start Training Epoch 239, lr=0.008720 ... ==> Finished Epoch 239. ==> Start Training Epoch 240, lr=0.008715 ... ==> Finished Epoch 240. ==> Start Training Epoch 241, lr=0.008710 ... ==> Finished Epoch 241. ==> Start Training Epoch 242, lr=0.008705 ... ==> Finished Epoch 242. ==> Start Training Epoch 243, lr=0.008700 ... ==> Finished Epoch 243. ==> Start Training Epoch 244, lr=0.008695 ... ==> Finished Epoch 244. ==> Start Training Epoch 245, lr=0.008690 ... ==> Finished Epoch 245. ==> Start Training Epoch 246, lr=0.008685 ... ==> Finished Epoch 246. ==> Start Training Epoch 247, lr=0.008680 ... ==> Finished Epoch 247. ==> Start Training Epoch 248, lr=0.008675 ... ==> Finished Epoch 248. ==> Start Training Epoch 249, lr=0.008670 ... ==> Finished Epoch 249. ==> Start Training Epoch 250, lr=0.008665 ... ==> Finished Epoch 250. ++> Evaluate at epoch 250 ... PSNR = 20.682181 ++> Evaluate epoch 250 Finished. ==> Start Training Epoch 251, lr=0.008660 ... ==> Finished Epoch 251. ==> Start Training Epoch 252, lr=0.008655 ... ==> Finished Epoch 252. ==> Start Training Epoch 253, lr=0.008650 ... ==> Finished Epoch 253. ==> Start Training Epoch 254, lr=0.008645 ... ==> Finished Epoch 254. ==> Start Training Epoch 255, lr=0.008640 ... ==> Finished Epoch 255. ==> Start Training Epoch 256, lr=0.008635 ... ==> Finished Epoch 256. ==> Start Training Epoch 257, lr=0.008630 ... ==> Finished Epoch 257. ==> Start Training Epoch 258, lr=0.008625 ... ==> Finished Epoch 258. ==> Start Training Epoch 259, lr=0.008620 ... ==> Finished Epoch 259. ==> Start Training Epoch 260, lr=0.008615 ... ==> Finished Epoch 260. ==> Start Training Epoch 261, lr=0.008610 ... ==> Finished Epoch 261. ==> Start Training Epoch 262, lr=0.008605 ... ==> Finished Epoch 262. ==> Start Training Epoch 263, lr=0.008600 ... ==> Finished Epoch 263. ==> Start Training Epoch 264, lr=0.008595 ... ==> Finished Epoch 264. ==> Start Training Epoch 265, lr=0.008590 ... ==> Finished Epoch 265. ==> Start Training Epoch 266, lr=0.008585 ... ==> Finished Epoch 266. ==> Start Training Epoch 267, lr=0.008580 ... ==> Finished Epoch 267. ==> Start Training Epoch 268, lr=0.008575 ... ==> Finished Epoch 268. ==> Start Training Epoch 269, lr=0.008570 ... ==> Finished Epoch 269. ==> Start Training Epoch 270, lr=0.008565 ... ==> Finished Epoch 270. ==> Start Training Epoch 271, lr=0.008561 ... ==> Finished Epoch 271. ==> Start Training Epoch 272, lr=0.008556 ... ==> Finished Epoch 272. ==> Start Training Epoch 273, lr=0.008551 ... ==> Finished Epoch 273. ==> Start Training Epoch 274, lr=0.008546 ... ==> Finished Epoch 274. ==> Start Training Epoch 275, lr=0.008541 ... ==> Finished Epoch 275. ==> Start Training Epoch 276, lr=0.008536 ... ==> Finished Epoch 276. ==> Start Training Epoch 277, lr=0.008531 ... ==> Finished Epoch 277. ==> Start Training Epoch 278, lr=0.008526 ... ==> Finished Epoch 278. ==> Start Training Epoch 279, lr=0.008521 ... ==> Finished Epoch 279. ==> Start Training Epoch 280, lr=0.008516 ... ==> Finished Epoch 280. ==> Start Training Epoch 281, lr=0.008511 ... ==> Finished Epoch 281. ==> Start Training Epoch 282, lr=0.008506 ... ==> Finished Epoch 282. ==> Start Training Epoch 283, lr=0.008502 ... ==> Finished Epoch 283. ==> Start Training Epoch 284, lr=0.008497 ... ==> Finished Epoch 284. ==> Start Training Epoch 285, lr=0.008492 ... ==> Finished Epoch 285. ==> Start Training Epoch 286, lr=0.008487 ... ==> Finished Epoch 286. ==> Start Training Epoch 287, lr=0.008482 ... ==> Finished Epoch 287. ==> Start Training Epoch 288, lr=0.008477 ... ==> Finished Epoch 288. ==> Start Training Epoch 289, lr=0.008472 ... ==> Finished Epoch 289. ==> Start Training Epoch 290, lr=0.008467 ... ==> Finished Epoch 290. ==> Start Training Epoch 291, lr=0.008463 ... ==> Finished Epoch 291. ==> Start Training Epoch 292, lr=0.008458 ... ==> Finished Epoch 292. ==> Start Training Epoch 293, lr=0.008453 ... ==> Finished Epoch 293. ==> Start Training Epoch 294, lr=0.008448 ... ==> Finished Epoch 294. ==> Start Training Epoch 295, lr=0.008443 ... ==> Finished Epoch 295. ==> Start Training Epoch 296, lr=0.008438 ... ==> Finished Epoch 296. ==> Start Training Epoch 297, lr=0.008433 ... ==> Finished Epoch 297. ==> Start Training Epoch 298, lr=0.008428 ... ==> Finished Epoch 298. ==> Start Training Epoch 299, lr=0.008424 ... ==> Finished Epoch 299. ==> Start Training Epoch 300, lr=0.008419 ... ==> Finished Epoch 300. ++> Evaluate at epoch 300 ... PSNR = 20.659329 ++> Evaluate epoch 300 Finished. ==> Start Training Epoch 301, lr=0.008414 ... ==> Finished Epoch 301. ==> Start Training Epoch 302, lr=0.008409 ... ==> Finished Epoch 302. ==> Start Training Epoch 303, lr=0.008404 ... ==> Finished Epoch 303. ==> Start Training Epoch 304, lr=0.008399 ... ==> Finished Epoch 304. ==> Start Training Epoch 305, lr=0.008395 ... ==> Finished Epoch 305. ==> Start Training Epoch 306, lr=0.008390 ... ==> Finished Epoch 306. ==> Start Training Epoch 307, lr=0.008385 ... ==> Finished Epoch 307. ==> Start Training Epoch 308, lr=0.008380 ... ==> Finished Epoch 308. ==> Start Training Epoch 309, lr=0.008375 ... ==> Finished Epoch 309. ==> Start Training Epoch 310, lr=0.008370 ... ==> Finished Epoch 310. ==> Start Training Epoch 311, lr=0.008366 ... ==> Finished Epoch 311. ==> Start Training Epoch 312, lr=0.008361 ... ==> Finished Epoch 312. ==> Start Training Epoch 313, lr=0.008356 ... ==> Finished Epoch 313. ==> Start Training Epoch 314, lr=0.008351 ... ==> Finished Epoch 314. ==> Start Training Epoch 315, lr=0.008346 ... ==> Finished Epoch 315. ==> Start Training Epoch 316, lr=0.008342 ... ==> Finished Epoch 316. ==> Start Training Epoch 317, lr=0.008337 ... ==> Finished Epoch 317. ==> Start Training Epoch 318, lr=0.008332 ... ==> Finished Epoch 318. ==> Start Training Epoch 319, lr=0.008327 ... ==> Finished Epoch 319. ==> Start Training Epoch 320, lr=0.008322 ... ==> Finished Epoch 320. ==> Start Training Epoch 321, lr=0.008318 ... ==> Finished Epoch 321. ==> Start Training Epoch 322, lr=0.008313 ... ==> Finished Epoch 322. ==> Start Training Epoch 323, lr=0.008308 ... ==> Finished Epoch 323. ==> Start Training Epoch 324, lr=0.008303 ... ==> Finished Epoch 324. ==> Start Training Epoch 325, lr=0.008299 ... ==> Finished Epoch 325. ==> Start Training Epoch 326, lr=0.008294 ... ==> Finished Epoch 326. ==> Start Training Epoch 327, lr=0.008289 ... ==> Finished Epoch 327. ==> Start Training Epoch 328, lr=0.008284 ... ==> Finished Epoch 328. ==> Start Training Epoch 329, lr=0.008279 ... ==> Finished Epoch 329. ==> Start Training Epoch 330, lr=0.008275 ... ==> Finished Epoch 330. ==> Start Training Epoch 331, lr=0.008270 ... ==> Finished Epoch 331. ==> Start Training Epoch 332, lr=0.008265 ... ==> Finished Epoch 332. ==> Start Training Epoch 333, lr=0.008260 ... ==> Finished Epoch 333. ==> Start Training Epoch 334, lr=0.008256 ... ==> Finished Epoch 334. ==> Start Training Epoch 335, lr=0.008251 ... ==> Finished Epoch 335. ==> Start Training Epoch 336, lr=0.008246 ... ==> Finished Epoch 336. ==> Start Training Epoch 337, lr=0.008241 ... ==> Finished Epoch 337. ==> Start Training Epoch 338, lr=0.008237 ... ==> Finished Epoch 338. ==> Start Training Epoch 339, lr=0.008232 ... ==> Finished Epoch 339. ==> Start Training Epoch 340, lr=0.008227 ... ==> Finished Epoch 340. ==> Start Training Epoch 341, lr=0.008222 ... ==> Finished Epoch 341. ==> Start Training Epoch 342, lr=0.008218 ... ==> Finished Epoch 342. ==> Start Training Epoch 343, lr=0.008213 ... ==> Finished Epoch 343. ==> Start Training Epoch 344, lr=0.008208 ... ==> Finished Epoch 344. ==> Start Training Epoch 345, lr=0.008204 ... ==> Finished Epoch 345. ==> Start Training Epoch 346, lr=0.008199 ... ==> Finished Epoch 346. ==> Start Training Epoch 347, lr=0.008194 ... ==> Finished Epoch 347. ==> Start Training Epoch 348, lr=0.008189 ... ==> Finished Epoch 348. ==> Start Training Epoch 349, lr=0.008185 ... ==> Finished Epoch 349. ==> Start Training Epoch 350, lr=0.008180 ... ==> Finished Epoch 350. ++> Evaluate at epoch 350 ... PSNR = 20.634949 ++> Evaluate epoch 350 Finished. ==> Start Training Epoch 351, lr=0.008175 ... ==> Finished Epoch 351. ==> Start Training Epoch 352, lr=0.008171 ... ==> Finished Epoch 352. ==> Start Training Epoch 353, lr=0.008166 ... ==> Finished Epoch 353. ==> Start Training Epoch 354, lr=0.008161 ... ==> Finished Epoch 354. ==> Start Training Epoch 355, lr=0.008156 ... ==> Finished Epoch 355. ==> Start Training Epoch 356, lr=0.008152 ... ==> Finished Epoch 356. ==> Start Training Epoch 357, lr=0.008147 ... ==> Finished Epoch 357. ==> Start Training Epoch 358, lr=0.008142 ... ==> Finished Epoch 358. ==> Start Training Epoch 359, lr=0.008138 ... ==> Finished Epoch 359. ==> Start Training Epoch 360, lr=0.008133 ... ==> Finished Epoch 360. ==> Start Training Epoch 361, lr=0.008128 ... ==> Finished Epoch 361. ==> Start Training Epoch 362, lr=0.008124 ... ==> Finished Epoch 362. ==> Start Training Epoch 363, lr=0.008119 ... ==> Finished Epoch 363. ==> Start Training Epoch 364, lr=0.008114 ... ==> Finished Epoch 364. ==> Start Training Epoch 365, lr=0.008110 ... ==> Finished Epoch 365. ==> Start Training Epoch 366, lr=0.008105 ... ==> Finished Epoch 366. ==> Start Training Epoch 367, lr=0.008100 ... ==> Finished Epoch 367. ==> Start Training Epoch 368, lr=0.008096 ... ==> Finished Epoch 368. ==> Start Training Epoch 369, lr=0.008091 ... ==> Finished Epoch 369. ==> Start Training Epoch 370, lr=0.008086 ... ==> Finished Epoch 370. ==> Start Training Epoch 371, lr=0.008082 ... ==> Finished Epoch 371. ==> Start Training Epoch 372, lr=0.008077 ... ==> Finished Epoch 372. ==> Start Training Epoch 373, lr=0.008072 ... ==> Finished Epoch 373. ==> Start Training Epoch 374, lr=0.008068 ... ==> Finished Epoch 374. ==> Start Training Epoch 375, lr=0.008063 ... ==> Finished Epoch 375. ==> Start Training Epoch 376, lr=0.008058 ... ==> Finished Epoch 376. ==> Start Training Epoch 377, lr=0.008054 ... ==> Finished Epoch 377. ==> Start Training Epoch 378, lr=0.008049 ... ==> Finished Epoch 378. ==> Start Training Epoch 379, lr=0.008045 ... ==> Finished Epoch 379. ==> Start Training Epoch 380, lr=0.008040 ... ==> Finished Epoch 380. ==> Start Training Epoch 381, lr=0.008035 ... ==> Finished Epoch 381. ==> Start Training Epoch 382, lr=0.008031 ... ==> Finished Epoch 382. ==> Start Training Epoch 383, lr=0.008026 ... ==> Finished Epoch 383. ==> Start Training Epoch 384, lr=0.008021 ... ==> Finished Epoch 384. ==> Start Training Epoch 385, lr=0.008017 ... ==> Finished Epoch 385. ==> Start Training Epoch 386, lr=0.008012 ... ==> Finished Epoch 386. ==> Start Training Epoch 387, lr=0.008008 ... ==> Finished Epoch 387. ==> Start Training Epoch 388, lr=0.008003 ... ==> Finished Epoch 388. ==> Start Training Epoch 389, lr=0.007998 ... ==> Finished Epoch 389. ==> Start Training Epoch 390, lr=0.007994 ... ==> Finished Epoch 390. ==> Start Training Epoch 391, lr=0.007989 ... ==> Finished Epoch 391. ==> Start Training Epoch 392, lr=0.007985 ... ==> Finished Epoch 392. ==> Start Training Epoch 393, lr=0.007980 ... ==> Finished Epoch 393. ==> Start Training Epoch 394, lr=0.007975 ... ==> Finished Epoch 394. ==> Start Training Epoch 395, lr=0.007971 ... ==> Finished Epoch 395. ==> Start Training Epoch 396, lr=0.007966 ... ==> Finished Epoch 396. ==> Start Training Epoch 397, lr=0.007962 ... ==> Finished Epoch 397. ==> Start Training Epoch 398, lr=0.007957 ... ==> Finished Epoch 398. ==> Start Training Epoch 399, lr=0.007952 ... ==> Finished Epoch 399. ==> Start Training Epoch 400, lr=0.007948 ... ==> Finished Epoch 400. ++> Evaluate at epoch 400 ... PSNR = 20.621731 ++> Evaluate epoch 400 Finished. ==> Start Training Epoch 401, lr=0.007943 ... ==> Finished Epoch 401. ==> Start Training Epoch 402, lr=0.007939 ... ==> Finished Epoch 402. ==> Start Training Epoch 403, lr=0.007934 ... ==> Finished Epoch 403. ==> Start Training Epoch 404, lr=0.007930 ... ==> Finished Epoch 404. ==> Start Training Epoch 405, lr=0.007925 ... ==> Finished Epoch 405. ==> Start Training Epoch 406, lr=0.007920 ... ==> Finished Epoch 406. ==> Start Training Epoch 407, lr=0.007916 ... ==> Finished Epoch 407. ==> Start Training Epoch 408, lr=0.007911 ... ==> Finished Epoch 408. ==> Start Training Epoch 409, lr=0.007907 ... ==> Finished Epoch 409. ==> Start Training Epoch 410, lr=0.007902 ... ==> Finished Epoch 410. ==> Start Training Epoch 411, lr=0.007898 ... ==> Finished Epoch 411. ==> Start Training Epoch 412, lr=0.007893 ... ==> Finished Epoch 412. ==> Start Training Epoch 413, lr=0.007889 ... ==> Finished Epoch 413. ==> Start Training Epoch 414, lr=0.007884 ... ==> Finished Epoch 414. ==> Start Training Epoch 415, lr=0.007880 ... ==> Finished Epoch 415. ==> Start Training Epoch 416, lr=0.007875 ... ==> Finished Epoch 416. ==> Start Training Epoch 417, lr=0.007870 ... ==> Finished Epoch 417. ==> Start Training Epoch 418, lr=0.007866 ... ==> Finished Epoch 418. ==> Start Training Epoch 419, lr=0.007861 ... ==> Finished Epoch 419.

ashawkey commented 2 years ago

Myabe you could try to use a much larger --bg_radius like 64 or 128? It is basically an (experimental) approximation of the infinite-far background sphere, so the camera move should be neglectable compared to the bg radius. Or you could simply turn it off, since the --bound is already large enough to model far objects. Finally, you could try to do the same thing with the original instant-ngp to get an expectation of how well the scene can be modelled.

menglongyue commented 2 years ago

ok, I will try it. I still have a question. I change the aabb_scale parameter to 4 in llff2nerf.py(default value is 2) . how to set this parameter?

ashawkey commented 2 years ago

In fact aabb_scale has no use in this codebase, it's replaced by --bound as stated in the readme. You may use the same aabb_scale as bound for the original instant-ngp codebase.

yyyue502 commented 2 years ago

I met similar issuse when training object with white background, some artifacts are projected to the bounding box, you may try to clip near and far or consider using the corse-to-fine sampling strategy like NeRF.