search

pierotofy / OpenSplat

Production-grade 3D gaussian splatting with CPU/GPU support for Windows, Mac and Linux 🚀
https://antimatter15.com/splat/?url=https://splat.uav4geo.com/banana.splat
GNU Affero General Public License v3.0
924 stars 87 forks source link

Fix ROCm 6.x training error #42

Closed pfxuan closed 8 months ago

pfxuan commented 8 months ago

After this fix, AMD GPU build should start working via the new ROCm 6.x docker environment 🎉.

The following is the hardware setup and initial test result. 

root@5fb916383b63:/code/build# time ./opensplat /data/banana -n 2000
Using CUDA
Reading 14241 points
Loading /data/banana/images/frame_00001.JPG
Loading /data/banana/images/frame_00002.JPG
Loading /data/banana/images/frame_00003.JPG
Loading /data/banana/images/frame_00004.JPG
Loading /data/banana/images/frame_00005.JPG
Loading /data/banana/images/frame_00006.JPG
Loading /data/banana/images/frame_00008.JPG
Loading /data/banana/images/frame_00009.JPG
Loading /data/banana/images/frame_00010.JPG
Loading /data/banana/images/frame_00011.JPG
Loading /data/banana/images/frame_00013.JPG
Loading /data/banana/images/frame_00014.JPG
Loading /data/banana/images/frame_00015.JPG
Loading /data/banana/images/frame_00016.JPG
Step 10: 0.208075
Step 20: 0.237863
Step 30: 0.181474
Step 40: 0.209521
Step 50: 0.160717
Step 60: 0.205582
Step 70: 0.147323
Step 80: 0.150524
Step 90: 0.147064
Step 100: 0.144563
Step 110: 0.125039
Step 120: 0.142733
Step 130: 0.189102
Step 140: 0.136359
Step 150: 0.138213
Step 160: 0.136127
Step 170: 0.178151
Step 180: 0.13395
Step 190: 0.142111
Step 200: 0.131241
Step 210: 0.178902
Step 220: 0.105384
Step 230: 0.137437
Step 240: 0.134837
Step 250: 0.128858
Step 260: 0.120589
Step 270: 0.174268
Step 280: 0.12046
Step 290: 0.102276
Step 300: 0.119286
Step 310: 0.129531
Step 320: 0.124861
Step 330: 0.123276
Step 340: 0.121402
Step 350: 0.115895
Step 360: 0.159928
Step 370: 0.0886419
Step 380: 0.109183
Step 390: 0.0915482
Step 400: 0.109655
Step 410: 0.111456
Step 420: 0.110098
Step 430: 0.0880052
Step 440: 0.0919362
Step 450: 0.0903232
Step 460: 0.108424
Step 470: 0.124387
Step 480: 0.10601
Step 490: 0.16722
Step 500: 0.0900435
Step 510: 0.117083
Step 520: 0.163578
Step 530: 0.1069
Step 540: 0.118836
Step 550: 0.105309
Step 560: 0.156448
Step 570: 0.164224
Step 580: 0.108291
Step 590: 0.153091
Step 600: 0.121695
Added 9591 gaussians, new count 23832
Culled 9895 gaussians, remaining 13937
Step 610: 0.166405
Step 620: 0.110598
Step 630: 0.125043
Step 640: 0.084933
Step 650: 0.0845598
Step 660: 0.119641
Step 670: 0.117407
Step 680: 0.117641
Step 690: 0.101347
Step 700: 0.150917
Added 15037 gaussians, new count 28974
Culled 5439 gaussians, remaining 23535
Step 710: 0.0916314
Step 720: 0.154107
Step 730: 0.124596
Step 740: 0.103116
Step 750: 0.120372
Step 760: 0.0796096
Step 770: 0.117717
Step 780: 0.115493
Step 790: 0.0769396
Step 800: 0.0984292
Added 22787 gaussians, new count 46322
Culled 8011 gaussians, remaining 38311
Step 810: 0.101665
Step 820: 0.118277
Step 830: 0.153499
Step 840: 0.0754875
Step 850: 0.112758
Step 860: 0.0972251
Step 870: 0.0860925
Step 880: 0.11283
Step 890: 0.108513
Step 900: 0.0681005
Added 31275 gaussians, new count 69586
Culled 10132 gaussians, remaining 59454
Step 910: 0.14729
Step 920: 0.0950705
Step 930: 0.148599
Step 940: 0.105241
Step 950: 0.106126
Step 960: 0.145349
Step 970: 0.106236
Step 980: 0.0896305
Step 990: 0.0987537
Step 1000: 0.0637969
Added 40189 gaussians, new count 99643
Culled 12016 gaussians, remaining 87627
Step 1010: 0.0732109
Step 1020: 0.0837388
Step 1030: 0.0838937
Step 1040: 0.103043
Step 1050: 0.0619884
Step 1060: 0.0830906
Step 1070: 0.122514
Step 1080: 0.0703483
Step 1090: 0.126708
Step 1100: 0.0943512
Added 48113 gaussians, new count 135740
Culled 13658 gaussians, remaining 122082
Step 1110: 0.141307
Step 1120: 0.0895955
Step 1130: 0.0609604
Step 1140: 0.0559767
Step 1150: 0.0964497
Step 1160: 0.0889039
Step 1170: 0.0929496
Step 1180: 0.0889424
Step 1190: 0.0522798
Step 1200: 0.06965
Added 55532 gaussians, new count 177614
Culled 14556 gaussians, remaining 163058
Step 1210: 0.138581
Step 1220: 0.115415
Step 1230: 0.065682
Step 1240: 0.116656
Step 1250: 0.113661
Step 1260: 0.0887083
Step 1270: 0.0662218
Step 1280: 0.0500107
Step 1290: 0.121573
Step 1300: 0.112594
Added 61540 gaussians, new count 224598
Culled 15636 gaussians, remaining 208962
Step 1310: 0.0591685
Step 1320: 0.0816074
Step 1330: 0.0896751
Step 1340: 0.0858046
Step 1350: 0.048018
Step 1360: 0.0954068
Step 1370: 0.0753745
Step 1380: 0.0552086
Step 1390: 0.0540024
Step 1400: 0.0960227
Added 64682 gaussians, new count 273644
Culled 16057 gaussians, remaining 257587
Step 1410: 0.0906328
Step 1420: 0.0786731
Step 1430: 0.0466605
Step 1440: 0.0808401
Step 1450: 0.0787142
Step 1460: 0.0566929
Step 1470: 0.0433018
Step 1480: 0.0594259
Step 1490: 0.0671842
Step 1500: 0.0391344
Added 67010 gaussians, new count 324597
Culled 16008 gaussians, remaining 308589
Step 1510: 0.117637
Step 1520: 0.0458945
Step 1530: 0.106889
Step 1540: 0.0520811
Step 1550: 0.0548199
Step 1560: 0.0660458
Step 1570: 0.0406176
Step 1580: 0.0684525
Step 1590: 0.054903
Step 1600: 0.0789318
Added 66922 gaussians, new count 375511
Culled 15448 gaussians, remaining 360063
Step 1610: 0.0779538
Step 1620: 0.0453232
Step 1630: 0.100514
Step 1640: 0.0383056
Step 1650: 0.042757
Step 1660: 0.0451564
Step 1670: 0.0513167
Step 1680: 0.0797595
Step 1690: 0.096207
Step 1700: 0.0456395
Added 67374 gaussians, new count 427437
Culled 15828 gaussians, remaining 411609
Step 1710: 0.04383
Step 1720: 0.0661522
Step 1730: 0.0685059
Step 1740: 0.0884331
Step 1750: 0.0449189
Step 1760: 0.0433282
Step 1770: 0.0476067
Step 1780: 0.0420889
Step 1790: 0.0565312
Step 1800: 0.0900256
Added 66777 gaussians, new count 478386
Culled 15137 gaussians, remaining 463249
Step 1810: 0.065665
Step 1820: 0.0536985
Step 1830: 0.0370926
Step 1840: 0.0438131
Step 1850: 0.0828206
Step 1860: 0.0422622
Step 1870: 0.0545112
Step 1880: 0.0589518
Step 1890: 0.0361636
Step 1900: 0.0545315
Added 65206 gaussians, new count 528455
Culled 14989 gaussians, remaining 513466
Step 1910: 0.0546802
Step 1920: 0.0598722
Step 1930: 0.0569776
Step 1940: 0.053701
Step 1950: 0.0508809
Step 1960: 0.0445479
Step 1970: 0.050296
Step 1980: 0.0339478
Step 1990: 0.0498263
Step 2000: 0.0350233
Added 63772 gaussians, new count 577238
Culled 14569 gaussians, remaining 562669
Wrote splat.ply

real    1m12.359s
user    1m14.793s
sys 0m4.042s

GPU usage: amdgpu_top

First AMD GPU based 3DGS: first_amd_3dgs_result

pierotofy commented 8 months ago

Ohhh, this is awesome! 🥳 Thanks for testing and for the fix.