Closed hxSong22 closed 6 months ago
I'm not exactly sure what went wrong in this case. I think training instability can occur in rare cases, so I would try restarting training. For what it's worth, I was able to make training converge using the following settings (same scene, same number of iterations):
"+experiment=re10k",
"data_loader.train.num_workers=0",
"data_loader.val.num_workers=0",
"data_loader.test.num_workers=0",
"data_loader.train.batch_size=1",
"dataset.overfit_to_scene=db92831e45859439",
This gives me the following outputs:
train step 2160; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000844
train step 2161; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000923
train step 2162; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000900
train step 2163; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000862
train step 2164; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000944
train step 2165; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000939
train step 2166; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000873
train step 2167; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000847
train step 2168; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000872
train step 2169; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000866
train step 2170; scene = ['db92831e45859439']; context = [[0, 25]]; loss = 0.000889
Are you fine-tuning a pre-trained checkpoint? What are your settings? More details might help narrow down the issue.
Thanks for your kind response! I discover that this problem is caused by the version of my python packages. Sorry for the mistake!
Thanks for your excellent work! However when I try to overfit to a fixed scenen like 'db92831e45859439' provided in your processed dataset, the code seems not work because the loss is so large. I wonder what should I do besides set the overfit_to_scene parameter in main.yaml?![WeChat734f2b38fb8e644350e24a4e9c0d079d](https://github.com/dcharatan/pixelsplat/assets/118874036/5e5668d7-3689-45a4-b042-93a16da760e4)