wgcban / ddpm-cd

Remote Sensing Change Detection using Denoising Diffusion Probabilistic Models
https://www.wgcban.com/research#h.ar24vwqlm021
MIT License
264 stars 31 forks source link

Hardware + training time? #10

Closed bkj closed 1 year ago

bkj commented 1 year ago

Hello --

Are you able to share what hardware you used to train the model described in your paper, and what the wall-clock training time was?

Thanks!

wgcban commented 1 year ago

@bkj

All experiments used one NVIDIA RTX 8000 (48GB memory) GPU.

I remember that training the diffusion model took at least 3-4 days, but if you can train it longer, you will get better results.

Training the change detection classifier is relatively fast, and it depends on the dataset size.

Cassiatora commented 1 year ago

@wgcban hi,I want to know the data batchsize during training the diffusion model. And the batchsize of the code is 8?

wgcban commented 1 year ago

@Cassiatora, yes, we used the same batchsize during the experiments.

Cassiatora commented 1 year ago

@wgcban This batchsize is 8, but there are millions of pictures in Google Earth, is it iterated to 1,000,000 steps before passing an epoch. How many epochs are there at the end of training?

wgcban commented 1 year ago

@Cassiatora As I remember, I trained it for very long time, perhaps one-two weeks.

wgcban commented 1 year ago

@Cassiatora I will confirm the exact batch size I used during the training soon. I increased the batch size such that it fits to 48 GB GPU.